You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If any of the tasks within the task group raise a CancelledError, then that does not propagate out of the task group. In my case, this happens because the code within the task group is cancelled from somewhere else.
Related to this, the following snippet exposes very weird behavior:
fromanyioimportcreate_task_groupimportasyncioasyncdeftest():
# Create a task group with one very long running task in here.asyncdefin_task_group():
awaitasyncio.sleep(100)
asyncwithcreate_task_group() astg:
tg.start_soon(in_task_group)
# - The main function should cancel the coroutine around this point -# I would not expect any of the following to be printed.# Somehow, the task within the task group gets cancelled, but this function# does not get cancelled.print('Before sleep')
awaitasyncio.sleep(2)
print('After sleep')
asyncdefmain():
# Create task.task=asyncio.create_task(test())
# Cancel task.awaitasyncio.sleep(1)
task.cancel()
awaittaskasyncio.run(main())
Well...this is how anyio has always worked. On trio we don't have this problem because cancel scopes are the only way to cancel things there. On asyncio, cancelling a task just means raising CancelledError once in the target task. I think that most people have not had this problem because they've used structural concurrency all the way. @graingert any thoughts?
Well, imagine we have a library that uses structured concurrency everywhere within that library. But this library is called by a different codebase that does not follow structured concurrency. The above issue means that it's impossible to use libraries written with anyio in a code base that doesn't use anyio by itself. Is that correct?
@agronholm, honestly, I don't know. I worked around this by using a cancel scope anyway. But I think the issue remains that at some point, a task created by a library using anyio, could be cancelled using normal asyncio cancellation.
I wonder whether it's feasible to create a little wrapper that runs the anyio task in a cancel scope, catches (elsewhere) the CancelledError that asyncio raises, and as a result cancels the cancel scope.
A potential solution that I came up a couple days ago would be to wait for the subtasks in a shielded cancel scope. That way, only native cancellations would get through and would then be forced to propagate out of the task group.
To detect native cancellations prior to that, we would check, upon cancellation, if one of the enclosing cancel scopes had been cancelled. If not, it's a native cancellation.
I think I found a way around this, but on Python 3.9 and above only: we can set a special cancellation message in the exception to detect cancel scope based cancellations.
Related to this, the following snippet exposes very weird behavior:
[...]
I translated this example to the new (3.11) asyncio.TaskGroup idiom and it seems to work.
Details
importasyncioasyncdeftest():
# Create a task group with one very long running task in here.asyncdefin_task_group():
awaitasyncio.sleep(100)
try:
asyncwithasyncio.TaskGroup() astg:
tg.create_task(in_task_group())
exceptasyncio.CancelledError:
print("pass")
# - The main function should cancel the coroutine around this point -# I would not expect any of the following to be printed.# Somehow, the task within the task group gets cancelled, but this function# does not get cancelled.print('Before sleep')
awaitasyncio.sleep(2)
print('After sleep')
asyncdefmain():
# Create task.task=asyncio.create_task(test())
# Cancel task.awaitasyncio.sleep(1)
task.cancel()
awaittaskasyncio.run(main())
@gvanrossum : Thanks for responding (I just see your reply). This is indeed what I'd expect. The CancelledError should propagate out of the TaskGroup.
(I'm no longer sure whether ever catching a CancelledError is a good idea though. With anyio, I prefer finally blocks that are shielded from cancellation.)
The first order of business I have with AnyIO right now is getting the v3.7.0 release out. It contains a ton of other fixes that have waited a long time to be released. One of these fixes will even make tackling this issue a bit easier by reducing the amount of new code that would have to be written. I'm just about ready to make that release. After that, I can fully focus on fixing this particular issue, but I will have to scrap my previous attempt and start from scratch, as the code got pretty complicated back then and I was overwhelmed with the complexity. As for an ETA, I hope to get this done in 2-3 weeks.
Reacted by Dave Hirschfeld
Reacted by Dave Hirschfeld, Dr. Ernie Prabhakar and Santiago Fraire Willemoes
I've restarted my efforts based on current master. I have a test in place that reproduces the issue. I think I might start writing a fix that works on Python 3.11+ (where thanks to uncancel() we can actually track the cancellations without too many hacks). Then I could work on said hacks for earlier Pythons, as I was trying to do before going on a hiatus.
Reacted by Marat Sharafutdinov, T-256, Reid Meyer and Santiago Fraire Willemoes
Happy to report that I'm making progress again, and hopefully I can make a release candidate of v4.0 this week, unless I encounter unexpectedly difficult corner cases.
Reacted by Joshua Bronson, Dave Hirschfeld, T-256, Dr. Ernie Prabhakar, Santiago Fraire Willemoes and Alex Sheluchin
Happy to report that I'm making progress again, and hopefully I can make a release candidate of v4.0 this week, unless I encounter unexpectedly difficult corner cases.
@jonathanslenders would you mind testing the PR to make sure that it solves your original problem? Do note that the exception raising behavior has changed in a backwards incompatible fashion when you test.
Reacted by Santiago Fraire Willemoes and Reid Meyer
The merged PR should solve the problem as described. There is still the issue of cancel scopes not distinguishing between native cancellations and AnyIO cancellations on Python < 3.11. I have a PR for that in progress, but I'm running into some very subtle issues and am not yet confident about it, so that fix might not materialize in v4.0.0.
Once #591 is merged, and another upcoming minor PR related to cancellation exceptions is dealt with, I'll be ready to release v4.0.0rc1. Given the extensive backwards incompatible changes coming in that release, I want to let the release candidates stew for at least 2-3 weeks before making a final release.
Activity
jonathanslenders commentedon Oct 6, 2021
Related to this, the following snippet exposes very weird behavior:
agronholm commentedon Oct 7, 2021
Well...this is how anyio has always worked. On trio we don't have this problem because cancel scopes are the only way to cancel things there. On asyncio, cancelling a task just means raising
CancelledErroronce in the target task. I think that most people have not had this problem because they've used structural concurrency all the way.@graingert any thoughts?
jonathanslenders commentedon Oct 7, 2021
Well, imagine we have a library that uses structured concurrency everywhere within that library. But this library is called by a different codebase that does not follow structured concurrency. The above issue means that it's impossible to use libraries written with anyio in a code base that doesn't use anyio by itself. Is that correct?
agronholm commentedon Oct 8, 2021
Suppose a task group was already cancelled via SC, and then another task natively cancels the host task. How would AnyIO tell the difference?
jonathanslenders commentedon Oct 13, 2021
@agronholm, honestly, I don't know. I worked around this by using a cancel scope anyway. But I think the issue remains that at some point, a task created by a library using anyio, could be cancelled using normal asyncio cancellation.
I wonder whether it's feasible to create a little wrapper that runs the anyio task in a cancel scope, catches (elsewhere) the
CancelledErrorthat asyncio raises, and as a result cancels the cancel scope.agronholm commentedon Oct 13, 2021
A potential solution that I came up a couple days ago would be to wait for the subtasks in a shielded cancel scope. That way, only native cancellations would get through and would then be forced to propagate out of the task group.
agronholm commentedon Oct 13, 2021
To detect native cancellations prior to that, we would check, upon cancellation, if one of the enclosing cancel scopes had been cancelled. If not, it's a native cancellation.
agronholm commentedon Feb 12, 2022
I think I found a way around this, but on Python 3.9 and above only: we can set a special cancellation message in the exception to detect cancel scope based cancellations.
gvanrossum commentedon Feb 16, 2022
@jonathanslenders (#374 (comment)):
I translated this example to the new (3.11) asyncio.TaskGroup idiom and it seems to work.
Details
(Only this is really changed:)
This produces the following output:
Is that what you would like to happen?
jonathanslenders commentedon Jul 5, 2022
@gvanrossum : Thanks for responding (I just see your reply). This is indeed what I'd expect. The
CancelledErrorshould propagate out of theTaskGroup.(I'm no longer sure whether ever catching a
CancelledErroris a good idea though. With anyio, I prefer finally blocks that are shielded from cancellation.)gvanrossum commentedon Jul 5, 2022
Thanks for confirming.
agronholm commentedon Oct 18, 2022
Fixing this is a high priority for v4.0. My initial attempts have not gone very well, as I'm seeing recursion errors from exception groups.
11 remaining items
drernie commentedon May 10, 2023
Any ETA on this? Anything we can do to help?
agronholm commentedon May 12, 2023
The first order of business I have with AnyIO right now is getting the v3.7.0 release out. It contains a ton of other fixes that have waited a long time to be released. One of these fixes will even make tackling this issue a bit easier by reducing the amount of new code that would have to be written. I'm just about ready to make that release. After that, I can fully focus on fixing this particular issue, but I will have to scrap my previous attempt and start from scratch, as the code got pretty complicated back then and I was overwhelmed with the complexity. As for an ETA, I hope to get this done in 2-3 weeks.
agronholm commentedon May 16, 2023
I've restarted my efforts based on current
master. I have a test in place that reproduces the issue. I think I might start writing a fix that works on Python 3.11+ (where thanks touncancel()we can actually track the cancellations without too many hacks). Then I could work on said hacks for earlier Pythons, as I was trying to do before going on a hiatus.T-256 commentedon May 17, 2023
Hi @agronholm,
If you didn't start the progress, I'll be happy to help.
Are there any good points to start?
reidmeyer commentedon Jun 1, 2023
Any luck?
agronholm commentedon Jul 10, 2023
Happy to report that I'm making progress again, and hopefully I can make a release candidate of v4.0 this week, unless I encounter unexpectedly difficult corner cases.
reidmeyer commentedon Jul 10, 2023
Awesome to hear!
agronholm commentedon Jul 12, 2023
@jonathanslenders would you mind testing the PR to make sure that it solves your original problem? Do note that the exception raising behavior has changed in a backwards incompatible fashion when you test.
agronholm commentedon Jul 17, 2023
The merged PR should solve the problem as described. There is still the issue of cancel scopes not distinguishing between native cancellations and AnyIO cancellations on Python < 3.11. I have a PR for that in progress, but I'm running into some very subtle issues and am not yet confident about it, so that fix might not materialize in v4.0.0.
drernie commentedon Jul 17, 2023
Sweet! Any ETA on the formal v4.0.0 release?
agronholm commentedon Jul 17, 2023
Once #591 is merged, and another upcoming minor PR related to cancellation exceptions is dealt with, I'll be ready to release v4.0.0rc1. Given the extensive backwards incompatible changes coming in that release, I want to let the release candidates stew for at least 2-3 weeks before making a final release.
agronholm commentedon Jul 17, 2023
The expected release date for v4.0.0rc1 would be later this week.
agronholm commentedon Jul 23, 2023
There was no release yet, but I've posted a status update in discussions.
Remove stale ONVIF asyncio.CancelledError workaround for anyio home-aβ¦