Skip to content

Task group is swallowing CancelledError.Β #374

Closed

Description

@jonathanslenders

If any of the tasks within the task group raise a CancelledError, then that does not propagate out of the task group. In my case, this happens because the code within the task group is cancelled from somewhere else.

from anyio import create_task_group
import asyncio

async def main():
    async def in_task_group():
        raise asyncio.CancelledError

    async with create_task_group() as tg:
        tg.start_soon(in_task_group)

asyncio.run(main())

Would this be a bug? Or is this expected behavior?

Activity

  1. jonathanslenders commented on Oct 6, 2021

    @jonathanslenders
    ContributorAuthor

    Related to this, the following snippet exposes very weird behavior:

    from anyio import create_task_group
    import asyncio
    
    async def test():
        # Create a task group with one very long running task in here.
        async def in_task_group():
            await asyncio.sleep(100)
    
        async with create_task_group() as tg:
            tg.start_soon(in_task_group)
    
        # - The main function should cancel the coroutine around this point -
    
        # I would not expect any of the following to be printed.
        # Somehow, the task within the task group gets cancelled, but this function
        # does not get cancelled.
    
        print('Before sleep')
        await asyncio.sleep(2)
        print('After sleep')
    
    async def main():
        # Create task.
        task = asyncio.create_task(test())
    
        # Cancel task.
        await asyncio.sleep(1)
        task.cancel()
    
        await task
    
    asyncio.run(main())
  2. agronholm commented on Oct 7, 2021

    @agronholm
    Owner

    Well...this is how anyio has always worked. On trio we don't have this problem because cancel scopes are the only way to cancel things there. On asyncio, cancelling a task just means raising CancelledError once in the target task. I think that most people have not had this problem because they've used structural concurrency all the way.
    @graingert any thoughts?

  3. jonathanslenders commented on Oct 7, 2021

    @jonathanslenders
    ContributorAuthor

    Well, imagine we have a library that uses structured concurrency everywhere within that library. But this library is called by a different codebase that does not follow structured concurrency. The above issue means that it's impossible to use libraries written with anyio in a code base that doesn't use anyio by itself. Is that correct?

  4. added
    asyncioInvolves the asyncio backend
    designRequires a design decision
    on Oct 7, 2021
  5. agronholm commented on Oct 8, 2021

    @agronholm
    Owner

    Suppose a task group was already cancelled via SC, and then another task natively cancels the host task. How would AnyIO tell the difference?

  6. jonathanslenders commented on Oct 13, 2021

    @jonathanslenders
    ContributorAuthor

    @agronholm, honestly, I don't know. I worked around this by using a cancel scope anyway. But I think the issue remains that at some point, a task created by a library using anyio, could be cancelled using normal asyncio cancellation.

    I wonder whether it's feasible to create a little wrapper that runs the anyio task in a cancel scope, catches (elsewhere) the CancelledError that asyncio raises, and as a result cancels the cancel scope.

  7. agronholm commented on Oct 13, 2021

    @agronholm
    Owner

    A potential solution that I came up a couple days ago would be to wait for the subtasks in a shielded cancel scope. That way, only native cancellations would get through and would then be forced to propagate out of the task group.

  8. agronholm commented on Oct 13, 2021

    @agronholm
    Owner

    To detect native cancellations prior to that, we would check, upon cancellation, if one of the enclosing cancel scopes had been cancelled. If not, it's a native cancellation.

  9. agronholm commented on Feb 12, 2022

    @agronholm
    Owner

    I think I found a way around this, but on Python 3.9 and above only: we can set a special cancellation message in the exception to detect cancel scope based cancellations.

  10. gvanrossum commented on Feb 16, 2022

    @gvanrossum

    @jonathanslenders (#374 (comment)):

    Related to this, the following snippet exposes very weird behavior:
    [...]

    I translated this example to the new (3.11) asyncio.TaskGroup idiom and it seems to work.

    Details
    import asyncio
    
    async def test():
        # Create a task group with one very long running task in here.
        async def in_task_group():
            await asyncio.sleep(100)
    
        try:
            async with asyncio.TaskGroup() as tg:
                tg.create_task(in_task_group())
        except asyncio.CancelledError:
            print("pass")
    
        # - The main function should cancel the coroutine around this point -
    
        # I would not expect any of the following to be printed.
        # Somehow, the task within the task group gets cancelled, but this function
        # does not get cancelled.
    
        print('Before sleep')
        await asyncio.sleep(2)
        print('After sleep')
    
    async def main():
        # Create task.
        task = asyncio.create_task(test())
    
        # Cancel task.
        await asyncio.sleep(1)
        task.cancel()
    
        await task
    
    asyncio.run(main())

    (Only this is really changed:)

        try:
            async with asyncio.TaskGroup() as tg:
                tg.create_task(in_task_group())
        except asyncio.CancelledError:
            print("pass")

    This produces the following output:

    pass
    Before sleep
    After sleep
    

    Is that what you would like to happen?

  11. jonathanslenders commented on Jul 5, 2022

    @jonathanslenders
    ContributorAuthor

    @gvanrossum : Thanks for responding (I just see your reply). This is indeed what I'd expect. The CancelledError should propagate out of the TaskGroup.

    (I'm no longer sure whether ever catching a CancelledError is a good idea though. With anyio, I prefer finally blocks that are shielded from cancellation.)

  12. gvanrossum commented on Jul 5, 2022

    @gvanrossum

    Thanks for confirming.

  13. added this to the 4.0 milestone on Oct 18, 2022
  14. agronholm commented on Oct 18, 2022

    @agronholm
    Owner

    Fixing this is a high priority for v4.0. My initial attempts have not gone very well, as I'm seeing recursion errors from exception groups.

  15. 11 remaining items

  16. drernie commented on May 10, 2023

    @drernie

    Any ETA on this? Anything we can do to help?

  17. agronholm commented on May 12, 2023

    @agronholm
    Owner

    The first order of business I have with AnyIO right now is getting the v3.7.0 release out. It contains a ton of other fixes that have waited a long time to be released. One of these fixes will even make tackling this issue a bit easier by reducing the amount of new code that would have to be written. I'm just about ready to make that release. After that, I can fully focus on fixing this particular issue, but I will have to scrap my previous attempt and start from scratch, as the code got pretty complicated back then and I was overwhelmed with the complexity. As for an ETA, I hope to get this done in 2-3 weeks.

  18. agronholm commented on May 16, 2023

    @agronholm
    Owner

    I've restarted my efforts based on current master. I have a test in place that reproduces the issue. I think I might start writing a fix that works on Python 3.11+ (where thanks to uncancel() we can actually track the cancellations without too many hacks). Then I could work on said hacks for earlier Pythons, as I was trying to do before going on a hiatus.

  19. T-256 commented on May 17, 2023

    @T-256

    Hi @agronholm,
    If you didn't start the progress, I'll be happy to help.
    Are there any good points to start?

  20. reidmeyer commented on Jun 1, 2023

    @reidmeyer

    Any luck?

  21. agronholm commented on Jul 10, 2023

    @agronholm
    Owner

    Happy to report that I'm making progress again, and hopefully I can make a release candidate of v4.0 this week, unless I encounter unexpectedly difficult corner cases.

  22. reidmeyer commented on Jul 10, 2023

    @reidmeyer

    Happy to report that I'm making progress again, and hopefully I can make a release candidate of v4.0 this week, unless I encounter unexpectedly difficult corner cases.

    Awesome to hear!

  23. agronholm commented on Jul 12, 2023

    @agronholm
    Owner

    @jonathanslenders would you mind testing the PR to make sure that it solves your original problem? Do note that the exception raising behavior has changed in a backwards incompatible fashion when you test.

  24. agronholm commented on Jul 17, 2023

    @agronholm
    Owner

    The merged PR should solve the problem as described. There is still the issue of cancel scopes not distinguishing between native cancellations and AnyIO cancellations on Python < 3.11. I have a PR for that in progress, but I'm running into some very subtle issues and am not yet confident about it, so that fix might not materialize in v4.0.0.

  25. drernie commented on Jul 17, 2023

    @drernie

    Sweet! Any ETA on the formal v4.0.0 release?

  26. agronholm commented on Jul 17, 2023

    @agronholm
    Owner

    Once #591 is merged, and another upcoming minor PR related to cancellation exceptions is dealt with, I'll be ready to release v4.0.0rc1. Given the extensive backwards incompatible changes coming in that release, I want to let the release candidates stew for at least 2-3 weeks before making a final release.

  27. agronholm commented on Jul 17, 2023

    @agronholm
    Owner

    The expected release date for v4.0.0rc1 would be later this week.

  28. agronholm commented on Jul 23, 2023

    @agronholm
    Owner

    There was no release yet, but I've posted a status update in discussions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    asyncioInvolves the asyncio backenddesignRequires a design decision

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions