Skip to content

New asyncio.exceptions.CancelledError from 4.5.2Β #2633

Closed

Description

@rra

Version: 4.5.2
Platform: Python 3.11.2 on Debian unstable
Description:

Starting with 4.5.2, I'm seeing a new exception when attempting to write to Redis using the asyncio layer. The problem seems to be related to FastAPI: if I create a FastAPI app with any Starlette custom middlware, and then try to set a key in Redis using a separate connection pool created with redis.asyncio.from_url, the set request fails immediately with an uncaught asyncio.exceptions.CancelledError and the following backtrace:

.tox/py/lib/python3.11/site-packages/gafaelfawr/storage/base.py:140: in store
    await self._redis.set(key, encrypted_data, ex=lifetime)
.tox/py/lib/python3.11/site-packages/redis/asyncio/client.py:509: in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:1408: in get_connection
    if await connection.can_read_destructive():
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:817: in can_read_destructive
    return await self._parser.can_read_destructive()
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:250: in can_read_destructive
    return await self._stream.read(1)
/usr/lib/python3.11/asyncio/streams.py:689: in read
    await self._wait_for_data('read')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <StreamReader transport=<_SelectorSocketTransport fd=15 read=polling write=<idle, bufsize=0>>>
func_name = 'read'

    async def _wait_for_data(self, func_name):
        """Wait until feed_data() or feed_eof() is called.
    
        If stream was paused, automatically resume it.
        """
        # StreamReader uses a future to link the protocol feed_data() method
        # to a read coroutine. Running two read coroutines at the same time
        # would have an unexpected behaviour. It would not possible to know
        # which coroutine would get the next data.
        if self._waiter is not None:
            raise RuntimeError(
                f'{func_name}() called while another coroutine is '
                f'already waiting for incoming data')
    
        assert not self._eof, '_wait_for_data after EOF'
    
        # Waiting for data while paused will make deadlock, so prevent it.
        # This is essential for readexactly(n) for case when n > self._limit.
        if self._paused:
            self._paused = False
            self._transport.resume_reading()
    
        self._waiter = self._loop.create_future()
        try:
>           await self._waiter
E           asyncio.exceptions.CancelledError

/usr/lib/python3.11/asyncio/streams.py:522: CancelledError

GitHub Actions failure log: https://github.com/lsst-sqre/gafaelfawr/actions/runs/4473916874/jobs/7861846517

Reverting to redis 4.5.1 makes the problem disappear again, so it appears to be triggered by some change in 4.5.2.

Here's the shortest test case that I've come up with. It depends on FastAPI and Starlette but none of my code base. My guess is that something is happening during addition of the middleware that's breaking something about the asyncio work inside redis, but I'm not sure what that could be. Oddly, adding one of the standard Starlette middleware classes does not trigger this problem, only a custom middleware class derived from BaseHTTPMiddleware.

from collections.abc import Callable, Awaitable

import redis.asyncio
from fastapi import FastAPI, APIRouter, Request, Response
from httpx import AsyncClient
from starlette.middleware.base import BaseHTTPMiddleware


class DummyMiddleware(BaseHTTPMiddleware):
    async def dispatch(
        self,
        request: Request,
        call_next: Callable[[Request], Awaitable[Response]],
    ) -> Response:
        return await call_next(request)


@pytest.mark.asyncio
async def test_redis() -> None:
    app = FastAPI()
    app.add_middleware(DummyMiddleware)

    @app.get("/")
    async def index() -> dict[str, str]:
        return {}

    async with AsyncClient(app=app, base_url="https://example.com/") as client:
        await client.get("/")
        pool = redis.asyncio.from_url("redis://localhost:6379/0")
        await pool.set("key", "value")

(If I had to guess, it would be the switch to asyncio.timeout, but I can't find evidence to support that. I know asyncio.timeout does raise CancelledError and then converts it to TimeoutError, but I couldn't find any place where that machinery shouldn't work correctly. This error happens immediately, so it doesn't seem to be a timeout and therefore I may be barking up the wrong tree.)

Activity

  1. rra commented on Mar 21, 2023

    @rra
    Author

    Ah, Starlette's BaseHTTPMiddleware calls anyio.create_task_group(). I suspect that is related somehow, although I'm still not seeing exactly how. https://github.com/encode/starlette/blob/master/starlette/middleware/base.py#L39

  2. added a commit that references this issue on Mar 21, 2023
  3. sileht commented on Mar 22, 2023

    @sileht
    Contributor

    I guess the CancellationError is raised here, where it should be an asyncio.Timeout.
    CleanShot 2023-03-22 at 10 31 26

  4. sileht commented on Mar 22, 2023

    @sileht
    Contributor

    asyncio check that asyncio.current_task().uncancel() == 0 before converting CancelledError into TimeoutError, so somehow this value is unexpected, not 0.

  5. sileht commented on Mar 22, 2023

    @sileht
    Contributor

    The thing is that this bug was always there, but with the legacy async_timeout lib, the bug was hidden because of this line:
    https://github.com/aio-libs/async-timeout/blob/master/async_timeout/__init__.py#L210

    All CancelledError was converted into TimeoutError even if the cancellation source was not the timeout.

  6. sileht commented on Mar 22, 2023

    @sileht
    Contributor

    Looks related to agronholm/anyio#374

  7. curtiscook commented on Mar 22, 2023

    @curtiscook

    FWIW: I'm experiencing the same issue. I've noticed that the connection does work, but seems to break if you await a function that's not Redis. We were able to reproduce this on the command line w/o starlette (even though we are using FastAPI)
    given

    def get_redis_pool() -> Redis:
        global _redis_pool
    
        if _redis_pool is None or _redis_pool.connection is None:
            _redis_pool = aioredis.from_url(REDIS_URL, max_connections=10)
        assert _redis_pool is not None
        return _redis_pool
    
    
    async def get_key(key: str) -> Any:
        redis = get_redis_pool()
        return await redis.get(key)
    
    
    async def set_key(key: str, value: Any, expire_seconds: Optional[int] = None) -> bool:
        redis = get_redis_pool()
        if expire_seconds is not None:
            exp = timedelta(seconds=expire_seconds)
            await redis.set(key, value, ex=exp)
        else:
            await redis.set(key, value)
        return True
    

    This function will work fine:

    async def foo():
    	await get_key("key2")
    	await set_key("key2", "value")
    

    This will break:

    async def foo():
    	await get_key("key2")
    	await bar()
    	await set_key("key2", "value")
    

    Might be related that bar uses HTTPX

  8. sileht commented on Mar 22, 2023

    @sileht
    Contributor

    Both starlette and httpx rely on anyio

  9. rra commented on Mar 22, 2023

    @rra
    Author

    Yeah, this is looking to me like an anyio bug caused by use of task groups, presumably related to this admonition in the Python library documentation:

    The asyncio components that enable structured concurrency, like asyncio.TaskGroup and asyncio.timeout(), are implemented using cancellation internally and might misbehave if a coroutine swallows asyncio.CancelledError.

    I don't completely understand the mechanism yet, but enough keywords are lining up that this looks like it. @curtiscook's example is interesting, though, since I can't find a call to create_task_group in httpx or httpcore. Maybe it's somewhere else in httpx's dependencies, or maybe that bar function uses anyio?

  10. rra commented on Mar 22, 2023

    @rra
    Author

    BTW, for those who are specifically having this issue with Starlette middleware, it looks like Starlette has rewritten their built-in middleware to avoid the use of BaseHTTPMiddleware at the cost of having to interact at a lower level with the request and response. That avoids creating a task group, and I wonder if that would make the problem go away. I'm considering rewriting our middleware similarly just as an experiment.

  11. decaz commented on Mar 23, 2023

    @decaz

    Looks like version 4.5.3 release didn't resolve the issue.

  12. bellini666 commented on Mar 23, 2023

    @bellini666
    Contributor

    FWIW: I'm experiencing the same issue. I've noticed that the connection does work, but seems to break if you await a function that's not Redis. We were able to reproduce this on the command line w/o starlette (even though we are using FastAPI) given

    def get_redis_pool() -> Redis:
        global _redis_pool
    
        if _redis_pool is None or _redis_pool.connection is None:
            _redis_pool = aioredis.from_url(REDIS_URL, max_connections=10)
        assert _redis_pool is not None
        return _redis_pool
    
    
    async def get_key(key: str) -> Any:
        redis = get_redis_pool()
        return await redis.get(key)
    
    
    async def set_key(key: str, value: Any, expire_seconds: Optional[int] = None) -> bool:
        redis = get_redis_pool()
        if expire_seconds is not None:
            exp = timedelta(seconds=expire_seconds)
            await redis.set(key, value, ex=exp)
        else:
            await redis.set(key, value)
        return True
    

    This function will work fine:

    async def foo():
    	await get_key("key2")
    	await set_key("key2", "value")
    

    This will break:

    async def foo():
    	await get_key("key2")
    	await bar()
    	await set_key("key2", "value")
    

    Might be related that bar uses HTTPX

    The same here! I have an async function that will sometimes call an external resource using httpx to validate the input. Everything works fine when httpx is not called, but when it is called then I get CancelledError.

    This started happening after upgrading to 4.5.2/4.5.3 (not sure which one exactly since we upgraded from 4.5.1 directly)

  13. ice1x commented on Mar 23, 2023

    @ice1x

    Seems like blocker for Redis isn't it?

  14. sileht commented on Mar 23, 2023

    @sileht
    Contributor

    This bug was always there:

  15. bellini666 commented on Mar 23, 2023

    @bellini666
    Contributor

    This bug was always there:

    But why does CancelledError gets raised when doing some other async stuff in the same task? E.g. the httpx issue mentioned by me here and by @curtiscook here

    I don't understand why CancelledError is being raised when calling a redis api just because I called httx before it (if I don't call httpx, CancelledError is not raised, but when calling, CancelledError` will be raised every time)

    I worked around this error by wrapping my call to redis in an asyncio.shield, and although it should be harmless, it really looks like something wrong is happening here.

    I'll try to create a minimum reproducible demo

  16. 12 remaining items

  17. pinheiro-bruno commented on Jun 5, 2023

    @pinheiro-bruno

    I'm facing the exactly same issue on version 4.5.5 using python 3.11.2

    The issue started after anyio version was bumped from 3.6.2 to 3.7.0. The suggested solution of updating python version to 3.11.3 did the trick for me.

  18. Mark90 commented on Jun 9, 2023

    @Mark90

    For me the issue also started to occur after anyio was bumped to 3.7.0.

    Posting here because I had a slightly different version configuration;

    anyio==3.7.0
    redis==4.4.4
    python==3.11.4
    

    Upgrading to redis==4.5.5 resolved it for me.

  19. HacKanCuBa commented on Jun 22, 2023

    @HacKanCuBa

    With Python 3.11.2, this issue still persists, even using redis==4.5.5 (with hiredis==2.2.3)

    Edit: I confirm it all works when upgrading to Python 3.11.4 πŸ‘―

  20. added a commit that references this issue on Jul 3, 2023
  21. bsnacks000 commented on Jul 12, 2023

    @bsnacks000

    We had the same problem... resolved with 3.11.4 bump from 3.11.2... thanks for figuring that out ... πŸ‘

    python = "^3.11.4"
    redis = "^4.6.0"
    httpx = "0.23.0"
    anyio = "^3.7.1"
    
  22. added a commit that references this issue on Jul 16, 2023
  23. hendrixmar commented on Jul 18, 2023

    @hendrixmar

    Thanks for the help it worked upgrading to 3.11.4

  24. Naltharial commented on Apr 1, 2024

    @Naltharial

    @dvora-h Apologies for resurrecting an old issue, but it seems like it's relevant again. I'm seeing this happen on 3.11.8 with AnyIO 4, PyRedis 5 using @bellini666 's test script.

    Using python3.11 (3.11.8)
    without_httpx
    b'1'
    b'2'
    with_httpx_async
    b'1'
    {'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.27.0', 'X-Amzn-Trace-Id': 'Root=1-660b04d4-0ed17ec26d7b388724aaf03b'}, 'origin': '212.85.182.26', 'url': 'https://httpbin.org/get'}
    Traceback (most recent call last):
      File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 35, in <module>
        asyncio.run(main())
      File "/usr/lib/python3.11/asyncio/runners.py", line 188, in run
        return runner.run(main)
               ^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/asyncio/runners.py", line 120, in run
        return self._loop.run_until_complete(task)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 31, in main
        await with_httpx_async()
      File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 23, in with_httpx_async
        print(await conn.get("bar"))
              ^^^^^^^^^^^^^^^^^^^^^
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/client.py", line 605, in execute_command
        conn = self.connection or await pool.get_connection(command_name, **options)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1076, in get_connection
        await self.ensure_connection(connection)
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1115, in ensure_connection
        if await connection.can_read_destructive():
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 504, in can_read_destructive
        return await self._parser.can_read_destructive()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/_parsers/base.py", line 185, in can_read_destructive
        return await self._stream.read(1)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/asyncio/streams.py", line 690, in read
        await self._wait_for_data('read')
      File "/usr/lib/python3.11/asyncio/streams.py", line 523, in _wait_for_data
        await self._waiter
    asyncio.exceptions.CancelledError
    Exception ignored in: <function AbstractConnection.__del__ at 0x7f06a59e40e0>
    Traceback (most recent call last):
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 216, in __del__
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 223, in _close
      File "/usr/lib/python3.11/asyncio/streams.py", line 345, in close
      File "/usr/lib/python3.11/asyncio/selector_events.py", line 831, in close
      File "/usr/lib/python3.11/asyncio/base_events.py", line 758, in call_soon
      File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    RuntimeError: Event loop is closed
    

    it looks like HTTPx is closing the event loop behind it, no matter if Redis is still trying to use it. Is this related to the same issue? Notably, this only occurs under Ubuntu, I can't reproduce it on Windows, so my debugging capacity is limited.

    It's resolved (obviously) by using the sync Redis API instead of the async one, but upgrading the libraries didn't work.

    Package                       Version
    ----------------------------- -----------
    ...
    anyio                         4.3.0
    httpx                         0.27.0
    redis                         5.0.3
    trio                          0.25.0
    ...
    
  25. Naltharial commented on Apr 2, 2024

    @Naltharial

    The event loop closing might be a red herring, this is the output on Py3.11.8 / Windows:

    b'1'
    b'2'
    with_httpx_async
    b'1'
    {'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.27.0', 'X-Amzn-Trace-Id': 'Root=1-660bc9f9-43ac10016bb2539865135414'}, 'origin': '212.85.182.26', 'url': 'https://httpbin.org/get'}
    b'2'
    Exception ignored in: <function AbstractConnection.__del__ at 0x000002D87C3D1620>
    Traceback (most recent call last):
      File "C:\Users\Uporabnik\AppData\Local\pypoetry\Cache\virtualenvs\assembler-JBayZ5rl-py3.11\Lib\site-packages\redis\asyncio\connection.py", line 216, in __del__
      File "C:\Users\Uporabnik\AppData\Local\pypoetry\Cache\virtualenvs\assembler-JBayZ5rl-py3.11\Lib\site-packages\redis\asyncio\connection.py", line 223, in _close
      File "C:\Python311\Lib\asyncio\streams.py", line 358, in close
      File "C:\Python311\Lib\asyncio\proactor_events.py", line 109, in close
      File "C:\Python311\Lib\asyncio\base_events.py", line 762, in call_soon
      File "C:\Python311\Lib\asyncio\base_events.py", line 520, in _check_closed
    RuntimeError: Event loop is closed
    

    The b'2' comes through and no CancelledError, but something is still trying to access the event loop from Redis even past the end. The usual culprit for Linux/Windows async inconsistencies is uvloop, but its presence didn't seem to change anything.

  26. Mark90 commented on Apr 2, 2024

    @Mark90

    I'm not hitting this issue right now, but I got curious and I can also reproduce it with the latest test script from @bellini666. I tested a few more scenarios.
    Note that I'm running Windows, but this is tested with Ubuntu 20.04 in Windows Subsystem Linux (basically a VM).

    For each python version I tested with, I started with redis-py 5.0.3 and downgraded through 5.0.2 -> 5.0.1 -> 5.0.0 -> 4.6.0 to see if/when behavior changed. The envs were clean and I did not install trio nor uvloop.

    python anyio httpx redis result
    3.11.1 4.3.0 0.27.0 5.0.0-5.0.3 CE + RE on conn.get() after httpx
    3.11.1 4.3.0 0.27.0 4.6.0 works
    3.11.8 4.3.0 0.27.0 5.0.2, 5.0.3 works, but RE on shutdown
    3.11.8 4.3.0 0.27.0 4.6.0, 5.0.0, 5.0.1 works
    3.12.1 4.3.0 0.27.0 5.0.2, 5.0.3 works, but RE on shutdown
    3.12.1 4.3.0 0.27.0 4.6.0, 5.0.0, 5.0.1 works

    CE = asyncio.exceptions.CancelledError (same as @Naltharial)
    RE = RuntimeError: Event loop is closed (Exception ignored in AbstractConnection.del, same as @Naltharial)

    Some other observations:

    • In the case of 3.11.1, moving the conn.get() into the with AsyncClient block did not trigger the issue as long as it comes before the client.get() call. Right below the client.get() call (within the context) is where the problem starts
    • I tried swapping httpx out for aiohttp and this does not trigger the issue, so somehow httpx and redis-py bite (could still be a be a timing-related coincidence)

    Because on 3.11.8/3.12.1 the behavior seems to start from redis-py 5.0.2 onwards, perhaps #2999 could be related? (edit: except that does not explain the combination 3.11.1 and redis-py 5.0.0..)

  27. yinziyan1206 commented on Apr 8, 2024

    @yinziyan1206

    @dvora-h Apologies for resurrecting an old issue, but it seems like it's relevant again. I'm seeing this happen on 3.11.8 with AnyIO 4, PyRedis 5 using @bellini666 's test script.

    Using python3.11 (3.11.8)
    without_httpx
    b'1'
    b'2'
    with_httpx_async
    b'1'
    {'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.27.0', 'X-Amzn-Trace-Id': 'Root=1-660b04d4-0ed17ec26d7b388724aaf03b'}, 'origin': '212.85.182.26', 'url': 'https://httpbin.org/get'}
    Traceback (most recent call last):
      File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 35, in <module>
        asyncio.run(main())
      File "/usr/lib/python3.11/asyncio/runners.py", line 188, in run
        return runner.run(main)
               ^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/asyncio/runners.py", line 120, in run
        return self._loop.run_until_complete(task)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 31, in main
        await with_httpx_async()
      File "/mnt/d/Development/Infold/assembler/asyn_bug.py", line 23, in with_httpx_async
        print(await conn.get("bar"))
              ^^^^^^^^^^^^^^^^^^^^^
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/client.py", line 605, in execute_command
        conn = self.connection or await pool.get_connection(command_name, **options)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1076, in get_connection
        await self.ensure_connection(connection)
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 1115, in ensure_connection
        if await connection.can_read_destructive():
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 504, in can_read_destructive
        return await self._parser.can_read_destructive()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/_parsers/base.py", line 185, in can_read_destructive
        return await self._stream.read(1)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/asyncio/streams.py", line 690, in read
        await self._wait_for_data('read')
      File "/usr/lib/python3.11/asyncio/streams.py", line 523, in _wait_for_data
        await self._waiter
    asyncio.exceptions.CancelledError
    Exception ignored in: <function AbstractConnection.__del__ at 0x7f06a59e40e0>
    Traceback (most recent call last):
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 216, in __del__
      File "/home/naltharial/.cache/pypoetry/virtualenvs/assembler-01ENdme1-py3.11/lib/python3.11/site-packages/redis/asyncio/connection.py", line 223, in _close
      File "/usr/lib/python3.11/asyncio/streams.py", line 345, in close
      File "/usr/lib/python3.11/asyncio/selector_events.py", line 831, in close
      File "/usr/lib/python3.11/asyncio/base_events.py", line 758, in call_soon
      File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    RuntimeError: Event loop is closed
    

    it looks like HTTPx is closing the event loop behind it, no matter if Redis is still trying to use it. Is this related to the same issue? Notably, this only occurs under Ubuntu, I can't reproduce it on Windows, so my debugging capacity is limited.

    It's resolved (obviously) by using the sync Redis API instead of the async one, but upgrading the libraries didn't work.

    Package                       Version
    ----------------------------- -----------
    ...
    anyio                         4.3.0
    httpx                         0.27.0
    redis                         5.0.3
    trio                          0.25.0
    ...
    

    I have the same problem. And here is the example

    async def main():
        cache = Redis(connection_pool=pool)
        async with cache:
            print(await cache.get("key"))  # it's ok
            async with httpx.AsyncClient() as client:
                print(await cache.get("key"))  # it's ok
                response = await client.get("https://www.github.com")
                print(await cache.get("key"))  # here is broken
                # TODO

    PS: It works well for redis-py<5 or anyio<4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions