Repository navigation
Description
Version: 4.5.2
Platform: Python 3.11.2 on Debian unstable
Description:
Starting with 4.5.2, I'm seeing a new exception when attempting to write to Redis using the asyncio layer. The problem seems to be related to FastAPI: if I create a FastAPI app with any Starlette custom middlware, and then try to set a key in Redis using a separate connection pool created with redis.asyncio.from_url, the set request fails immediately with an uncaught asyncio.exceptions.CancelledError and the following backtrace:
.tox/py/lib/python3.11/site-packages/gafaelfawr/storage/base.py:140: in store
await self._redis.set(key, encrypted_data, ex=lifetime)
.tox/py/lib/python3.11/site-packages/redis/asyncio/client.py:509: in execute_command
conn = self.connection or await pool.get_connection(command_name, **options)
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:1408: in get_connection
if await connection.can_read_destructive():
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:817: in can_read_destructive
return await self._parser.can_read_destructive()
.tox/py/lib/python3.11/site-packages/redis/asyncio/connection.py:250: in can_read_destructive
return await self._stream.read(1)
/usr/lib/python3.11/asyncio/streams.py:689: in read
await self._wait_for_data('read')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <StreamReader transport=<_SelectorSocketTransport fd=15 read=polling write=<idle, bufsize=0>>>
func_name = 'read'
async def _wait_for_data(self, func_name):
"""Wait until feed_data() or feed_eof() is called.
If stream was paused, automatically resume it.
"""
# StreamReader uses a future to link the protocol feed_data() method
# to a read coroutine. Running two read coroutines at the same time
# would have an unexpected behaviour. It would not possible to know
# which coroutine would get the next data.
if self._waiter is not None:
raise RuntimeError(
f'{func_name}() called while another coroutine is '
f'already waiting for incoming data')
assert not self._eof, '_wait_for_data after EOF'
# Waiting for data while paused will make deadlock, so prevent it.
# This is essential for readexactly(n) for case when n > self._limit.
if self._paused:
self._paused = False
self._transport.resume_reading()
self._waiter = self._loop.create_future()
try:
> await self._waiter
E asyncio.exceptions.CancelledError
/usr/lib/python3.11/asyncio/streams.py:522: CancelledError
GitHub Actions failure log: https://github.com/lsst-sqre/gafaelfawr/actions/runs/4473916874/jobs/7861846517
Reverting to redis 4.5.1 makes the problem disappear again, so it appears to be triggered by some change in 4.5.2.
Here's the shortest test case that I've come up with. It depends on FastAPI and Starlette but none of my code base. My guess is that something is happening during addition of the middleware that's breaking something about the asyncio work inside redis, but I'm not sure what that could be. Oddly, adding one of the standard Starlette middleware classes does not trigger this problem, only a custom middleware class derived from BaseHTTPMiddleware.
from collections.abc import Callable, Awaitable
import redis.asyncio
from fastapi import FastAPI, APIRouter, Request, Response
from httpx import AsyncClient
from starlette.middleware.base import BaseHTTPMiddleware
class DummyMiddleware(BaseHTTPMiddleware):
async def dispatch(
self,
request: Request,
call_next: Callable[[Request], Awaitable[Response]],
) -> Response:
return await call_next(request)
@pytest.mark.asyncio
async def test_redis() -> None:
app = FastAPI()
app.add_middleware(DummyMiddleware)
@app.get("/")
async def index() -> dict[str, str]:
return {}
async with AsyncClient(app=app, base_url="https://example.com/") as client:
await client.get("/")
pool = redis.asyncio.from_url("redis://localhost:6379/0")
await pool.set("key", "value")(If I had to guess, it would be the switch to asyncio.timeout, but I can't find evidence to support that. I know asyncio.timeout does raise CancelledError and then converts it to TimeoutError, but I couldn't find any place where that machinery shouldn't work correctly. This error happens immediately, so it doesn't seem to be a timeout and therefore I may be barking up the wrong tree.)
Activity
rra commentedon Mar 21, 2023
Ah, Starlette's
BaseHTTPMiddlewarecallsanyio.create_task_group(). I suspect that is related somehow, although I'm still not seeing exactly how. https://github.com/encode/starlette/blob/master/starlette/middleware/base.py#L39Avoid redis 4.5.2 for now
sileht commentedon Mar 22, 2023
I guess the CancellationError is raised here, where it should be an asyncio.Timeout.

sileht commentedon Mar 22, 2023
asyncio check that
asyncio.current_task().uncancel() == 0before converting CancelledError into TimeoutError, so somehow this value is unexpected, not 0.sileht commentedon Mar 22, 2023
The thing is that this bug was always there, but with the legacy async_timeout lib, the bug was hidden because of this line:
https://github.com/aio-libs/async-timeout/blob/master/async_timeout/__init__.py#L210
All CancelledError was converted into TimeoutError even if the cancellation source was not the timeout.
sileht commentedon Mar 22, 2023
Looks related to agronholm/anyio#374
curtiscook commentedon Mar 22, 2023
FWIW: I'm experiencing the same issue. I've noticed that the connection does work, but seems to break if you await a function that's not Redis. We were able to reproduce this on the command line w/o starlette (even though we are using FastAPI)
given
This function will work fine:
This will break:
Might be related that bar uses HTTPX
sileht commentedon Mar 22, 2023
Both starlette and httpx rely on anyio
rra commentedon Mar 22, 2023
Yeah, this is looking to me like an anyio bug caused by use of task groups, presumably related to this admonition in the Python library documentation:
I don't completely understand the mechanism yet, but enough keywords are lining up that this looks like it. @curtiscook's example is interesting, though, since I can't find a call to
create_task_groupin httpx or httpcore. Maybe it's somewhere else in httpx's dependencies, or maybe thatbarfunction uses anyio?rra commentedon Mar 22, 2023
BTW, for those who are specifically having this issue with Starlette middleware, it looks like Starlette has rewritten their built-in middleware to avoid the use of
BaseHTTPMiddlewareat the cost of having to interact at a lower level with the request and response. That avoids creating a task group, and I wonder if that would make the problem go away. I'm considering rewriting our middleware similarly just as an experiment.decaz commentedon Mar 23, 2023
Looks like version 4.5.3 release didn't resolve the issue.
bellini666 commentedon Mar 23, 2023
The same here! I have an async function that will sometimes call an external resource using
httpxto validate the input. Everything works fine whenhttpxis not called, but when it is called then I getCancelledError.This started happening after upgrading to 4.5.2/4.5.3 (not sure which one exactly since we upgraded from 4.5.1 directly)
ice1x commentedon Mar 23, 2023
Seems like blocker for Redis isn't it?
sileht commentedon Mar 23, 2023
This bug was always there:
bellini666 commentedon Mar 23, 2023
But why does
CancelledErrorgets raised when doing some other async stuff in the same task? E.g. thehttpxissue mentioned by me here and by @curtiscook hereI don't understand why
CancelledErroris being raised when calling a redis api just because I calledhttxbefore it (if I don't callhttpx,CancelledErroris not raised, but when calling,CancelledError` will be raised every time)I worked around this error by wrapping my call to redis in an
asyncio.shield, and although it should be harmless, it really looks like something wrong is happening here.I'll try to create a minimum reproducible demo
12 remaining items
pinheiro-bruno commentedon Jun 5, 2023
I'm facing the exactly same issue on version 4.5.5 using python 3.11.2
The issue started after anyio version was bumped from 3.6.2 to 3.7.0. The suggested solution of updating python version to 3.11.3 did the trick for me.
Mark90 commentedon Jun 9, 2023
For me the issue also started to occur after anyio was bumped to 3.7.0.
Posting here because I had a slightly different version configuration;
Upgrading to
redis==4.5.5resolved it for me.Bump redis to >=4.5.5 because of redis/redis-py#2633
Bump redis to >=4.5.5 because of redis/redis-py#2633
Bump redis to >=4.5.5 because of redis/redis-py#2633 (#286)
HacKanCuBa commentedon Jun 22, 2023
With Python 3.11.2, this issue still persists, even using redis==4.5.5 (with hiredis==2.2.3)
Edit: I confirm it all works when upgrading to Python 3.11.4 π―
Merge master to 5.0 (#2827)
bsnacks000 commentedon Jul 12, 2023
We had the same problem... resolved with 3.11.4 bump from 3.11.2... thanks for figuring that out ... π
Merge 5.0 to master (#2849)
hendrixmar commentedon Jul 18, 2023
Thanks for the help it worked upgrading to 3.11.4
Naltharial commentedon Apr 1, 2024
@dvora-h Apologies for resurrecting an old issue, but it seems like it's relevant again. I'm seeing this happen on 3.11.8 with AnyIO 4, PyRedis 5 using @bellini666 's test script.
it looks like HTTPx is closing the event loop behind it, no matter if Redis is still trying to use it. Is this related to the same issue? Notably, this only occurs under Ubuntu, I can't reproduce it on Windows, so my debugging capacity is limited.
It's resolved (obviously) by using the sync Redis API instead of the async one, but upgrading the libraries didn't work.
Naltharial commentedon Apr 2, 2024
The event loop closing might be a red herring, this is the output on Py3.11.8 / Windows:
The
b'2'comes through and noCancelledError, but something is still trying to access the event loop from Redis even past the end. The usual culprit for Linux/Windows async inconsistencies isuvloop, but its presence didn't seem to change anything.Mark90 commentedon Apr 2, 2024
I'm not hitting this issue right now, but I got curious and I can also reproduce it with the latest test script from @bellini666. I tested a few more scenarios.
Note that I'm running Windows, but this is tested with Ubuntu 20.04 in Windows Subsystem Linux (basically a VM).
For each python version I tested with, I started with redis-py 5.0.3 and downgraded through 5.0.2 -> 5.0.1 -> 5.0.0 -> 4.6.0 to see if/when behavior changed. The envs were clean and I did not install
trionoruvloop.conn.get()after httpxCE = asyncio.exceptions.CancelledError (same as @Naltharial)
RE = RuntimeError: Event loop is closed (Exception ignored in AbstractConnection.del, same as @Naltharial)
Some other observations:
conn.get()into thewith AsyncClientblock did not trigger the issue as long as it comes before theclient.get()call. Right below theclient.get()call (within the context) is where the problem startshttpxout foraiohttpand this does not trigger the issue, so somehow httpx and redis-py bite (could still be a be a timing-related coincidence)Because on 3.11.8/3.12.1 the behavior seems to start from redis-py 5.0.2 onwards, perhaps #2999 could be related? (edit: except that does not explain the combination 3.11.1 and redis-py 5.0.0..)
yinziyan1206 commentedon Apr 8, 2024
I have the same problem. And here is the example
PS: It works well for
redis-py<5 oranyio<4.