Repository navigation
fix(asyncio): raise ConnectionError when writing to a peer-closed connection - #4288
Conversation
β¦nection
On CPython 3.12, writing to a transport whose connection was already lost
raises TypeError ('NoneType' object is not callable) because writelines
lacks the _conn_lost guard and _call_connection_lost clears _write_ready.
The TypeError is not translated to ConnectionError, so it bypasses the
retry machinery and surfaces to callers.
Guard the write in send_packed_command: if the writer is missing or its
transport is closing, fail fast with ConnectionError so the existing
disconnect + retry path handles it on every Python version.
Fixes redis#4287
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
petyaslavova
left a comment
There was a problem hiding this comment.
Thanks for the thorough investigation - the diagnosis is right, and raising ConnectionError itself rather than a subclass is the correct call given that ERRORS_ALLOW_RETRY matches on exact types.
Two things before we merge. First, the guard re-reads self._writer afterwards, and asyncio.wait_for suspends between the check and writelines() on the socket_timeout path, so a concurrent disconnect() still surfaces AttributeError: 'NoneType' object has no attribute 'writelines' and still bypasses retry. Please bind the writer to a local once, guard the local, and write through it - that also covers the variant reported in #3546 and in open PR #4217.
Second, the 3.12-only framing is not accurate: the _SelectorSocketTransport.writelines override was added in 3.12 and still calls _write_ready() unguarded in 3.13 and 3.14. I measured TypeError on 3.12 and 3.14 and AttributeError: 'NoneType' object has no attribute '_add_writer' on 3.13, with 3.11 correctly raising ConnectionError. Your guard is version-agnostic, so only the code comment needs updating - but it makes the case for handling AttributeError too.
β¦sing Address review: bind the writer to a local inside _send_packed_command, which now serves both write branches, so the guard runs inside the wait_for task after any suspension and there is no await between the guard and writelines. A concurrent disconnect() can no longer surface AttributeError from a cleared self._writer. The CPython bug is gh-136234, fixed in 3.13.10+ and 3.14.1+ and never backported to 3.12; 3.13.0-3.13.9 leak AttributeError from the transport internals and 3.12/3.14.0 leak TypeError. Since the guard cannot stop errors raised inside the transport, TypeError/AttributeError from the write are translated to ConnectionError (cause preserved) only when the transport reports closing; programming errors on a healthy transport re-raise unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the careful review β both points were right, and both are addressed in 1ea25fc. Writer race. Version framing. You were right and I've corrected the comment. I reproduced your measurements against a real transport with a genuine peer RST: AttributeError handling. Since the guard cannot stop errors raised inside the transport (the 3.13.x |
fallenmi
left a comment
There was a problem hiding this comment.
Reviewed exact head 1ea25fc2f50f95d58c9846e05cc917e3f7faea85 against exact base afe6523ec5e07b49fe15e1748c049cc2ab2644a6 and current main c78ccfcbd05d129993bb7bc61b1ab3c3993e0b23.
The pre-write guard and single local-writer path address both failure classes without widening exception translation:
- An exact-base harness with a closing transport returned normally and called
writelines; the exact head raises the concreteredis.exceptions.ConnectionErrorbefore writing. - A 10-case exact-head matrix covered closed-before-write, transport-side
TypeError/AttributeErrorwhile closing with the cause preserved, a genuineTypeErroron an open transport with identity preserved, and_writercleared after the health check, with and withoutsocket_timeout. - The six focused upstream cases pass on the exact head and after a clean merge into current
main. Ruff check/format, compileall, and diff check passed; live CI is 368 successful with 0 failing or pending.
I also ran the full test file: 61 passed and 1 skipped; the remaining 11 failures and 8 setup errors all require the unavailable local Redis endpoint, so I did not count that as a green full-suite result.
Approved.
Review drafted and validated with OpenAI Codex.
petyaslavova
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround - both points are addressed. _send_packed_command is now the single write path and the writer is bound to a local inside the wait_for task with no suspension before writelines(), so the concurrent-disconnect() variant from #3546 is covered too. Gating the TypeError/AttributeError translation on transport.is_closing() keeps genuine encoding errors intact, and raising exact redis.exceptions.ConnectionError is what ERRORS_ALLOW_RETRY needs, since the cluster paths match on type(e).
Merging this after a green CI. Two things I am tracking separately, no action needed here: is_connected still reports a closed-transport connection as healthy, so the pool keeps handing one out and the first use costs a failed command plus a retry, and the read path has no equivalent guard.
Pull Request check-list
Description of change
Fixes #4287, #3546.
When the server (or an intermediary) closes a pooled connection while it is idle,
AbstractConnection.is_connectedstill reports the connection as healthy (it only checks that_reader/_writerare non-None), so the dead connection is reused. On CPython 3.12 the subsequent write then raisesTypeError: 'NoneType' object is not callableinstead ofConnectionError: 3.12's_SelectorSocketTransport.writelinesis missing the_conn_lostguard thatwritehas, and_call_connection_losthas already cleared the transport's_write_readycallback. TheTypeErroris not translated toConnectionError, so it bypassesRetryandRedisCluster.ERRORS_ALLOW_RETRYand surfaces to application code. CPython fixedwritelinesin 3.13+, but 3.12 is security-only, so the fix will not be backported.This change guards the write in
send_packed_command: if the writer is missing or its transport is closing, raiseConnectionError(the exact class, sinceERRORS_ALLOW_RETRYmatching uses exact types) before touching the transport. The guard sits inside the existingtryblock, so theexcept BaseExceptionclause disconnects the dead connection before re-raising, and the existing retry machinery takes over β on every Python version, covering both thesocket_timeoutpath and the inlinewritelinespath.Regression tests (mock-based, no live server needed):
send_packed_commandraise exactlyConnectionError, without writing, and the connection is disconnected;writelines/drainon both write paths (socket_timeoutset and unset).Verified against a real asyncio transport as well (peer sends a genuine RST via
SO_LINGER(1, 0)): before this change the write raises theTypeErrorabove on CPython 3.12; with it,ConnectionError.π€ Generated with Claude Code
Note
Medium Risk
Touches the hot path for every async Redis command write and error classification; behavior change is intentional (TypeError β ConnectionError on dead sockets) but could affect callers that relied on the old exception type.
Overview
Fixes #4287 by hardening async command writes when a pooled connection was closed by the server but still looks connected.
_send_packed_commandnow refuses writes if the stream writer is missing ortransport.is_closing()is true, raisingConnectionError(exact type, so cluster/retry allowlists still match). It centralizeswritelines/drainfor both thesocket_timeoutand non-timeout paths. On older CPython builds where a dead transport can surfaceTypeErrororAttributeErrorfromwritelines, those are mapped toConnectionErrorwhen the transport is closing; unrelated errors are re-raised. Existingsend_packed_commanderror handling still disconnects on failure.Adds mock-based tests for closed transport, successful writes, translated transport errors, and preserved
TypeErroron an open transport.Reviewed by Cursor Bugbot for commit 0cd03be. Bugbot is set up for automated code reviews on this repo. Configure here.