Skip to content

fix(asyncio): raise ConnectionError when writing to a peer-closed connection - #4288

Merged
petyaslavova merged 3 commits into
redis:masterfrom
abersheeran:fix-asyncio-send-on-closed-transport
Aug 28, 2026
Merged

petyaslavova merged 3 commits into
redis:masterfrom
abersheeran:fix-asyncio-send-on-closed-transport

Conversation

@abersheeran

@abersheeran abersheeran commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request check-list

  • Do tests and lints pass with this change?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?

Description of change

Fixes #4287, #3546.

When the server (or an intermediary) closes a pooled connection while it is idle, AbstractConnection.is_connected still reports the connection as healthy (it only checks that _reader/_writer are non-None), so the dead connection is reused. On CPython 3.12 the subsequent write then raises TypeError: 'NoneType' object is not callable instead of ConnectionError: 3.12's _SelectorSocketTransport.writelines is missing the _conn_lost guard that write has, and _call_connection_lost has already cleared the transport's _write_ready callback. The TypeError is not translated to ConnectionError, so it bypasses Retry and RedisCluster.ERRORS_ALLOW_RETRY and surfaces to application code. CPython fixed writelines in 3.13+, but 3.12 is security-only, so the fix will not be backported.

This change guards the write in send_packed_command: if the writer is missing or its transport is closing, raise ConnectionError (the exact class, since ERRORS_ALLOW_RETRY matching uses exact types) before touching the transport. The guard sits inside the existing try block, so the except BaseException clause disconnects the dead connection before re-raising, and the existing retry machinery takes over β€” on every Python version, covering both the socket_timeout path and the inline writelines path.

Regression tests (mock-based, no live server needed):

  • a closed transport makes send_packed_command raise exactly ConnectionError, without writing, and the connection is disconnected;
  • a healthy transport still writes through writelines/drain on both write paths (socket_timeout set and unset).

Verified against a real asyncio transport as well (peer sends a genuine RST via SO_LINGER(1, 0)): before this change the write raises the TypeError above on CPython 3.12; with it, ConnectionError.

πŸ€– Generated with Claude Code


Note

Medium Risk
Touches the hot path for every async Redis command write and error classification; behavior change is intentional (TypeError β†’ ConnectionError on dead sockets) but could affect callers that relied on the old exception type.

Overview
Fixes #4287 by hardening async command writes when a pooled connection was closed by the server but still looks connected.

_send_packed_command now refuses writes if the stream writer is missing or transport.is_closing() is true, raising ConnectionError (exact type, so cluster/retry allowlists still match). It centralizes writelines/drain for both the socket_timeout and non-timeout paths. On older CPython builds where a dead transport can surface TypeError or AttributeError from writelines, those are mapped to ConnectionError when the transport is closing; unrelated errors are re-raised. Existing send_packed_command error handling still disconnects on failure.

Adds mock-based tests for closed transport, successful writes, translated transport errors, and preserved TypeError on an open transport.

Reviewed by Cursor Bugbot for commit 0cd03be. Bugbot is set up for automated code reviews on this repo. Configure here.

…nection

On CPython 3.12, writing to a transport whose connection was already lost
raises TypeError ('NoneType' object is not callable) because writelines
lacks the _conn_lost guard and _call_connection_lost clears _write_ready.
The TypeError is not translated to ConnectionError, so it bypasses the
retry machinery and surfaces to callers.

Guard the write in send_packed_command: if the writer is missing or its
transport is closing, fail fast with ConnectionError so the existing
disconnect + retry path handles it on every Python version.

Fixes redis#4287

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@petyaslavova petyaslavova left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough investigation - the diagnosis is right, and raising ConnectionError itself rather than a subclass is the correct call given that ERRORS_ALLOW_RETRY matches on exact types.

Two things before we merge. First, the guard re-reads self._writer afterwards, and asyncio.wait_for suspends between the check and writelines() on the socket_timeout path, so a concurrent disconnect() still surfaces AttributeError: 'NoneType' object has no attribute 'writelines' and still bypasses retry. Please bind the writer to a local once, guard the local, and write through it - that also covers the variant reported in #3546 and in open PR #4217.

Second, the 3.12-only framing is not accurate: the _SelectorSocketTransport.writelines override was added in 3.12 and still calls _write_ready() unguarded in 3.13 and 3.14. I measured TypeError on 3.12 and 3.14 and AttributeError: 'NoneType' object has no attribute '_add_writer' on 3.13, with 3.11 correctly raising ConnectionError. Your guard is version-agnostic, so only the code comment needs updating - but it makes the case for handling AttributeError too.

…sing

Address review: bind the writer to a local inside _send_packed_command,
which now serves both write branches, so the guard runs inside the
wait_for task after any suspension and there is no await between the
guard and writelines. A concurrent disconnect() can no longer surface
AttributeError from a cleared self._writer.

The CPython bug is gh-136234, fixed in 3.13.10+ and 3.14.1+ and never
backported to 3.12; 3.13.0-3.13.9 leak AttributeError from the transport
internals and 3.12/3.14.0 leak TypeError. Since the guard cannot stop
errors raised inside the transport, TypeError/AttributeError from the
write are translated to ConnectionError (cause preserved) only when the
transport reports closing; programming errors on a healthy transport
re-raise unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abersheeran

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review β€” both points were right, and both are addressed in 1ea25fc.

Writer race. _send_packed_command is now the single write path (the socket_timeout=None branch delegates to it instead of writing inline), and it binds writer = self._writer once, guards the local, and writes through it. Since the bind+guard run inside the wait_for task β€” i.e. after any suspension β€” and there is no await between the guard and writelines(), a concurrent disconnect() can no longer surface AttributeError from a cleared self._writer on either branch. That covers the #3546 / #4217 variant as you suggested.

Version framing. You were right and I've corrected the comment. I reproduced your measurements against a real transport with a genuine peer RST: AttributeError: 'NoneType' object has no attribute '_add_writer' on 3.13.0, TypeError: 'NoneType' object is not callable on 3.14.0, and no leak on 3.14.6. The CPython fix is gh-136234, shipped in 3.13.10+ and 3.14.1+ and never backported to security-only 3.12 β€” so affected versions are all of 3.12, 3.13.0–3.13.9, and 3.14.0. On fixed versions writelines drops silently and the subsequent drain() raises ConnectionResetError, which the existing OSError handling already translates.

AttributeError handling. Since the guard cannot stop errors raised inside the transport (the 3.13.x AttributeError comes from transport.writelines internals, past our check), the write is now wrapped and TypeError / AttributeError are translated to ConnectionError (cause preserved via from e) β€” but only when writer.transport.is_closing(), so a genuine programming error on a healthy transport (e.g. non-bytes in command) re-raises unchanged. New tests cover both leak symptoms, the cause chain, and the healthy-transport passthrough; the full tests/test_asyncio/test_connection.py passes (75 passed, 2 skipped). I also updated the issue body of #4287 to correct the version claims.

@fallenmi fallenmi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 1ea25fc2f50f95d58c9846e05cc917e3f7faea85 against exact base afe6523ec5e07b49fe15e1748c049cc2ab2644a6 and current main c78ccfcbd05d129993bb7bc61b1ab3c3993e0b23.

The pre-write guard and single local-writer path address both failure classes without widening exception translation:

  • An exact-base harness with a closing transport returned normally and called writelines; the exact head raises the concrete redis.exceptions.ConnectionError before writing.
  • A 10-case exact-head matrix covered closed-before-write, transport-side TypeError/AttributeError while closing with the cause preserved, a genuine TypeError on an open transport with identity preserved, and _writer cleared after the health check, with and without socket_timeout.
  • The six focused upstream cases pass on the exact head and after a clean merge into current main. Ruff check/format, compileall, and diff check passed; live CI is 368 successful with 0 failing or pending.

I also ran the full test file: 61 passed and 1 skipped; the remaining 11 failures and 8 setup errors all require the unavailable local Redis endpoint, so I did not count that as a green full-suite result.

Approved.

Review drafted and validated with OpenAI Codex.

@petyaslavova petyaslavova left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround - both points are addressed. _send_packed_command is now the single write path and the writer is bound to a local inside the wait_for task with no suspension before writelines(), so the concurrent-disconnect() variant from #3546 is covered too. Gating the TypeError/AttributeError translation on transport.is_closing() keeps genuine encoding errors intact, and raising exact redis.exceptions.ConnectionError is what ERRORS_ALLOW_RETRY needs, since the cluster paths match on type(e).

Merging this after a green CI. Two things I am tracking separately, no action needed here: is_connected still reports a closed-transport connection as healthy, so the pool keeps handing one out and the first use costs a failed command plus a retry, and the read path has no equivalent guard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

asyncio: TypeError/AttributeError instead of ConnectionError when writing to a peer-closed connection (CPython 3.12, 3.13.0-3.13.9, 3.14.0)

3 participants