Skip to content

Pipeline sync() reports read failures to the caller - #4761

Merged
ggivo merged 5 commits into
redis:masterfrom
insaf021:pipeline-sync-failed-node
Oct 19, 2026
Merged

ggivo merged 5 commits into
redis:masterfrom
insaf021:pipeline-sync-failed-node

Conversation

@insaf021

@insaf021 insaf021 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

MultiNodePipelineBase.sync() only logged a node's JedisConnectionException: sync() returned normally with that node's responses left unset, and the cleanup ran on the worker thread through the calling thread's iterator, so it removed whichever node that thread had last visited instead of the failed one. A caller that did not read every Response never learned the batch was partly lost; one that did got IllegalStateException from Response.get(), and the next command routed to the failed node threw NullPointerException in appendCommand.

Implements the contract agreed in #4760:

  1. every node is still read to completion, so the healthy nodes' responses are set;
  2. each response of a failed node is completed with the connection error, so Response.get() rethrows it and the commands left in doubt can be identified;
  3. the failed nodes are dropped from pipelinedResponses and connections on the calling thread, so the pipeline stays consistent and the next command for such a node takes a fresh connection while the other nodes keep theirs;
  4. sync() rethrows the first failed node's JedisConnectionException in pipeline order, with further node failures attached as suppressed. close() surfaces it the way Pipeline.close() does.

Single-node Pipeline.sync()/syncAndReturnAll() already propagated the error but left their buffered responses unset. They now complete them the same way and drop them, so a later sync() does not try to read those replies again, and Response.get() behaves identically on both pipelines.

Response gained a package-private setFailure(JedisException) for this; a connection failure means no reply was read, so it is kept separate from the JedisDataException carried by an error reply. No public API change.

Unit tests cover the deferred cleanup, the response completion, both the worker and the inline single-node read path, two nodes failing in one sync(), and the single-node pipeline. Behaviour change in 4 is in the 8.1.0 release notes, and the failure contract is documented under Pipelining in docs/advanced-usage.md.

Closes #4760.


Note

Medium Risk
Behavior change for ClusterPipeline.sync()/close() callers that assumed success after a partial read failure; exception handling and retry logic may need updates.

Overview
Pipeline sync()/close() now surface reply-read failures instead of failing silently or leaving responses in an ambiguous state.

For cluster pipelines, a connection loss during sync() used to be logged only: sync() returned normally, Response.get() on the failed node raised IllegalStateException, and follow-up commands could hit NullPointerException. The implementation now still reads every other node, marks each lost reply via Response.completeExceptionally(), drops only the failed nodes (so reuse can open fresh connections), and rethrows the first failure in pipeline order with additional node failures as suppressed exceptions.

Single-node Pipeline already threw from sync(), but buffered Responses stayed “not synced”; they are completed with the same failure, cleared from the pipeline buffer, and get() behaves like the cluster path.

Response gains package-private completion for read failures (distinct from Redis error replies). 8.1.0 release notes and advanced-usage document the contract: callers must handle sync()/close() exceptions and must not blindly replay commands that may already have run on the server; cluster pipelines do not refresh the slot cache after failover.

Reviewed by Cursor Bugbot for commit 3021ec5. Bugbot is set up for automated code reviews on this repo. Configure here.

The worker that hits a JedisConnectionException called remove() on the calling thread's iterator over pipelinedResponses, which drops the entry that thread last returned rather than the failed node, leaving the failed node with a queue but no connection. Record failed nodes on the workers and remove them from both maps on the calling thread after the reads.
@insaf021
insaf021 force-pushed the pipeline-sync-failed-node branch from 90e4ba0 to ee765ee Compare September 30, 2026 18:32
@ggivo

ggivo commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR. Let's settle the intended behaviour on the issue first — see my questions on #4760 — before reviewing the implementation here.

@insaf021

insaf021 commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Answered on #4760. I'll leave this as it is until the behaviour is agreed there, then rework it to match.

MultiNodePipelineBase.sync() only logged a node's JedisConnectionException,
so sync() returned normally and the node's responses were left unset: a caller
that did not read every Response never learned the batch was partly lost, and
one that did got IllegalStateException from Response.get().

Every node is still read to completion, so healthy nodes keep setting their
responses. A failed node's responses are now completed with its connection
error, Response.get() rethrows it, and sync() rethrows the first failure in
pipeline order with any further node failures suppressed. close() surfaces it
the way Pipeline.close() does.

Single-node Pipeline.sync() and syncAndReturnAll() propagated the error but
left their buffered responses unset; they now complete them the same way and
drop them, so a later sync() does not read those replies again.
@insaf021 insaf021 changed the title defer failed-node cleanup in MultiNodePipelineBase.sync report pipeline sync failures to the caller Oct 8, 2026
@insaf021

insaf021 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Reworked to the contract from #4760, all four points in this PR.

Failed nodes are recorded by their workers and dropped from both maps on the calling thread after the reads, and each of their pending responses is completed with the JedisConnectionException - getMany discards the replies it had already read, so none of them was set. sync() then rethrows the first failure in pipeline order (not completion order, so it's deterministic) with any further node failures suppressed; close() already surfaces whatever sync() throws. Healthy nodes are unaffected and still set their responses.

On your point about the single-node pipeline: Pipeline.sync() and syncAndReturnAll() now complete their buffered responses the same way before propagating, and drop them so a later sync() doesn't try to read those replies off a dead connection again. Response.get() behaves identically on both pipelines now.

That needed somewhere to put the error: Response.exception is a JedisDataException from an error reply, and a connection failure means no reply was read at all, so I added a package-private setFailure(JedisException) rather than widening that field. No public API change.

Tests cover the deferred cleanup, the response completion, the inline single-node read path, two nodes failing in one sync(), and the single-node pipeline. Release note for point 4 is in 8.1.0 and the contract is documented under Pipelining in docs/advanced-usage.md. Full unit suite green on JDK 11 (3468 run, 0 failures).

One unrelated note: the doctests check is red on QueryAggExample.run:241 expected: <5> but was: <4>, which also fails intermittently on master and doesn't touch this path.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 3ccfcac. Configure here.

@insaf021

insaf021 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

CI on the reworked branch: everything green except doctests, which now fails on QueryRangeExample.run:239 expected: <3> but was: <2> and HomeJsonExample.run:226 IndexOutOfBounds rather than the QueryAggExample I mentioned above. Different search examples each run, all of them index-visibility races on FT.SEARCH with no pipeline in the path, and the job fails intermittently on master too.

ggivo added 2 commits October 19, 2026 11:02
A keyless command is pipelined under a null node key, which the
ConcurrentHashMap tracking failures rejected: a failed read then threw
NullPointerException inside the worker, skipping the connection close
and the failure report, and on the inline path left syncing stuck so
later sync() and close() calls did nothing.

Failures other than a lost connection had the same silent outcome: they
died in the executor and sync() returned normally with the node's
responses unset. Every throwable is now recorded and rethrown unchanged
from sync(). syncing is reset in a finally.

Response.completeExceptionally keeps Jedis exceptions as they are and
wraps anything else in a JedisException, so get() never rethrows an
Error; it has no effect on a response already completed. The protected
exception field keeps its type for compatibility; read failures are held
separately.
@ggivo

ggivo commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Pushed fc403da on top of your branch with a few changes:

  • Keyless commands are pipelined under a null node key, which the ConcurrentHashMap rejected (Bugbot's finding): a failed read threw NPE in the worker and could leave syncing stuck. failures is now a synchronized HashMap, and syncing is reset in a finally.
  • The worker catches Throwable, not just JedisConnectionException, so other failures no longer die in the executor while sync() returns normally. sync() rethrows the original unchanged; the first failure in pipeline order wins, the rest are suppressed.
  • Response.setFailure is now completeExceptionally(Throwable): Jedis exceptions are kept as is, anything else is wrapped in a JedisException, so get() never rethrows an Error. It has no effect on a response that is already completed. The protected exception field keeps its type for compatibility; merging the two fields is tracked in Response cleanup for 9.0: remove setDependency, make the exception field private #4793.
  • Release note and advanced-usage.md tightened, including that a single-node pipeline's connection is broken after a failed sync().

Covered by new cases in MultiNodePipelineSyncTest and PipelineSyncFailureTest, plus ResponseTest.
Let me know if anything looks off.

@ggivo ggivo changed the title report pipeline sync failures to the caller Pipeline sync() reports read failures to the caller Oct 19, 2026
@ggivo
ggivo merged commit 0273255 into redis:master Oct 19, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ClusterPipeline.sync() hides node failures and leaves the pipeline inconsistent

2 participants