Repository navigation
Pipeline sync() reports read failures to the caller - #4761
Conversation
The worker that hits a JedisConnectionException called remove() on the calling thread's iterator over pipelinedResponses, which drops the entry that thread last returned rather than the failed node, leaving the failed node with a queue but no connection. Record failed nodes on the workers and remove them from both maps on the calling thread after the reads.
90e4ba0 to
ee765ee
Compare
|
Thanks for the PR. Let's settle the intended behaviour on the issue first — see my questions on #4760 — before reviewing the implementation here. |
|
Answered on #4760. I'll leave this as it is until the behaviour is agreed there, then rework it to match. |
MultiNodePipelineBase.sync() only logged a node's JedisConnectionException, so sync() returned normally and the node's responses were left unset: a caller that did not read every Response never learned the batch was partly lost, and one that did got IllegalStateException from Response.get(). Every node is still read to completion, so healthy nodes keep setting their responses. A failed node's responses are now completed with its connection error, Response.get() rethrows it, and sync() rethrows the first failure in pipeline order with any further node failures suppressed. close() surfaces it the way Pipeline.close() does. Single-node Pipeline.sync() and syncAndReturnAll() propagated the error but left their buffered responses unset; they now complete them the same way and drop them, so a later sync() does not read those replies again.
|
Reworked to the contract from #4760, all four points in this PR. Failed nodes are recorded by their workers and dropped from both maps on the calling thread after the reads, and each of their pending responses is completed with the On your point about the single-node pipeline: That needed somewhere to put the error: Tests cover the deferred cleanup, the response completion, the inline single-node read path, two nodes failing in one One unrelated note: the |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 3ccfcac. Configure here.
|
CI on the reworked branch: everything green except |
A keyless command is pipelined under a null node key, which the ConcurrentHashMap tracking failures rejected: a failed read then threw NullPointerException inside the worker, skipping the connection close and the failure report, and on the inline path left syncing stuck so later sync() and close() calls did nothing. Failures other than a lost connection had the same silent outcome: they died in the executor and sync() returned normally with the node's responses unset. Every throwable is now recorded and rethrown unchanged from sync(). syncing is reset in a finally. Response.completeExceptionally keeps Jedis exceptions as they are and wraps anything else in a JedisException, so get() never rethrows an Error; it has no effect on a response already completed. The protected exception field keeps its type for compatibility; read failures are held separately.
|
Pushed fc403da on top of your branch with a few changes:
Covered by new cases in |

MultiNodePipelineBase.sync()only logged a node'sJedisConnectionException:sync()returned normally with that node's responses left unset, and the cleanup ran on the worker thread through the calling thread's iterator, so it removed whichever node that thread had last visited instead of the failed one. A caller that did not read everyResponsenever learned the batch was partly lost; one that did gotIllegalStateExceptionfromResponse.get(), and the next command routed to the failed node threwNullPointerExceptioninappendCommand.Implements the contract agreed in #4760:
Response.get()rethrows it and the commands left in doubt can be identified;pipelinedResponsesandconnectionson the calling thread, so the pipeline stays consistent and the next command for such a node takes a fresh connection while the other nodes keep theirs;sync()rethrows the first failed node'sJedisConnectionExceptionin pipeline order, with further node failures attached as suppressed.close()surfaces it the wayPipeline.close()does.Single-node
Pipeline.sync()/syncAndReturnAll()already propagated the error but left their buffered responses unset. They now complete them the same way and drop them, so a latersync()does not try to read those replies again, andResponse.get()behaves identically on both pipelines.Responsegained a package-privatesetFailure(JedisException)for this; a connection failure means no reply was read, so it is kept separate from theJedisDataExceptioncarried by an error reply. No public API change.Unit tests cover the deferred cleanup, the response completion, both the worker and the inline single-node read path, two nodes failing in one
sync(), and the single-node pipeline. Behaviour change in 4 is in the 8.1.0 release notes, and the failure contract is documented under Pipelining indocs/advanced-usage.md.Closes #4760.
Note
Medium Risk
Behavior change for
ClusterPipeline.sync()/close()callers that assumed success after a partial read failure; exception handling and retry logic may need updates.Overview
Pipeline
sync()/close()now surface reply-read failures instead of failing silently or leaving responses in an ambiguous state.For cluster pipelines, a connection loss during
sync()used to be logged only:sync()returned normally,Response.get()on the failed node raisedIllegalStateException, and follow-up commands could hitNullPointerException. The implementation now still reads every other node, marks each lost reply viaResponse.completeExceptionally(), drops only the failed nodes (so reuse can open fresh connections), and rethrows the first failure in pipeline order with additional node failures as suppressed exceptions.Single-node
Pipelinealready threw fromsync(), but bufferedResponses stayed “not synced”; they are completed with the same failure, cleared from the pipeline buffer, andget()behaves like the cluster path.Responsegains package-private completion for read failures (distinct from Redis error replies). 8.1.0 release notes and advanced-usage document the contract: callers must handlesync()/close()exceptions and must not blindly replay commands that may already have run on the server; cluster pipelines do not refresh the slot cache after failover.Reviewed by Cursor Bugbot for commit 3021ec5. Bugbot is set up for automated code reviews on this repo. Configure here.