Repository navigation
client: avoid finishing a stream twice when stream creation fails - #9191
Conversation
|
For transparency: I used an LLM, namely Opus 4.8 heavily on reproducing and fixing this issue. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9191 +/- ##
==========================================
+ Coverage 83.09% 83.27% +0.18%
==========================================
Files 419 421 +2
Lines 33858 34086 +228
==========================================
+ Hits 28134 28385 +251
+ Misses 4291 4272 -19
+ Partials 1433 1429 -4
🚀 New features to boost your workflow:
|
Omni could stop responding to all API requests after a period without API activity, leaving the web UI on a blank page while static assets still loaded, and required a restart to recover. The connections that carry the API gateway and internal proxy traffic run in-process and relied on gRPC idle mode, which tears down and later re-establishes a connection that has been inactive. A bug in gRPC could leave such a connection permanently stuck once it went idle, so every subsequent request blocked forever. Idle mode provides no benefit for these connections because they are in-process and live for the whole lifetime of the process, so disable it. This avoids the stuck connection and also removes the latency of re-establishing it on the first request after a quiet period. A fix for the underlying gRPC issue has been submitted upstream in grpc/grpc-go#9191, but disabling idle mode here is the correct configuration regardless. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
19d6e56 to
c33f3be
Compare
|
@Pranjali-2501 gentle nudge - can you or another member of the project have a look into this when you get a chance? Just in case this was missed. |
|
Apologies for the delay in reviewing this. I will have this reviewed shortly. |
eshitachandwani
left a comment
There was a problem hiding this comment.
Thank you for fixing this, mostly looks good. I just have a few comments
c33f3be to
b8cf495
Compare
Thanks for the review, I addressed all your comments, amended into the same single commit. |
|
LGTM , adding @easwars for a second review. |
| // Commit (releasing any resources held for retries, e.g. the | ||
| // config selector's OnCommitted callback) but do not call | ||
| // cs.finish: no attempt was created, and newClientStream's | ||
| // deferred cleanup finishes the stream on this error return. | ||
| cs.commitAttemptLocked() |
There was a problem hiding this comment.
I don't think this is the correct fix. Only calling commitAttemptLocked from here and not calling cs.finish means that other cleanup things that happen in the latter are skipped for this stream.
For example, a binary log entry that either contains a "Cancel" or "ServerTrailer" will be skipped. See:
Line 1108 in 7d87045
cs.finish is skipped as well.
The crux of the problem is two things:
- The
OnFinishcall option does not stick to its contract of executing the callback provided to it not more than once.- Ideally, the
OnFinishcall option should wrap the caller provider callback in async.OnceFuncto guarantee this
- Ideally, the
- Doing the above will guarantee that all
OnFinishcall options are only ever invoked once, but we still have the issue ofendOfClientStreambeing invoked more than once ifcs.finishis called fromwithRetryduring stream creation failure (which results inendOfClientStreambeing called once) and the deferred block innewClientStreamrunning it the second time because of stream creation failure.
We should guarantee that endOfClientStream only ever runs once as well.
There was a problem hiding this comment.
I reworked it as suggested: cs.finish is called from withRetry again, and it now skips endOfClientStream when cs.attempt is nil.
A nil attempt only happens when stream creation failed, and on that path the deferred cleanup in newClientStream already calls endOfClientStream, so it runs exactly once either way.
I also tried wrapping the callback in a sync.Once in OnFinish, but later dropped it. The same CallOption value can be reused across RPCs (e.g. via WithDefaultCallOptions), and the once would swallow the callback on every call after the first. Since the callbacks are only invoked from endOfClientStream, that guarantee already covers them. This approach is also simpler.
Pls let me know if I miss anything.
When stream creation failed after the attempt could not be created (for example because the RPC context was already canceled while the channel was exiting idle mode), the end-of-stream cleanup ran twice: once via `cs.finish` called from `withRetry`, and once via the deferred cleanup in `newClientStream`. This invoked every `OnFinish` callback twice for a single call, including the idleness manager's `OnCallEnd`, which corrupted the active-call counter and left the channel permanently stuck in IDLE. All subsequent RPCs on the channel then failed with "context deadline exceeded while waiting for connections to become ready". The call was also double-counted in channelz stats. Skip `endOfClientStream` in `cs.finish` when no attempt was ever created: in that case stream creation has failed and the deferred cleanup in `newClientStream` performs it instead. This guarantees the cleanup, including all `OnFinish` callbacks, runs at most once per call. RELEASE NOTES: * client: Fix a bug where a ClientConn could get permanently stuck in IDLE after an RPC was canceled while the channel was exiting idle mode
b8cf495 to
31f923f
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request prevents duplicate cleanup calls when an RPC fails during stream creation before any attempt is started. In stream.go, the finish method is updated to only call endOfClientStream if an attempt was actually created. A new end-to-end test, TestChannelIdleness_RPCFailingBeforeAttempt_FinishesOnce, is added to verify this behavior. There are no review comments to address, and I have no feedback to provide.
|
@utkuozdemir Thanks for your contribution |
easwars correctly flagged that wrapping the callback in sync.Once breaks WithDefaultCallOptions: that dial option stores and reuses the same CallOption value across every RPC on the connection (call.go:36, stream.go:245), so a Once baked into the constructor only ever fires for the first RPC made on the connection, never again after that. Digging further, callInfo.onFinish (the field before() appends to) is never read anywhere in the codebase. The actual invocation path is endOfClientStream calling o.OnFinish(err) directly on the CallOption from the original opts slice, guarded to run exactly once per RPC by clientStream.finish's cs.finished check (already the fix for grpc#9191). The tests this PR added only exercised the unused callInfo.onFinish field, so they passed without validating real behavior. With that guard already in place and no reproduction against the real invocation path, there is no live bug to fix here beyond what grpc#9191 already covers. Reverting to plain field assignment, matching master. Assisted-by: Claude Sonnet 5 Signed-off-by: zanarelli <zanarelli.dev@gmail.com>
In
clientStream.withRetry, whennewAttemptLockedfails before an attempt is created,withRetrycalledcs.finishand then returned the error.newClientStreamalso runsendOfClientStreamfrom its deferred cleanup on an error return, so the stream was finished twice and everygrpc.OnFinishcallback ran twice for a single RPC, breaking its documented "called only once" contract.One of those callbacks is the idleness manager's
OnCallEnd. The extraOnCallEndpushes the manager's int32activeCallsCountone below its-math.MaxInt32idle sentinel and corrupts the counter. After that the channel can get permanently stuck in IDLE, and later RPCs fail with "context deadline exceeded while waiting for connections to become ready" even though the backend is reachable. It happens when RPCs are canceled around the idle-timeout boundary.On this path no attempt is created, so there is nothing to finish: the context is canceled by
newClientStreamWithParams's deferred cleanup, andnewClientStream's deferredendOfClientStreamfinishes the stream exactly once. Replace thecs.finishcall withcs.commitAttemptLockedso the stream is still committed, releasing resources held for retries such as the config selector'sOnCommittedcallback, without being finished a second time. This mirrors theretryLockedgive-up path.Add a regression test that fails before this change and passes after it.
RELEASE NOTES: