Skip to content

client: avoid finishing a stream twice when stream creation fails - #9191

Merged
easwars merged 1 commit into
grpc:masterfrom
utkuozdemir:fix/idle-double-finish-newattempt
Jul 17, 2026
Merged

easwars merged 1 commit into
grpc:masterfrom
utkuozdemir:fix/idle-double-finish-newattempt

Conversation

@utkuozdemir

Copy link
Copy Markdown
Contributor

In clientStream.withRetry, when newAttemptLocked fails before an attempt is created, withRetry called cs.finish and then returned the error. newClientStream also runs endOfClientStream from its deferred cleanup on an error return, so the stream was finished twice and every grpc.OnFinish callback ran twice for a single RPC, breaking its documented "called only once" contract.

One of those callbacks is the idleness manager's OnCallEnd. The extra OnCallEnd pushes the manager's int32 activeCallsCount one below its -math.MaxInt32 idle sentinel and corrupts the counter. After that the channel can get permanently stuck in IDLE, and later RPCs fail with "context deadline exceeded while waiting for connections to become ready" even though the backend is reachable. It happens when RPCs are canceled around the idle-timeout boundary.

On this path no attempt is created, so there is nothing to finish: the context is canceled by newClientStreamWithParams's deferred cleanup, and newClientStream's deferred endOfClientStream finishes the stream exactly once. Replace the cs.finish call with cs.commitAttemptLocked so the stream is still committed, releasing resources held for retries such as the config selector's OnCommitted callback, without being finished a second time. This mirrors the retryLocked give-up path.

Add a regression test that fails before this change and passes after it.

RELEASE NOTES:

  • client: Fix a bug where a ClientConn could get permanently stuck in IDLE after an RPC was canceled while the channel was exiting idle mode

@utkuozdemir

Copy link
Copy Markdown
Contributor Author

For transparency: I used an LLM, namely Opus 4.8 heavily on reproducing and fixing this issue.

@codecov

codecov Bot commented Jun 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.27%. Comparing base (5c7f936) to head (31f923f).
⚠️ Report is 18 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9191      +/-   ##
==========================================
+ Coverage   83.09%   83.27%   +0.18%     
==========================================
  Files         419      421       +2     
  Lines       33858    34086     +228     
==========================================
+ Hits        28134    28385     +251     
+ Misses       4291     4272      -19     
+ Partials     1433     1429       -4     
Files with missing lines Coverage Δ
stream.go 82.22% <100.00%> (+0.49%) ⬆️

... and 48 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

utkuozdemir added a commit to utkuozdemir/sidero-omni that referenced this pull request Jun 19, 2026
Omni could stop responding to all API requests after a period without API activity, leaving the web UI on a blank page while static assets still loaded, and required a restart to recover.

The connections that carry the API gateway and internal proxy traffic run in-process and relied on gRPC idle mode, which tears down and later re-establishes a connection that has been inactive. A bug in gRPC could leave such a connection permanently stuck once it went idle, so every subsequent request blocked forever.

Idle mode provides no benefit for these connections because they are in-process and live for the whole lifetime of the process, so disable it. This avoids the stuck connection and also removes the latency of re-establishing it on the first request after a quiet period.

A fix for the underlying gRPC issue has been submitted upstream in grpc/grpc-go#9191, but disabling idle mode here is the correct configuration regardless.

Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
@utkuozdemir
utkuozdemir force-pushed the fix/idle-double-finish-newattempt branch from 19d6e56 to c33f3be Compare June 19, 2026 21:45
@utkuozdemir

Copy link
Copy Markdown
Contributor Author

@Pranjali-2501 gentle nudge - can you or another member of the project have a look into this when you get a chance? Just in case this was missed.

@easwars

easwars commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Apologies for the delay in reviewing this. I will have this reviewed shortly.

@easwars easwars self-assigned this Jul 8, 2026
@easwars easwars added Type: Bug Area: Client Includes Channel/Subchannel/Streams, Connectivity States, RPC Retries, Dial/Call Options and more. labels Jul 8, 2026
@easwars easwars added this to the 1.83 Release milestone Jul 8, 2026

@eshitachandwani eshitachandwani left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for fixing this, mostly looks good. I just have a few comments

Comment thread stream.go Resolved
Comment thread internal/idle/idle_e2e_test.go Resolved
Comment thread internal/idle/idle_e2e_test.go Resolved
Comment thread internal/idle/idle_e2e_test.go Resolved
@utkuozdemir

Copy link
Copy Markdown
Contributor Author

Thank you for fixing this, mostly looks good. I just have a few comments

Thanks for the review, I addressed all your comments, amended into the same single commit.

@eshitachandwani

Copy link
Copy Markdown
Member

LGTM , adding @easwars for a second review.

Comment thread internal/idle/idle_e2e_test.go Resolved
Comment thread internal/idle/idle_e2e_test.go Resolved
Comment thread stream.go Outdated
Comment on lines +849 to +853
// Commit (releasing any resources held for retries, e.g. the
// config selector's OnCommitted callback) but do not call
// cs.finish: no attempt was created, and newClientStream's
// deferred cleanup finishes the stream on this error return.
cs.commitAttemptLocked()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is the correct fix. Only calling commitAttemptLocked from here and not calling cs.finish means that other cleanup things that happen in the latter are skipped for this stream.

For example, a binary log entry that either contains a "Cancel" or "ServerTrailer" will be skipped. See:

grpc-go/stream.go

Line 1108 in 7d87045

if len(cs.binlogs) != 0 {
. And, similarly, the code that cancels the stream's context at the end of cs.finish is skipped as well.

The crux of the problem is two things:

  • The OnFinish call option does not stick to its contract of executing the callback provided to it not more than once.
    • Ideally, the OnFinish call option should wrap the caller provider callback in a sync.OnceFunc to guarantee this
  • Doing the above will guarantee that all OnFinish call options are only ever invoked once, but we still have the issue of endOfClientStream being invoked more than once if cs.finish is called from withRetry during stream creation failure (which results in endOfClientStream being called once) and the deferred block in newClientStream running it the second time because of stream creation failure.

We should guarantee that endOfClientStream only ever runs once as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reworked it as suggested: cs.finish is called from withRetry again, and it now skips endOfClientStream when cs.attempt is nil.

A nil attempt only happens when stream creation failed, and on that path the deferred cleanup in newClientStream already calls endOfClientStream, so it runs exactly once either way.

I also tried wrapping the callback in a sync.Once in OnFinish, but later dropped it. The same CallOption value can be reused across RPCs (e.g. via WithDefaultCallOptions), and the once would swallow the callback on every call after the first. Since the callbacks are only invoked from endOfClientStream, that guarantee already covers them. This approach is also simpler.

Pls let me know if I miss anything.

@easwars easwars assigned utkuozdemir and unassigned easwars Jul 16, 2026
When stream creation failed after the attempt could not be created (for
example because the RPC context was already canceled while the channel
was exiting idle mode), the end-of-stream cleanup ran twice: once via
`cs.finish` called from `withRetry`, and once via the deferred cleanup
in `newClientStream`. This invoked every `OnFinish` callback twice for
a single call, including the idleness manager's `OnCallEnd`, which
corrupted the active-call counter and left the channel permanently
stuck in IDLE. All subsequent RPCs on the channel then failed with
"context deadline exceeded while waiting for connections to become
ready". The call was also double-counted in channelz stats.

Skip `endOfClientStream` in `cs.finish` when no attempt was ever
created: in that case stream creation has failed and the deferred
cleanup in `newClientStream` performs it instead. This guarantees the
cleanup, including all `OnFinish` callbacks, runs at most once per
call.

RELEASE NOTES:
* client: Fix a bug where a ClientConn could get permanently stuck in IDLE after an RPC was canceled while the channel was exiting idle mode
@easwars

easwars commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request prevents duplicate cleanup calls when an RPC fails during stream creation before any attempt is started. In stream.go, the finish method is updated to only call endOfClientStream if an attempt was actually created. A new end-to-end test, TestChannelIdleness_RPCFailingBeforeAttempt_FinishesOnce, is added to verify this behavior. There are no review comments to address, and I have no feedback to provide.

@easwars
easwars merged commit 75f3c0b into grpc:master Jul 17, 2026
15 of 16 checks passed
@easwars

easwars commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@utkuozdemir Thanks for your contribution

@utkuozdemir
utkuozdemir deleted the fix/idle-double-finish-newattempt branch July 17, 2026 18:09
zanarellidev added a commit to zanarellidev/grpc-go-onfinish that referenced this pull request Jul 29, 2026
easwars correctly flagged that wrapping the callback in sync.Once
breaks WithDefaultCallOptions: that dial option stores and reuses the
same CallOption value across every RPC on the connection (call.go:36,
stream.go:245), so a Once baked into the constructor only ever fires
for the first RPC made on the connection, never again after that.

Digging further, callInfo.onFinish (the field before() appends to) is
never read anywhere in the codebase. The actual invocation path is
endOfClientStream calling o.OnFinish(err) directly on the CallOption
from the original opts slice, guarded to run exactly once per RPC by
clientStream.finish's cs.finished check (already the fix for grpc#9191).
The tests this PR added only exercised the unused callInfo.onFinish
field, so they passed without validating real behavior.

With that guard already in place and no reproduction against the real
invocation path, there is no live bug to fix here beyond what grpc#9191
already covers. Reverting to plain field assignment, matching master.

Assisted-by: Claude Sonnet 5
Signed-off-by: zanarelli <zanarelli.dev@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: Client Includes Channel/Subchannel/Streams, Connectivity States, RPC Retries, Dial/Call Options and more. Type: Bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants