Repository navigation
Description
Jetty version(s)
12.0.23 through 12.0.33 (regression from 12.0.22, introduced by #13261)
Jetty Environment
core
HTTP version
HTTP/2
Java version/vendor
OpenJDK 17
OS type/version
Linux (reproduced on RHEL 9 / aarch64)
Description
After upgrading from 12.0.22 to 12.0.33, HTTP/2 client requests may never receive a response or exception callback when the server goes down abruptly. The BufferingResponseListener.onComplete() is never invoked, causing the request's CompletableFuture to hang indefinitely.
In 12.0.22, HttpConnectionOverHTTP2.close(Throwable) explicitly aborted all active channels/exchanges via an abort() method that iterated activeChannels:
// 12.0.22
private void abort(Throwable failure) {
for (HttpChannel channel : activeChannels) {
HttpExchange exchange = channel.getHttpExchange();
if (exchange != null)
exchange.getRequest().abort(failure);
}
activeChannels.clear();
}In 12.0.33, this was replaced by onFailure() (introduced via #13261), which does not abort active channels:
// 12.0.33
void onFailure(Throwable failure, Callback callback) {
// ...
closed = true;
destroy(); // only destroys requestTimeouts (CyclicTimeouts)
callback.succeeded();
}The session failure callback chain in 12.0.33 is: notifyFailure() β onFailure() β callback β failStreams(). Since notifyFailure() runs before failStreams():
onFailure()setsclosed = trueand callsdestroy(), which destroys the connection-levelRequestTimeouts(CyclicTimeouts) β the safety-net timeout schedulerfailStreams()then fails all HTTP/2 streams
Any request that has been associated with an HttpChannelOverHTTP2 (added to activeChannels) but has not yet created an HTTP/2 stream (i.e., HttpSenderOverHTTP2.sendHeaders() β session.newStream() hasn't completed) falls through the cracks:
failStreams()doesn't see it (no stream in the session's stream map yet)- The
abort()safety net from 12.0.22 no longer exists - The
RequestTimeoutssafety net has been destroyed byonFailure()
The request's Response.CompleteListener callback is never invoked.
Suggested fix:
Either:
- Restore the explicit
abort()of active channels inonFailure(), or - Move the
notifyFailure()call to afterfailStreams()so thatRequestTimeoutsremains active as a safety net during stream failure processing, or - Add an explicit iteration of
activeChannelsinonFailure()to abort exchanges that don't have a stream yet
How to reproduce?
- Use
HttpClientTransportDynamicwith HTTP/2 and aMultiplexConnectionPool - Send many concurrent requests over HTTP/2 multiplexed connections
- Abruptly kill or OOM the server (causing TCP RST / connection close)
- Observe that some request listeners never receive a callback
The race window is between HttpConnectionOverHTTP2.send() acquiring a channel (under lock) and HttpSenderOverHTTP2.sendHeaders() calling session.newStream(). Under high concurrency with many multiplexed streams, this window is hit with non-trivial probability.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsβ Done
- StatusShow more project fieldsβ Done
Activity
joakime commentedon Mar 12, 2026
This could be related to an existing Issue
sbordet commentedon Mar 16, 2026
@bjorncs we're investigating this.
lorban commentedon Mar 18, 2026
@bjorncs I could reproduce this problem by modifying
GoAwayTest.testImmediateGoAwayAndDisconnectDoesNotLeakClientConnections()changing maxConnections to 1000.You analysis looks right to me: the
HttpConnectionOverHTTP2.activeChannelsmay contain some entries that are left untouched if a failure occurs betweenHttpConnectionOverHTTP2.send()andHttpSenderOverHTTP2.sendHeaders().The PR at #14692 contains a fix with which I cannot reproduce the problem anymore. Could you please give it a try and report back the outcome?
Thanks!
bjorncs commentedon Mar 25, 2026
Hi @lorban. I don't have an easy way of reproducing and verify the fix sadly.
sbordet commentedon Mar 30, 2026
@bjorncs this issue turned out to be more complex than anticipated, so we'll likely push it to the next release: 12.0.35/12.1.9.
2 remaining items