Skip to content

HTTP/2 client may never complete request callback when connection fails during channel-to-stream transitionΒ #14654

Description

@bjorncs

Jetty version(s)
12.0.23 through 12.0.33 (regression from 12.0.22, introduced by #13261)

Jetty Environment
core

HTTP version
HTTP/2

Java version/vendor
OpenJDK 17

OS type/version
Linux (reproduced on RHEL 9 / aarch64)

Description

After upgrading from 12.0.22 to 12.0.33, HTTP/2 client requests may never receive a response or exception callback when the server goes down abruptly. The BufferingResponseListener.onComplete() is never invoked, causing the request's CompletableFuture to hang indefinitely.

In 12.0.22, HttpConnectionOverHTTP2.close(Throwable) explicitly aborted all active channels/exchanges via an abort() method that iterated activeChannels:

// 12.0.22
private void abort(Throwable failure) {
    for (HttpChannel channel : activeChannels) {
        HttpExchange exchange = channel.getHttpExchange();
        if (exchange != null)
            exchange.getRequest().abort(failure);
    }
    activeChannels.clear();
}

In 12.0.33, this was replaced by onFailure() (introduced via #13261), which does not abort active channels:

// 12.0.33
void onFailure(Throwable failure, Callback callback) {
    // ...
    closed = true;
    destroy(); // only destroys requestTimeouts (CyclicTimeouts)
    callback.succeeded();
}

The session failure callback chain in 12.0.33 is: notifyFailure() β†’ onFailure() β†’ callback β†’ failStreams(). Since notifyFailure() runs before failStreams():

  1. onFailure() sets closed = true and calls destroy(), which destroys the connection-level RequestTimeouts (CyclicTimeouts) β€” the safety-net timeout scheduler
  2. failStreams() then fails all HTTP/2 streams

Any request that has been associated with an HttpChannelOverHTTP2 (added to activeChannels) but has not yet created an HTTP/2 stream (i.e., HttpSenderOverHTTP2.sendHeaders() β†’ session.newStream() hasn't completed) falls through the cracks:

  • failStreams() doesn't see it (no stream in the session's stream map yet)
  • The abort() safety net from 12.0.22 no longer exists
  • The RequestTimeouts safety net has been destroyed by onFailure()

The request's Response.CompleteListener callback is never invoked.

Suggested fix:

Either:

  1. Restore the explicit abort() of active channels in onFailure(), or
  2. Move the notifyFailure() call to after failStreams() so that RequestTimeouts remains active as a safety net during stream failure processing, or
  3. Add an explicit iteration of activeChannels in onFailure() to abort exchanges that don't have a stream yet

How to reproduce?

  1. Use HttpClientTransportDynamic with HTTP/2 and a MultiplexConnectionPool
  2. Send many concurrent requests over HTTP/2 multiplexed connections
  3. Abruptly kill or OOM the server (causing TCP RST / connection close)
  4. Observe that some request listeners never receive a callback

The race window is between HttpConnectionOverHTTP2.send() acquiring a channel (under lock) and HttpSenderOverHTTP2.sendHeaders() calling session.newStream(). Under high concurrency with many multiplexed streams, this window is hit with non-trivial probability.

Activity

  1. added
    BugFor general bugs on Jetty side
    on Mar 12, 2026
  2. joakime commented on Mar 12, 2026

    @joakime
    Contributor
  3. sbordet commented on Mar 16, 2026

    @sbordet
    Contributor

    @bjorncs we're investigating this.

  4. self-assigned this
    on Mar 18, 2026
  5. moved this to πŸ— In progress in Jetty 12.0.34 FROZENon Mar 18, 2026
  6. moved this to πŸ— In progress in Jetty 12.1.8 FROZENon Mar 18, 2026
  7. lorban commented on Mar 18, 2026

    @lorban
    Contributor

    @bjorncs I could reproduce this problem by modifying GoAwayTest.testImmediateGoAwayAndDisconnectDoesNotLeakClientConnections() changing maxConnections to 1000.

    You analysis looks right to me: the HttpConnectionOverHTTP2.activeChannels may contain some entries that are left untouched if a failure occurs between HttpConnectionOverHTTP2.send() and HttpSenderOverHTTP2.sendHeaders().

    The PR at #14692 contains a fix with which I cannot reproduce the problem anymore. Could you please give it a try and report back the outcome?

    Thanks!

  8. bjorncs commented on Mar 25, 2026

    @bjorncs
    ContributorAuthor

    Hi @lorban. I don't have an easy way of reproducing and verify the fix sadly.

  9. sbordet commented on Mar 30, 2026

    @sbordet
    Contributor

    @bjorncs this issue turned out to be more complex than anticipated, so we'll likely push it to the next release: 12.0.35/12.1.9.

  10. 2 remaining items

  11. moved this to πŸ— In progress in Jetty 12.0.35on May 22, 2026
  12. moved this to πŸ— In progress in Jetty 12.1.10 FROZENon May 22, 2026
  13. moved this from πŸ— In progress to βœ… Done in Jetty 12.0.35on May 22, 2026
  14. moved this from πŸ— In progress to βœ… Done in Jetty 12.1.10 FROZENon May 22, 2026
  15. moved this from βœ… Done to No status in Jetty 12.0.36 FROZENon Jun 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugFor general bugs on Jetty side

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions