Skip to content

Fixes #13244 - Client responses of HTTP conversations have mismatching request. - #13245

Closed
sbordet wants to merge 6 commits into
jetty-12.1.xfrom
fix/jetty-12.1.x/13244/conversation-request-mismatch
Closed

sbordet wants to merge 6 commits into
jetty-12.1.xfrom
fix/jetty-12.1.x/13244/conversation-request-mismatch

Conversation

@sbordet

@sbordet sbordet commented Jun 13, 2025

Copy link
Copy Markdown
Contributor

Fixed by scoping ResponseListeners to a Request, and when combining ResponseListeners in HttpConversation, the internal response listeners are scoped to the original request.

Moved notifications of the response listeners from ResponseListeners to HttpConversation and simplified calling code.

…g request.

Fixed by scoping `ResponseListeners` to a `Request`, and when combining `ResponseListeners` in `HttpConversation`, the internal response listeners are scoped to the original request.

Moved notifications of the response listeners from `ResponseListeners` to `HttpConversation` and simplified calling code.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
@sbordet sbordet moved this to πŸ‘€ In review in Jetty 12.1.0 - (FROZEN) Jun 13, 2025
@sbordet
sbordet requested review from gregw and lorban June 13, 2025 21:30
sbordet added 2 commits June 16, 2025 14:11
Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
sbordet added 2 commits June 16, 2025 22:46
Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

@gregw gregw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I find this PR really confusing and neither the javadoc nor tests are sufficient to make it clearer to me.

After our discussion, I thought we had agreed that it made most sense to keep the Response.getRequest() to be the immediate request to which the response is a reply to (i.e. the last request in the conversation). But this PR appears to be changing so that the request associated with a response is the original request in the conversation chain.

If that is indeed the intention, then the javadoc and tests need to be much clearer that it is the case.

However, I'm really not sure it should be that way. Consider a POST request to "/foo" that receives a 302 redirection to "/bar", which is a cached resource, so the ultimate response is a 304. Thus the listener will see a 304 response to a GET "/bar" request, but when that response is asked for its request, it will see a POST request to "/foo". I find that very confusing. I'd much rather that a response always referenced the request that it was responding to, and that there was new API to access the original request or any request in the chain.

Whatever the ultimate behaviour, the javadoc needs to be really clear about it.

* Returns the list of response listeners that needs to be notified of response events.
* This list changes as the conversation proceeds, as follows:
* <p>Updates the response listeners, eventually using the given override response listener,
* that must be notified instead of the response listeners of the first exchange.</p>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This javadoc doesn't read well. I can't tell what it is saying? When is this method called and why?

* and we also notify the last exchange response listeners (if it's not also the first).
*
* This scheme allows for protocol handlers to not worry about other protocol handlers, or to worry
* <p>Basically the override conversation listener replaces the first exchange response listener,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still doesn't explain it very well. Also the chatty style of "Basically" is not good.

@sbordet

sbordet commented Jun 20, 2025

Copy link
Copy Markdown
Contributor Author

After our discussion, I thought we had agreed that it made most sense to keep the Response.getRequest() to be the immediate request to which the response is a reply to (i.e. the last request in the conversation). But this PR appears to be changing so that the request associated with a response is the original request in the conversation chain.

I was on the line between the two solutions, but I think it's best to keep the last response associated with the first request, and hide the conversation.
We can eventually add later conversation APIs, if necessary, and even now if users are interested in conversations they can disable it and with a little effort perform the conversations themselves.

We can discuss further, but presenting the last request has the problem that the first request would be difficult to reach, and either needs new APIs or new coding styles, both a burden for the users, just in case there could be a conversation.

@lorban

lorban commented Jun 20, 2025

Copy link
Copy Markdown
Contributor

After our discussion, I thought we had agreed that it made most sense to keep the Response.getRequest() to be the immediate request to which the response is a reply to (i.e. the last request in the conversation). But this PR appears to be changing so that the request associated with a response is the original request in the conversation chain.

I was on the line between the two solutions, but I think it's best to keep the last response associated with the first request, and hide the conversation. We can eventually add later conversation APIs, if necessary, and even now if users are interested in conversations they can disable it and with a little effort perform the conversations themselves.

We can discuss further, but presenting the last request has the problem that the first request would be difficult to reach, and either needs new APIs or new coding styles, both a burden for the users, just in case there could be a conversation.

I personally see this change as a quality of life improvement: having the original request paired with the final response in a conversation is IMHO the least surprising thing to do. At least, I think this is what would be expected in 99% of the cases and in the last 1% where you want to very strictly control what happens with 30x or 401 responses, disabling conversations looks saner than adding conversation-specific listeners.

@sbordet sbordet self-assigned this Jun 24, 2025
Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
@sbordet
sbordet requested review from gregw and lorban June 24, 2025 18:29

@gregw gregw left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sbordet @lorban I think the issue we are trying to solve is:

     client.newRequest("localhost", 8080)
            .path("/foo")
            .attribute("A", "1")
            .onResponseSuccess(r -> { assert r.getRequest().getAttributes().get("A").equals("1");})
            .send();

We want the above to be true even if there is a conversation. However, I do not like the current solution of changing the return of getRequest to return the original request as it results in just as many strange outcomes:

     client.newRequest("localhost", 8080)
            .path("/foo")
            .attribute("A", "1")
            .onRequestBegin(r -> r.attribute("B", r.getPath())
            .onResponseSuccess(r -> { 
                assert r.getRequest().getAttributes().get("A").equals("1"); 
                assert r.getRequest().getAttributes().get("B").equals("foo");
                 updateCache(r.getRequest().getPath(), r.getHeaders().get(HttpHeader.LAST_MODIFIED));
             })
            .send();

I think i would be far better to have the request of the response always be the request that created the response, but only have a single Attributes map for the whole conversation. So the request for the 2nd, 3rd and 4th request in the conversation will inherit the attributes of the initial request. This solves the original problem without breaking the last modified example above. Attributes sent on intermediate requests may still overwrite each other, but at least they can check for existing values and ultimately have full control.

@gregw

gregw commented Jun 25, 2025

Copy link
Copy Markdown
Contributor

If the attributes were shared between requests in a conversation then the following could be done:

     client.newRequest("localhost", 8080)
            .path("/foo")
            .attribute("A", "1")
            .attribute("Paths", new ArrayList())
            .onRequestBegin(r -> {
              ((List)r.getAttributes.get("Paths)).add(r.getPath());
              r.attribute("B","2");
             })
            .onResponseSuccess(r -> { 
                assert r.getRequest().getAttributes().get("A").equals("1"); 
                assert r.getRequest().getAttributes().get("B").equals("2");
                updateCache(r.getRequest().getPath(), r.getHeaders().get(HttpHeader.LAST_MODIFIED));
                assert ((List<String>)r.getRequest().getAttributes().get("Paths").get(0).equals("/foo");
             })
            .send();

@sbordet

sbordet commented Jun 25, 2025 •

Copy link
Copy Markdown
Contributor Author

@gregw it's not only about the attributes. I can conjure an example where an application would need the original request path, and won't be able to access it.

Sure it can set the original path as an attribute, but again we are forcing applications to do things just in case there is a conversation, which is not good.

For your specific example above, an application would really need to look at the specific redirect code; e.g. for a 301 you really want to update the cache with the original path, maybe not for other 30x codes.
And so we go back to @lorban comment that for more complicated use cases you disable conversations and you handle the case manually.

I think we should make the simple case transparent, i.e. this is always true no matter if there is a conversation or not.

// Same with other events, below an example of the invariant.
request.send(result -> result.getRequest() == request && result.getResponse().getRequest() == request)

I am not opposed to explicit add an API to handle conversations, but I think it should be orthogonal, and the invariant above should always hold.

@gregw

gregw commented Jun 25, 2025

Copy link
Copy Markdown
Contributor

@sbordet I'm moderately opposed to that invariant. I think it fast more likely that response heard need to be associated with the last request that the first (see my last modified example). If they want the original path, they already have it or can pass it as an attribute or use the list mechanism I used as an example.

Let's chat tomorrow when we have a 3rd

@gregw

gregw commented Jun 26, 2025

Copy link
Copy Markdown
Contributor

@sbordet @lorban

Currently this PR has:

     client.newRequest("localhost", 8080)
            .path("/foo")
            .attribute("A", "1")
            .onRequestBegin(r -> r.attribute("B", r.getPath())
            .onResponseSuccess(r -> { 
                assert r.getRequest().getAttributes().get("A").equals("1");  // FIXED!!! This is the original issue
                assert r.getRequest().getAttributes().get("B").equals("foo"); // SURPRISE!!! Doesn't work for conversation
                updateCache(r.getRequest().getPath(), r.getHeaders().get(HttpHeader.LAST_MODIFIED)); // SURPRISE!!! Doesn't work for conversation
             })
            .send();

I think the invariant that we wish to solve is just

request.attribute("A",value).send(result -> result.getResponse().getRequest().getAttributes().get("A") == value);

We can do that with the attribute approach I suggested above.
It also has the benefit that any headers in a response (e.g etag, last-modified) and the status code (e.g. 304) are directly applicable to the response.getRequest().

@sbordet

sbordet commented Jun 26, 2025

Copy link
Copy Markdown
Contributor Author

@gregw WRT your last example, assert r.getRequest().getAttributes().get("B").equals("foo") will pass, because attribute B is added to the original request, by the listener attached to the original request, no surprises here.

OTOH, r.getRequest().getPath() would be that of the original request.
Whether it is a surprise or not with respect to updating some cache, it entirely depends on what you want to do.
I could argue that in most cases the old path is exactly what you want, even with a redirect conversation.
For more complicated cases, where you want to cache the redirect, but not the original, you disable conversations.

Without this PR, this example is broken:

client.newRequest("localhost", 8080)
    .path("/foo")
    .attribute("requestNanoTime", System.nanoTime())
    .onResponseSuccess(rs -> {
        rs.getRequest().getAttributes().get("requestTime"); // => null in case of any conversation.
        rs.getRequest().getPath(); // => different from original in case of redirect conversation.
    })
    .send();

With this PR, there are no surprises, no matter whether there is a conversation or not.
The attributes will be there, the path will be the same, etc.
The block of code above would naturally be executed, and at a glance you would expect the path to be "/foo" and the attribute "requestNanoTime" be there.
Imagine if you're grouping request/response times by path, without this PR would be pretty hard to do.

Without this PR, the above is broken, and IMHO breaks the principle of least surprise, and application writers now have to go through hoops to make the code work in both the cases of no conversation and conversation.

The invariant you propose based on attributes is not enough: applications would like to reference the whole request, including path, method, version, headers, etc. with minimal code, as you would do without conversations.

@gregw

gregw commented Jun 26, 2025

Copy link
Copy Markdown
Contributor

@simone, the original request may have a different scheme, host, port, method, path, and cookie domain from the request that the ultimate response is for.

You have fixed the association of attributes, but have broken the association of everything else. The response status, headers and content may only be meaningful in relation to the final request not the first. Any attempt to interpret etags, dates, status, or cookies will often be wrong if it is applied to the first request. Potentially seriously CVE so.

So I'm still not in favour of this change

@sbordet

sbordet commented Jul 2, 2025

Copy link
Copy Markdown
Contributor Author

Closing as discussed.

Contextualizing the response's request to the first request may cause problem, for example:

GET / HTTP/1.1
Host: foo.com

HTTP/1.1 302
Location: http://bar.com/

GET / HTTP/1.1
Host: bar.com

HTTP/1.1 200
Set-Cookie: a=b

An application could see the Set-Cookie on the second response, and would have no way to know that it refers to bar.com, because in this PR the request would be the first request, whose host was foo.com.

@sbordet sbordet closed this Jul 2, 2025
@github-project-automation github-project-automation Bot moved this from πŸ‘€ In review to βœ… Done in Jetty 12.1.0 - (FROZEN) Jul 2, 2025
@sbordet
sbordet deleted the fix/jetty-12.1.x/13244/conversation-request-mismatch branch July 2, 2025 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

No open projects
Status: βœ… Done

Development

Successfully merging this pull request may close these issues.

Client responses of HTTP conversations have mismatching request

3 participants