Repository navigation
Optimize OpenTelemetry baggage propagation - #1551
Conversation
Covers inject and extract for the OTel bridge's one-header-per-field baggage propagator, the counterpart to the existing Brave-side W3CPropagationBenchmark. inject reads baggage from the current Context rather than from the Context passed to it, so the inject scenarios make baggage current for the whole trial; without that inject measures as a no-op. Also adds a logback config pinning the root logger to WARN. Logback defaults to DEBUG when no configuration is present, which made the guarded log.debug in extract actually format and write to the console - extract measured 11223 ns/op with logging on versus 171 ns/op with it off, i.e. the benchmark was measuring console I/O rather than propagation.
inject() and fields() had no test coverage at all. These tests pin the current behaviour before it gets optimised, in particular the two easy things to break: - remote field names are matched against baggage keys case-insensitively - the baggage key's casing wins over the configured remote field's casing when writing to the carrier
inject() rebuilt a lower-cased copy of the immutable remoteFields list on every call, then did an O(entries x fields) List.contains scan with a toLowerCase() allocation per baggage key, across two stream pipelines and two intermediate lists. It also called getAllBaggage() before checking whether any remote field was configured at all - the default in several setups. extract() allocated a SimpleEntry per remote field, a HashMap via Collectors.toMap, and a fresh BaggageEntryMetadata per entry for what is a compile-time constant. Now: the remote field names are precomputed once as an array and matched with equalsIgnoreCase, both hot paths are plain loops, the metadata object is a static constant, and inject() returns early when no remote fields are configured. Two notes on behaviour: - equalsIgnoreCase is locale-independent, whereas the toLowerCase() comparison it replaces used the default locale. Matching a remote field named "ID" against a baggage key "id" previously failed under a Turkish locale; it no longer does. - extract() no longer throws IllegalStateException when remoteFields contains duplicates, because Collectors.toMap is gone. Covered by a new test. fields() now returns an unmodifiable copy taken at construction time rather than the caller's live list. JMH, gc.alloc.rate.norm and ns/op: inject NO_REMOTE_FIELDS 1384 -> 48 B/op 182.3 -> 2.4 ns/op inject MATCHING 1680 -> 1136 B/op 274.5 -> 141.1 ns/op inject NO_MATCHING_BAGGAGE 1480 -> 944 B/op 238.0 -> 113.3 ns/op inject MANY_BAGGAGE_ENTRIES 10608 -> 9980 B/op 2892.8 -> 2089.2 ns/op extract 968 -> 324 B/op 170.7 -> 165.6 ns/op extractNoMatchingFields 592 -> 96 B/op 61.7 -> 14.6 ns/op The remaining MANY_BAGGAGE_ENTRIES cost is in OtelBaggageManager.getAllBaggage().
BaggageTextMapPropagator.inject() calls getAllBaggage(), which is where most of its remaining allocation lived. Each call built an ArrayDeque (a 16-slot backing array for at most two contexts), a CompositeBaggage whose constructor eagerly built two HashMaps plus two unmodifiable wrappers - including a stream().collect(toMap(...)) for asMap() that inject never reads - an Entry per baggage key per context, and then a third HashMap copy in toMap(). CompositeBaggage now builds a single map once and serves size(), forEach(), asMap() and getEntryValue() from it. getEntryValue() becomes an O(1) lookup instead of a linear stream scan. Contexts are passed as varargs, which drops the ArrayDeque and the descendingIterator. That last part is the risky bit: the merge order decides which context wins when both carry the same baggage key, and nothing tested it. It is now pinned by OtelBaggageManagerTests.should_prefer_trace_context_baggage_over_current_context_baggage, which documents the pre-existing behaviour - the trace context wins over the current context - and passes both before and after this change. Also in this file: entryForName() allocated an Entry per baggage key to find one, now it allocates at most one; baggageWithValue() re-streamed and lower-cased remoteFields on every call, and built a BaggageEntryMetadata for a constant string. getEntries() and Entry.fromBaggage() had one caller each and are gone. JMH, on top of the previous commit: inject MATCHING 1136 -> 652 B/op 141.1 -> 85.4 ns/op inject NO_MATCHING_BAGGAGE 944 -> 508 B/op 113.3 -> 74.3 ns/op inject MANY_BAGGAGE_ENTRIES 9980 -> 6536 B/op 2089.2 -> 1332.2 ns/op Versus the original baseline, inject is now 61-97% lower allocation and 54-99% faster depending on scenario.
extract() always returned context.with(newBaggage.build()), allocating a builder, a Baggage and a Context node even when the carrier carried none of the configured remote fields - which is the common case for most requests. The builder is now created lazily on the first hit, and the original context is returned untouched when there is nothing to propagate. This is a behaviour change, which is why it is a separate commit. When the incoming context had no baggage at all, the old code attached an empty Baggage to it, so Baggage.fromContextOrNull(result) was non-null; it is now null. When the context already had baggage the result is observationally equal either way. Nothing in the repository relied on the unconditional attach: the CompositeTextMapPropagator that used to branch on "did baggage get extracted" no longer exists, and OtelPropagator.extract() only inspects Span.fromContextOrNull. The remaining observable surface is EventPublishingContextWrapper's ScopeAttachedEvent/ScopeRestoredEvent#getBaggage(), which return Baggage.fromContextOrNull; the scope test suites pass. JMH, on top of the previous commit: extractNoMatchingFields 96 -> ~0 B/op 14.4 -> 3.9 ns/op The trade is a branch on the path where baggage is present, which measures at roughly 165 -> 180 ns/op for extract. Worth it to make the empty path free.
Required by the license plugin, which covers the jmh source set too.
aa8a4a6 to
9c327fc
Compare
|
|
||
| private final List<String> tagFields; | ||
|
|
||
| private final List<String> baggageFields; |
There was a problem hiding this comment.
This field is unused currently, but I believe it's supposed to be returned from getBaggageFields() instead of remoteFields that is currently returned. That's an existing bug, so I reverted an earlier change in this PR that removed it.
| } | ||
| newBaggage.put(remoteFieldName, value, PROPAGATION_UNLIMITED_METADATA); | ||
| if (debugEntries != null) { | ||
| debugEntries.put(remoteFieldName, value); |
There was a problem hiding this comment.
I don't like this debug stuff mixed in with the rest, but it's at least guarded by debug logging being enabled.
| } | ||
|
|
||
| jmh { | ||
| duplicateClassesStrategy = DuplicatesStrategy.EXCLUDE |
There was a problem hiding this comment.
These were being applied to lots of non-benchmark projects. I suppose we could move them to the if (project.name.contains('benchmarks')) { branch, but they also seemed redundant with what is declared in the one benchmark project in this repo.
|
I think there are more improvements and cleanup that can be done, but I'll merge this for now to get it included in M2. |
The common (default) case when no remote fields are configured was not as optimized as it could be. It was still fast but had significant unnecessary allocation. Now it is even faster and almost no allocation. Across all scenarios allocation has been significantly reduced. Extraction with matching fields is slightly slower but with less allocation. Below are results on my M1 macbook from the added benchmark.
1.8.0-M1 benchmark results (baseline)
This PR benchmark results