Skip to content

Optimize OpenTelemetry baggage propagation - #1551

Merged
shakuzen merged 8 commits into
micrometer-metrics:mainfrom
shakuzen:optimize-otel-baggage
Sep 17, 2026
Merged

shakuzen merged 8 commits into
micrometer-metrics:mainfrom
shakuzen:optimize-otel-baggage

Conversation

@shakuzen

@shakuzen shakuzen commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

The common (default) case when no remote fields are configured was not as optimized as it could be. It was still fast but had significant unnecessary allocation. Now it is even faster and almost no allocation. Across all scenarios allocation has been significantly reduced. Extraction with matching fields is slightly slower but with less allocation. Below are results on my M1 macbook from the added benchmark.

1.8.0-M1 benchmark results (baseline)

Benchmark                                                                                  (scenario)  Mode  Cnt      Score      Error   Units
BaggageTextMapPropagatorBenchmark.extract                                                         N/A  avgt   10    162.639 ±    2.093   ns/op
BaggageTextMapPropagatorBenchmark.extract:gc.alloc.rate                                           N/A  avgt   10   5675.600 ±   72.376  MB/sec
BaggageTextMapPropagatorBenchmark.extract:gc.alloc.rate.norm                                      N/A  avgt   10    968.001 ±    0.001    B/op
BaggageTextMapPropagatorBenchmark.extract:gc.count                                                N/A  avgt   10    108.000             counts
BaggageTextMapPropagatorBenchmark.extract:gc.time                                                 N/A  avgt   10     62.000                 ms
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage                                      N/A  avgt   10    169.528 ±    5.679   ns/op
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage:gc.alloc.rate                        N/A  avgt   10   5671.986 ±  183.674  MB/sec
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage:gc.alloc.rate.norm                   N/A  avgt   10   1008.001 ±    0.001    B/op
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage:gc.count                             N/A  avgt   10    107.000             counts
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage:gc.time                              N/A  avgt   10     64.000                 ms
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields                                         N/A  avgt   10     61.928 ±    1.379   ns/op
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields:gc.alloc.rate                           N/A  avgt   10   8870.144 ±  427.146  MB/sec
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields:gc.alloc.rate.norm                      N/A  avgt   10    576.000 ±   25.498    B/op
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields:gc.count                                N/A  avgt   10    130.000             counts
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields:gc.time                                 N/A  avgt   10     79.000                 ms
BaggageTextMapPropagatorBenchmark.inject                                                     MATCHING  avgt   10    261.158 ±    9.190   ns/op
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate                                       MATCHING  avgt   10   6308.617 ±  113.982  MB/sec
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate.norm                                  MATCHING  avgt   10   1728.002 ±   76.495    B/op
BaggageTextMapPropagatorBenchmark.inject:gc.count                                            MATCHING  avgt   10    119.000             counts
BaggageTextMapPropagatorBenchmark.inject:gc.time                                             MATCHING  avgt   10     69.000                 ms
BaggageTextMapPropagatorBenchmark.inject                                             NO_REMOTE_FIELDS  avgt   10    208.122 ±   44.758   ns/op
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate                               NO_REMOTE_FIELDS  avgt   10   6458.674 ± 1391.668  MB/sec
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate.norm                          NO_REMOTE_FIELDS  avgt   10   1384.001 ±    0.001    B/op
BaggageTextMapPropagatorBenchmark.inject:gc.count                                    NO_REMOTE_FIELDS  avgt   10    118.000             counts
BaggageTextMapPropagatorBenchmark.inject:gc.time                                     NO_REMOTE_FIELDS  avgt   10     69.000                 ms
BaggageTextMapPropagatorBenchmark.inject                                          NO_MATCHING_BAGGAGE  avgt   10    237.198 ±    9.946   ns/op
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate                            NO_MATCHING_BAGGAGE  avgt   10   5953.557 ±  249.505  MB/sec
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate.norm                       NO_MATCHING_BAGGAGE  avgt   10   1480.002 ±    0.001    B/op
BaggageTextMapPropagatorBenchmark.inject:gc.count                                 NO_MATCHING_BAGGAGE  avgt   10    112.000             counts
BaggageTextMapPropagatorBenchmark.inject:gc.time                                  NO_MATCHING_BAGGAGE  avgt   10     65.000                 ms
BaggageTextMapPropagatorBenchmark.inject                                         MANY_BAGGAGE_ENTRIES  avgt   10   2882.042 ±   39.087   ns/op
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate                           MANY_BAGGAGE_ENTRIES  avgt   10   3509.890 ±   46.709  MB/sec
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate.norm                      MANY_BAGGAGE_ENTRIES  avgt   10  10608.020 ±    0.001    B/op
BaggageTextMapPropagatorBenchmark.inject:gc.count                                MANY_BAGGAGE_ENTRIES  avgt   10     80.000             counts
BaggageTextMapPropagatorBenchmark.inject:gc.time                                 MANY_BAGGAGE_ENTRIES  avgt   10     46.000                 ms

This PR benchmark results

Benchmark                                                                                  (scenario)  Mode  Cnt      Score     Error   Units
BaggageTextMapPropagatorBenchmark.extract                                                         N/A  avgt   10    178.302 ±   4.359   ns/op
BaggageTextMapPropagatorBenchmark.extract:gc.alloc.rate                                           N/A  avgt   10   1839.923 ±  44.035  MB/sec
BaggageTextMapPropagatorBenchmark.extract:gc.alloc.rate.norm                                      N/A  avgt   10    344.001 ±   0.001    B/op
BaggageTextMapPropagatorBenchmark.extract:gc.count                                                N/A  avgt   10     56.000            counts
BaggageTextMapPropagatorBenchmark.extract:gc.time                                                 N/A  avgt   10     32.000                ms
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage                                      N/A  avgt   10    185.426 ±  13.123   ns/op
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage:gc.alloc.rate                        N/A  avgt   10   1978.571 ± 139.333  MB/sec
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage:gc.alloc.rate.norm                   N/A  avgt   10    384.001 ±   0.001    B/op
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage:gc.count                             N/A  avgt   10     60.000            counts
BaggageTextMapPropagatorBenchmark.extractIntoExistingBaggage:gc.time                              N/A  avgt   10     34.000                ms
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields                                         N/A  avgt   10      3.876 ±   0.618   ns/op
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields:gc.alloc.rate                           N/A  avgt   10      0.007 ±   0.001  MB/sec
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields:gc.alloc.rate.norm                      N/A  avgt   10     ≈ 10⁻⁵              B/op
BaggageTextMapPropagatorBenchmark.extractNoMatchingFields:gc.count                                N/A  avgt   10        ≈ 0            counts
BaggageTextMapPropagatorBenchmark.inject                                                     MATCHING  avgt   10     75.894 ±   1.737   ns/op
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate                                       MATCHING  avgt   10   7640.377 ± 173.278  MB/sec
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate.norm                                  MATCHING  avgt   10    608.001 ±   0.001    B/op
BaggageTextMapPropagatorBenchmark.inject:gc.count                                            MATCHING  avgt   10    143.000            counts
BaggageTextMapPropagatorBenchmark.inject:gc.time                                             MATCHING  avgt   10     82.000                ms
BaggageTextMapPropagatorBenchmark.inject                                             NO_REMOTE_FIELDS  avgt   10      2.272 ±   0.010   ns/op
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate                               NO_REMOTE_FIELDS  avgt   10  20141.000 ±  84.813  MB/sec
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate.norm                          NO_REMOTE_FIELDS  avgt   10     48.000 ±   0.001    B/op
BaggageTextMapPropagatorBenchmark.inject:gc.count                                    NO_REMOTE_FIELDS  avgt   10    224.000            counts
BaggageTextMapPropagatorBenchmark.inject:gc.time                                     NO_REMOTE_FIELDS  avgt   10    137.000                ms
BaggageTextMapPropagatorBenchmark.inject                                          NO_MATCHING_BAGGAGE  avgt   10     75.021 ±  14.354   ns/op
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate                            NO_MATCHING_BAGGAGE  avgt   10   6483.960 ± 388.231  MB/sec
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate.norm                       NO_MATCHING_BAGGAGE  avgt   10    508.001 ±  70.120    B/op
BaggageTextMapPropagatorBenchmark.inject:gc.count                                 NO_MATCHING_BAGGAGE  avgt   10    120.000            counts
BaggageTextMapPropagatorBenchmark.inject:gc.time                                  NO_MATCHING_BAGGAGE  avgt   10     74.000                ms
BaggageTextMapPropagatorBenchmark.inject                                         MANY_BAGGAGE_ENTRIES  avgt   10   1293.594 ±  65.479   ns/op
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate                           MANY_BAGGAGE_ENTRIES  avgt   10   4825.588 ± 247.209  MB/sec
BaggageTextMapPropagatorBenchmark.inject:gc.alloc.rate.norm                      MANY_BAGGAGE_ENTRIES  avgt   10   6540.009 ±   6.374    B/op
BaggageTextMapPropagatorBenchmark.inject:gc.count                                MANY_BAGGAGE_ENTRIES  avgt   10     89.000            counts
BaggageTextMapPropagatorBenchmark.inject:gc.time                                 MANY_BAGGAGE_ENTRIES  avgt   10     58.000                ms

Covers inject and extract for the OTel bridge's one-header-per-field baggage
propagator, the counterpart to the existing Brave-side W3CPropagationBenchmark.

inject reads baggage from the current Context rather than from the Context passed
to it, so the inject scenarios make baggage current for the whole trial; without
that inject measures as a no-op.

Also adds a logback config pinning the root logger to WARN. Logback defaults to
DEBUG when no configuration is present, which made the guarded log.debug in
extract actually format and write to the console - extract measured 11223 ns/op
with logging on versus 171 ns/op with it off, i.e. the benchmark was measuring
console I/O rather than propagation.
inject() and fields() had no test coverage at all. These tests pin the current
behaviour before it gets optimised, in particular the two easy things to break:

- remote field names are matched against baggage keys case-insensitively
- the baggage key's casing wins over the configured remote field's casing when
  writing to the carrier
inject() rebuilt a lower-cased copy of the immutable remoteFields list on every
call, then did an O(entries x fields) List.contains scan with a toLowerCase()
allocation per baggage key, across two stream pipelines and two intermediate
lists. It also called getAllBaggage() before checking whether any remote field
was configured at all - the default in several setups.

extract() allocated a SimpleEntry per remote field, a HashMap via
Collectors.toMap, and a fresh BaggageEntryMetadata per entry for what is a
compile-time constant.

Now: the remote field names are precomputed once as an array and matched with
equalsIgnoreCase, both hot paths are plain loops, the metadata object is a static
constant, and inject() returns early when no remote fields are configured.

Two notes on behaviour:

- equalsIgnoreCase is locale-independent, whereas the toLowerCase() comparison it
  replaces used the default locale. Matching a remote field named "ID" against a
  baggage key "id" previously failed under a Turkish locale; it no longer does.
- extract() no longer throws IllegalStateException when remoteFields contains
  duplicates, because Collectors.toMap is gone. Covered by a new test.

fields() now returns an unmodifiable copy taken at construction time rather than
the caller's live list.

JMH, gc.alloc.rate.norm and ns/op:

  inject NO_REMOTE_FIELDS      1384 -> 48 B/op    182.3 -> 2.4 ns/op
  inject MATCHING              1680 -> 1136 B/op  274.5 -> 141.1 ns/op
  inject NO_MATCHING_BAGGAGE   1480 -> 944 B/op   238.0 -> 113.3 ns/op
  inject MANY_BAGGAGE_ENTRIES 10608 -> 9980 B/op 2892.8 -> 2089.2 ns/op
  extract                       968 -> 324 B/op   170.7 -> 165.6 ns/op
  extractNoMatchingFields       592 -> 96 B/op     61.7 -> 14.6 ns/op

The remaining MANY_BAGGAGE_ENTRIES cost is in OtelBaggageManager.getAllBaggage().
BaggageTextMapPropagator.inject() calls getAllBaggage(), which is where most of
its remaining allocation lived. Each call built an ArrayDeque (a 16-slot backing
array for at most two contexts), a CompositeBaggage whose constructor eagerly
built two HashMaps plus two unmodifiable wrappers - including a
stream().collect(toMap(...)) for asMap() that inject never reads - an Entry per
baggage key per context, and then a third HashMap copy in toMap().

CompositeBaggage now builds a single map once and serves size(), forEach(),
asMap() and getEntryValue() from it. getEntryValue() becomes an O(1) lookup
instead of a linear stream scan. Contexts are passed as varargs, which drops the
ArrayDeque and the descendingIterator.

That last part is the risky bit: the merge order decides which context wins when
both carry the same baggage key, and nothing tested it. It is now pinned by
OtelBaggageManagerTests.should_prefer_trace_context_baggage_over_current_context_baggage,
which documents the pre-existing behaviour - the trace context wins over the
current context - and passes both before and after this change.

Also in this file: entryForName() allocated an Entry per baggage key to find one,
now it allocates at most one; baggageWithValue() re-streamed and lower-cased
remoteFields on every call, and built a BaggageEntryMetadata for a constant
string.

getEntries() and Entry.fromBaggage() had one caller each and are gone.

JMH, on top of the previous commit:

  inject MATCHING              1136 -> 652 B/op   141.1 -> 85.4 ns/op
  inject NO_MATCHING_BAGGAGE    944 -> 508 B/op   113.3 -> 74.3 ns/op
  inject MANY_BAGGAGE_ENTRIES  9980 -> 6536 B/op 2089.2 -> 1332.2 ns/op

Versus the original baseline, inject is now 61-97% lower allocation and 54-99%
faster depending on scenario.
extract() always returned context.with(newBaggage.build()), allocating a
builder, a Baggage and a Context node even when the carrier carried none of the
configured remote fields - which is the common case for most requests. The
builder is now created lazily on the first hit, and the original context is
returned untouched when there is nothing to propagate.

This is a behaviour change, which is why it is a separate commit. When the
incoming context had no baggage at all, the old code attached an empty Baggage
to it, so Baggage.fromContextOrNull(result) was non-null; it is now null. When
the context already had baggage the result is observationally equal either way.

Nothing in the repository relied on the unconditional attach: the
CompositeTextMapPropagator that used to branch on "did baggage get extracted" no
longer exists, and OtelPropagator.extract() only inspects
Span.fromContextOrNull. The remaining observable surface is
EventPublishingContextWrapper's ScopeAttachedEvent/ScopeRestoredEvent#getBaggage(),
which return Baggage.fromContextOrNull; the scope test suites pass.

JMH, on top of the previous commit:

  extractNoMatchingFields  96 -> ~0 B/op  14.4 -> 3.9 ns/op

The trade is a branch on the path where baggage is present, which measures at
roughly 165 -> 180 ns/op for extract. Worth it to make the empty path free.
Required by the license plugin, which covers the jmh source set too.
@shakuzen
shakuzen force-pushed the optimize-otel-baggage branch from aa8a4a6 to 9c327fc Compare September 14, 2026 12:51

private final List<String> tagFields;

private final List<String> baggageFields;

@shakuzen shakuzen Sep 14, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field is unused currently, but I believe it's supposed to be returned from getBaggageFields() instead of remoteFields that is currently returned. That's an existing bug, so I reverted an earlier change in this PR that removed it.

}
newBaggage.put(remoteFieldName, value, PROPAGATION_UNLIMITED_METADATA);
if (debugEntries != null) {
debugEntries.put(remoteFieldName, value);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like this debug stuff mixed in with the rest, but it's at least guarded by debug logging being enabled.

Comment thread build.gradle
}

jmh {
duplicateClassesStrategy = DuplicatesStrategy.EXCLUDE

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These were being applied to lots of non-benchmark projects. I suppose we could move them to the if (project.name.contains('benchmarks')) { branch, but they also seemed redundant with what is declared in the one benchmark project in this repo.

@shakuzen shakuzen changed the title Optimize otel baggage Optimize OTel baggage Sep 14, 2026
@shakuzen
shakuzen marked this pull request as ready for review September 14, 2026 14:10
@shakuzen shakuzen added enhancement New feature or request performance Issues related to general performance module: bridge-otel labels Sep 14, 2026
@shakuzen shakuzen added this to the 1.8.0-M2 milestone Sep 14, 2026
@shakuzen

Copy link
Copy Markdown
Member Author

I think there are more improvements and cleanup that can be done, but I'll merge this for now to get it included in M2.

@shakuzen shakuzen changed the title Optimize OTel baggage Optimize OpenTelemetry baggage propagation Sep 17, 2026
@shakuzen
shakuzen merged commit 574fd1e into micrometer-metrics:main Sep 17, 2026
8 checks passed
@shakuzen
shakuzen deleted the optimize-otel-baggage branch September 17, 2026 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request module: bridge-otel performance Issues related to general performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant