Skip to content

perf(propagation): use a hex lookup table to decode traceparent - #8739

Merged
dmathieu merged 9 commits into
open-telemetry:mainfrom
itssaharsh:propagation-hex-lookup-table
Sep 7, 2026
Merged

dmathieu merged 9 commits into
open-telemetry:mainfrom
itssaharsh:propagation-hex-lookup-table

Conversation

@itssaharsh

@itssaharsh itssaharsh commented Aug 12, 2026 •

Copy link
Copy Markdown
Member

perf(propagation): use a hex lookup table to decode traceparent

Summary

Replaces encoding/hex and the manual upperHex validation in the traceparent decode path with a single-pass reverse lookup table. (encoding/hex is still used by Inject for encoding.)

The "Why"

Currently, extractPart does a UTF-8 range loop over the string just to reject uppercase characters (to satisfy the W3C spec), and then runs encoding/hex.Decode.

By using a 256-byte reverse lookup table, we can handle both decoding and validation in one go. Invalid and uppercase characters map to 0xff. Since no valid nibble sets its top 4 bits, OR-ing the looked-up values and checking those bits catches invalid input in one check at the end.

Net effect: one less function, no separate validation scan, and no []byte conversion.

Impact

                                โ”‚ prop-base.txt โ”‚            prop-new.txt             โ”‚
                                โ”‚    sec/op     โ”‚   sec/op     vs base                โ”‚
Extract/Sampled-16                  904.3n ยฑ 2%   826.0n ยฑ 3%   -8.66% (p=0.000 n=12)
Extract/BogusVersion-16             196.4n ยฑ 1%   197.7n ยฑ 5%        ~ (p=0.378 n=12)
Extract/FutureAdditionalData-16     887.5n ยฑ 1%   788.6n ยฑ 1%  -11.14% (p=0.000 n=12)
geomean                             540.2n        504.9n        -6.52%

BogusVersion is unchanged as expected, since it bails on the version field before decoding anything substantial.

go test -run='^$' -bench='^BenchmarkExtract$' -benchmem -count=12 ./propagation/ > base.txt
# apply the patch
go test -run='^$' -bench='^BenchmarkExtract$' -benchmem -count=12 ./propagation/ > new.txt
benchstat base.txt new.txt

I ran this three times and got -8.66%, -13.96% and -15.65% on Extract/Sampled, with the baseline itself moving between 884n and 1181n. That's WSL2 on a laptop, so I'd read it as roughly 9-15% rather than a firm number. Direction was consistent and every run was p<0.001.

Allocations: unchanged. I'd assumed the []byte(part) cast was allocating and that removing it would show up here. It wasn't, escape analysis already kept it on the stack, so the win is entirely from dropping the range loop and the generic decode.

Implementation Notes

  • Code gen: The table is generated via gotmpl into propagation/internal/hextable/, matching the convention in sdk/log/internal/gen.go. trace/hex.go has an identical table but it's unexported and trace is a separate module the root module depends on, so propagation can't import it.
  • Reviewer question: If gotmpl feels like overkill for a single consumer, say so. propagation is in the root module, so a plain internal/hextable package would work with no codegen at all. Easy swap.
  • Future cleanup: Happy to migrate trace/hex.go onto the same template in a follow-up so there's genuinely one copy. Left it out to keep this focused.
  • Safety: This parses external headers, so replacing stdlib deserves scrutiny. The loop body is four lines with no conditionals in it, and validity is one check at the end. The 0xff sentinel and why it works are documented on the table. trace_context_test.go already has uppercase rejection tests for all four fields (version, trace ID, span ID, flags), which is exactly what upperHex provided, and those pass unchanged.
  • Still not optimal: the loop keeps 3 bounds checks per iteration (part[i], part[i+1], dst[i/2]). Unlike TraceIDFromHex these can't be removed with constant indices, since n varies across the fields (2, 32, 16).

Testing

  • go test -race ./... in the root module and sdk, 33 packages, all passing
  • make generate toolchain-check license-check misspell go-mod-tidy golangci-lint-fix verify-readmes verify-mods clean, no files modified. make generate reproduces the checked-in hextable.go byte for byte
  • Full make precommit doesn't go green on my machine, but both failures reproduce on an unmodified tree: otlpmetricgrpc's TestSelfObservability gets DeadlineExceeded instead of Unavailable because WSL2 doesn't refuse a dial to 127.0.0.1:1 fast enough, and otlploghttp needs ~66s under -race against the 60s default

extractPart called encoding/hex behind an upperHex guard, and upperHex
ranged over the string decoding UTF-8 just to reject the upper-case
A-F the spec disallows.

A reverse lookup table does both jobs at once: invalid characters map
to 0xff, so OR-ing every looked-up value and checking the top 4 bits
catches anything invalid, upper-case included. That drops upperHex and
the []byte conversion along with it.

Extract runs once per incoming request:

  Extract/Sampled-16               904.3n ยฑ 2%  826.0n ยฑ 3%   -8.66%
  Extract/FutureAdditionalData-16  887.5n ยฑ 1%  788.6n ยฑ 1%  -11.14%

p=0.000, n=12. Allocations are unchanged; the []byte conversion was
already stack allocated.

The table goes in internal/shared so there is a single source of truth
if other packages need it later.
@codecov

codecov Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

Codecov Report

โœ… All modified and coverable lines are covered by tests.
โœ… Project coverage is 88.5%. Comparing base (a922871) to head (5229015).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #8739     +/-   ##
=======================================
- Coverage   88.5%   88.5%   -0.1%     
=======================================
  Files        333     333             
  Lines      21101   21100      -1     
=======================================
- Hits       18685   18683      -2     
- Misses      2416    2417      +1     
Files with missing lines Coverage ฮ”
propagation/trace_context.go 98.3% <100.0%> (-0.1%) โฌ‡๏ธ

... and 1 file with indirect coverage changes

๐Ÿš€ New features to boost your workflow:
  • โ„๏ธ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread CHANGELOG.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Optimizes W3C traceparent extraction in go.opentelemetry.io/otel/propagation by replacing encoding/hex decoding plus a separate uppercase-validation scan with a single-pass reverse-lookup table that both decodes and rejects spec-disallowed uppercase hex.

Changes:

  • Rework traceparent part decoding to use a reverse hex lookup table with a single validity check.
  • Add a generated internal/hextable table (via gotmpl) and a shared template for reuse.
  • Add a changelog entry describing the performance/behavior-preserving decoding change.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
propagation/trace_context.go Switches extractPart to table-based hex decode + validation in one pass.
propagation/internal/hextable/hextable.go Adds generated 256-byte reverse lookup table for lowercase-hex decode/validation.
propagation/internal/gen.go Adds go:generate directive to generate hextable.go from the shared template.
internal/shared/hextable/hextable.go.tmpl Introduces reusable template for generating the reverse lookup table.
CHANGELOG.md Documents the traceparent decoding optimization in the Unreleased section.

๐Ÿ’ก Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread CHANGELOG.md Resolved
Co-authored-by: Tyler Yahn <MrAlias@users.noreply.github.com>
@itssaharsh
itssaharsh requested a review from MrAlias August 19, 2026 19:18
@itssaharsh

Copy link
Copy Markdown
Member Author

Is there anything I should add here :) @MrAlias

The template only ever had one consumer. trace/hex.go still has its own
private copy, so the generation machinery added moving parts without
removing the duplication it was meant to prevent.

If we ever do consolidate with trace/hex.go, we can revisit this then.
@itssaharsh

itssaharsh commented Sep 6, 2026 •

Copy link
Copy Markdown
Member Author

I wanted to ask will this get merged when the next release is out?
@dashpole @MrAlias

Comment thread CHANGELOG.md Resolved
Comment thread CHANGELOG.md Resolved
@dmathieu
dmathieu merged commit b42f111 into open-telemetry:main Sep 7, 2026
51 checks passed
itssaharsh added a commit to itssaharsh/opentelemetry-go that referenced this pull request Sep 19, 2026
Resolve CHANGELOG.md conflict: the open-telemetry#8739 entry landed upstream via its own
PR, so keep only the open-telemetry#8740 entry and place it in the existing Changed
section.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pellared added a commit that referenced this pull request Oct 2, 2026
This release contains the first stable release of the OpenTelemetry Go
Logs API and SDK.
Our project stability guarantees now apply to the following modules:

- `go.opentelemetry.io/otel/log`
- `go.opentelemetry.io/otel/sdk/log`

See our [versioning
policy](https://github.com/open-telemetry/opentelemetry-go/blob/main/VERSIONING.md)
for more information about these stability guarantees.

### Added

- Add `String` method for `KeyValue` type in
`go.opentelemetry.io/otel/attribute`. (#8205)
- Add `String` method for `Set` type in
`go.opentelemetry.io/otel/attribute`. (#8347)
- Add `WithMaxExportBatchSize` to `go.opentelemetry.io/otel/sdk/metric`
to configure the maximum export batch size for `PeriodicReader`. (#8960)
- Add `WithAttributeValueDepthLimit`, `DefaultAttributeValueDepthLimit`,
and `SpanLimits.AttributeValueDepthLimit` in
`go.opentelemetry.io/otel/sdk/trace` to configure the maximum depth of
span, event, link, and instrumentation scope attribute values. (#8938)
- Add `WithAttributeValueDepthLimit` in
`go.opentelemetry.io/otel/sdk/log` to configure the maximum depth of log
record and instrumentation scope attribute values. (#8938)
- Add `WithMaxResponseSize` to
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`,
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`, and
  `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8941)
- Add experimental `ProbabilitySampler` in
`go.opentelemetry.io/otel/sdk/trace/x` that conforms to the
[OpenTelemetry specification's threshold-based sampling
algorithm](https://opentelemetry.io/docs/specs/otel/trace/sdk/#probabilitysampler).
(#8123)
- Add experimental `*Binder` extension interfaces in
`go.opentelemetry.io/otel/metric/x` for instruments that support binding
attributes ahead of time. (#8760)
- Add the experimental `Finisher` synchronous metric instrument
extension interface to `go.opentelemetry.io/otel/metric/x`. (#8906)

### Changed

- Use the [OpenTelemetry Attribute Collection representation for
non-OTLP
protocols](https://opentelemetry.io/docs/specs/otel/common/#attribute-collection-representation-for-non-otlp)
for metric data diffs in
`go.opentelemetry.io/otel/sdk/metric/metricdata/metricdatatest`. (#8993)
- Apply a maximum depth of 64 by default to span, event, link, log
record, and instrumentation scope attribute values in
`go.opentelemetry.io/otel/sdk/trace` and
`go.opentelemetry.io/otel/sdk/log`. (#8938)
- Reduce allocations when applying attribute value length limits in
`go.opentelemetry.io/otel/sdk/trace` and
`go.opentelemetry.io/otel/sdk/log` by rebuilding composite values only
when truncation is needed. (#8912)
- Decode `traceparent` using a hex lookup table in
`go.opentelemetry.io/otel/propagation`, which rejects the
specification-disallowed upper-case characters without a separate scan
of the header. (#8739)
- Decode all characters at constant indices in `TraceIDFromHex` and
`SpanIDFromHex` in `go.opentelemetry.io/otel/trace` to eliminate bounds
checks. (#8740)

### Fixed

- Count valid U+FFFD replacement characters toward attribute value
length limits in `go.opentelemetry.io/otel/trace`,
`go.opentelemetry.io/otel/sdk/trace`, and
`go.opentelemetry.io/otel/sdk/log`.
- Truncate string attribute values to empty strings when the configured
length limit is zero, including malformed UTF-8, in
`go.opentelemetry.io/otel/trace`, `go.opentelemetry.io/otel/sdk/trace`,
and `go.opentelemetry.io/otel/sdk/log`. (#9054)
- Fix a data race in `NewPeriodicReader` where `r.inst` was assigned
after the background goroutine was launched in
`go.opentelemetry.io/otel/sdk/metric`. (#9028)
- Prevent precision loss for large `int64` values in `Sum`, `Histogram`,
and `ExponentialHistogram` aggregations in
`go.opentelemetry.io/otel/sdk/metric`. (#8981)
- Ensure `grpc.DialOption` values passed via `WithDialOption` take
precedence over conflicting internally-computed defaults in
  `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`,
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`, and
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`.
(#8805, #8834, #8836)
- Treat overflowing `OTEL_METRIC_EXPORT_INTERVAL` and
`OTEL_METRIC_EXPORT_TIMEOUT` values as invalid in
`go.opentelemetry.io/otel/sdk/metric`. (#8800)
- Prevent a panic in `NewFixedSizeReservoir` and
`FixedSizeReservoirProvider` when given a negative size in
`go.opentelemetry.io/otel/sdk/metric/exemplar`; negative sizes are now
clamped to zero, consistent with a size of zero. (#8832)
- Preserve exponential histogram `ZeroThreshold` during OTLP metric
export in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric`. (#8801)
- Ensure metric helpers in `go.opentelemetry.io/otel/semconv/v1.32.0`,
`go.opentelemetry.io/otel/semconv/v1.33.0`, and
`go.opentelemetry.io/otel/semconv/v1.34.0` record measurements only once
when no attributes are provided. (#8849)
- Treat empty `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` values
as unset in `go.opentelemetry.io/otel/sdk/trace`. (#8873)
- Normalize global `OTEL_EXPORTER_OTLP_ENDPOINT` path joining in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp` so
trailing-slash base URLs do not produce double-slash log export paths.
(#8864)
- Prevent a deadlock when formatting an error passed to `RecordError`
calls back into the span in `go.opentelemetry.io/otel/sdk/trace`.
(#8815)
- Prevent `SimpleSpanProcessor.Shutdown` from panicking when constructed
with a nil exporter in `go.opentelemetry.io/otel/sdk/trace`. (#8844)
- Propagate invalid exponential histogram scale errors to Prometheus
exporter self-observability metrics in
`go.opentelemetry.io/otel/exporters/prometheus`. (#8839)
- Retain instrument advisory attributes configured with
`go.opentelemetry.io/otel/metric/x.WithDefaultAttributes` when a
matching View in `go.opentelemetry.io/otel/sdk/metric` does not specify
an attribute filter. (#8859)
- Ignore HTTP URL paths when building OTLP/gRPC metric exporter targets
from `OTEL_EXPORTER_OTLP_ENDPOINT` and
`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, while preserving `unix://` and
`unix-abstract://` targets; treat `unix-abstract://` endpoints as
insecure in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc` and
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`.
(#8862)
- Ignore HTTP(S) paths when deriving gRPC trace exporter endpoints from
`OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`
in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`.
(#8852)
- Bound memory used while parsing dense W3C property metadata by storing
it compactly and materializing `Property` values on demand in
`go.opentelemetry.io/otel/baggage` and
`go.opentelemetry.io/otel/propagation`.

### Removed

- Remove the experimental `OTEL_GO_X_METRIC_EXPORT_BATCH_SIZE`
environment variable in `go.opentelemetry.io/otel/sdk/metric`.
  Use the `WithMaxExportBatchSize` option instead. (#8960)
@pellared pellared added this to the v1.47.0 milestone Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants