Repository navigation
fix: drop high-cardinality server.address from http_server metrics - #11208
Merged
Merged
Conversation
otelhttp derives server.address from the Host header, which creates a unique time series for every subdomain hostname on public gateways (e.g. each CID.ipfs.dweb.link). this caused multi-gigabyte prometheus responses and scrape timeouts. - cmd/ipfs/kubo/daemon.go: add OTel SDK View that drops server.address from all http.server.* metrics at the MeterProvider level - core/corehttp/gateway.go: add server.domain attribute to Gateway and HostnameGateway handlers, grouping by Gateway.PublicGateways suffix (e.g. "dweb.link"), with "localhost", "loopback", or "other" fallbacks - core/corehttp/commands.go: add server.domain="api" to RPC handler - core/corehttp/gateway.go: add server.domain="libp2p" to libp2p handler - docs/changelogs/v0.40.md: add changelog highlight - docs/metrics.md: document server_domain label and server_address drop
lidel
marked this pull request as ready for review
February 24, 2026 23:31
Contributor
Author
|
Merging to cherry-pick in 0.40 |
lidel
added a commit
that referenced
this pull request
Feb 24, 2026
โฆ11208) otelhttp derives server.address from the Host header, which creates a unique time series for every subdomain hostname on public gateways (e.g. each CID.ipfs.dweb.link). this caused multi-gigabyte prometheus responses and scrape timeouts. - cmd/ipfs/kubo/daemon.go: add OTel SDK View that drops server.address from all http.server.* metrics at the MeterProvider level - core/corehttp/gateway.go: add server.domain attribute to Gateway and HostnameGateway handlers, grouping by Gateway.PublicGateways suffix (e.g. "dweb.link"), with "localhost", "loopback", or "other" fallbacks - core/corehttp/commands.go: add server.domain="api" to RPC handler - core/corehttp/gateway.go: add server.domain="libp2p" to libp2p handler - docs/changelogs/v0.40.md: add changelog highlight - docs/metrics.md: document server_domain label and server_address drop (cherry picked from commit 6a082f6)
lidel
added a commit
that referenced
this pull request
Feb 25, 2026
#11208 dropped server.address via an OTel SDK View, but the SDK passes View-dropped attributes into exemplars as FilteredAttributes. CID subdomain hostnames (~85 runes for key+value) push exemplar labels past the Prometheus 128-rune limit, causing per-scrape log spam. Replace the View+server.domain approach with otelhttp.WithServerName, which sets server.address to a fixed short value at the source. Nothing leaks into exemplars because the attribute is never dropped by a View. - cmd/ipfs/kubo/daemon.go: remove WithView and attribute import - core/corehttp/gateway.go: remove server.domain logic (~90 lines), use WithServerName("gateway") and WithServerName("libp2p") - core/corehttp/commands.go: use WithServerName("api") - docs/metrics.md, docs/changelogs/v0.40.md: update accordingly
lidel
added a commit
that referenced
this pull request
Feb 25, 2026
the OTel SDK View from #11208 drops server.address from http.server.* metric labels, but the OTel spec requires filtered attributes to be carried as exemplar FilteredAttributes. on subdomain gateways the server.address value (e.g. "CID.ipfs.dweb.link") combined with trace_id and span_id exceeds the 128-rune prometheus exemplar limit. - cmd/ipfs/kubo/daemon.go: add exemplar.AlwaysOffFilter to MeterProvider - docs/changelogs/v0.40.md: document exemplar disable in metrics section
lidel
added a commit
that referenced
this pull request
Feb 25, 2026
* fix: disable otel exemplars to prevent prometheus rune overflow the OTel SDK View from #11208 drops server.address from http.server.* metric labels, but the OTel spec requires filtered attributes to be carried as exemplar FilteredAttributes. on subdomain gateways the server.address value (e.g. "CID.ipfs.dweb.link") combined with trace_id and span_id exceeds the 128-rune prometheus exemplar limit. - cmd/ipfs/kubo/daemon.go: add exemplar.AlwaysOffFilter to MeterProvider - docs/changelogs/v0.40.md: document exemplar disable in metrics section
lidel
added a commit
that referenced
this pull request
Feb 25, 2026
* fix: disable otel exemplars to prevent prometheus rune overflow the OTel SDK View from #11208 drops server.address from http.server.* metric labels, but the OTel spec requires filtered attributes to be carried as exemplar FilteredAttributes. on subdomain gateways the server.address value (e.g. "CID.ipfs.dweb.link") combined with trace_id and span_id exceeds the 128-rune prometheus exemplar limit. - cmd/ipfs/kubo/daemon.go: add exemplar.AlwaysOffFilter to MeterProvider - docs/changelogs/v0.40.md: document exemplar disable in metrics section (cherry picked from commit 221741e)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR removes
server_addressfromhttp_server*metrics, and adds saferserver_domainthat does not baloon cardinality on subdomain gatewaysExplainer
otelhttpderives server.address from theHostheader, which creates a unique time series for every subdomain hostname on public gateways (e.g. eachCID.ipfs.dweb.link). this caused multi-gigabyte prometheus responses and scrape timeouts on our staging test environment which received a chunk of production traffic fromipfs.io.cmd/ipfs/kubo/daemon.go: add OTel SDK View that drops server.address from all http.server.* metrics at the MeterProvider levelcore/corehttp/gateway.go: add server.domain attribute to Gateway and HostnameGateway handlers, grouping by Gateway.PublicGateways suffix (e.g. "dweb.link"), with "localhost", "loopback", or "other" fallbackscore/corehttp/commands.go: add server.domain="api" to RPC handlercore/corehttp/gateway.go: add server.domain="libp2p" to libp2p handlerdocs/changelogs/v0.40.md: add changelog highlightdocs/metrics.md: documentserver_domainlabel andserver_addressdrop