Skip to content

fix: drop high-cardinality server.address from http_server metrics - #11208

Merged
lidel merged 1 commit into
masterfrom
fix/otel-http-server-cardinality
Feb 24, 2026
Merged

lidel merged 1 commit into
masterfrom
fix/otel-http-server-cardinality

Conversation

@lidel

@lidel lidel commented Feb 24, 2026 •

Copy link
Copy Markdown
Contributor

This PR removes server_address from http_server* metrics, and adds safer server_domain that does not baloon cardinality on subdomain gateways

Explainer

otelhttp derives server.address from the Host header, which creates a unique time series for every subdomain hostname on public gateways (e.g. each CID.ipfs.dweb.link). this caused multi-gigabyte prometheus responses and scrape timeouts on our staging test environment which received a chunk of production traffic from ipfs.io.

  • cmd/ipfs/kubo/daemon.go: add OTel SDK View that drops server.address from all http.server.* metrics at the MeterProvider level
  • core/corehttp/gateway.go: add server.domain attribute to Gateway and HostnameGateway handlers, grouping by Gateway.PublicGateways suffix (e.g. "dweb.link"), with "localhost", "loopback", or "other" fallbacks
  • core/corehttp/commands.go: add server.domain="api" to RPC handler
  • core/corehttp/gateway.go: add server.domain="libp2p" to libp2p handler
  • docs/changelogs/v0.40.md: add changelog highlight
  • docs/metrics.md: document server_domain label and server_address drop

otelhttp derives server.address from the Host header, which creates
a unique time series for every subdomain hostname on public gateways
(e.g. each CID.ipfs.dweb.link). this caused multi-gigabyte prometheus
responses and scrape timeouts.

- cmd/ipfs/kubo/daemon.go: add OTel SDK View that drops server.address
  from all http.server.* metrics at the MeterProvider level
- core/corehttp/gateway.go: add server.domain attribute to Gateway and
  HostnameGateway handlers, grouping by Gateway.PublicGateways suffix
  (e.g. "dweb.link"), with "localhost", "loopback", or "other" fallbacks
- core/corehttp/commands.go: add server.domain="api" to RPC handler
- core/corehttp/gateway.go: add server.domain="libp2p" to libp2p handler
- docs/changelogs/v0.40.md: add changelog highlight
- docs/metrics.md: document server_domain label and server_address drop
@lidel
lidel marked this pull request as ready for review February 24, 2026 23:31
@lidel
lidel requested a review from a team as a code owner February 24, 2026 23:31
@lidel

lidel commented Feb 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Merging to cherry-pick in 0.40

@lidel
lidel merged commit 6a082f6 into master Feb 24, 2026
22 checks passed
@lidel
lidel deleted the fix/otel-http-server-cardinality branch February 24, 2026 23:32
lidel added a commit that referenced this pull request Feb 24, 2026
โ€ฆ11208)

otelhttp derives server.address from the Host header, which creates
a unique time series for every subdomain hostname on public gateways
(e.g. each CID.ipfs.dweb.link). this caused multi-gigabyte prometheus
responses and scrape timeouts.

- cmd/ipfs/kubo/daemon.go: add OTel SDK View that drops server.address
  from all http.server.* metrics at the MeterProvider level
- core/corehttp/gateway.go: add server.domain attribute to Gateway and
  HostnameGateway handlers, grouping by Gateway.PublicGateways suffix
  (e.g. "dweb.link"), with "localhost", "loopback", or "other" fallbacks
- core/corehttp/commands.go: add server.domain="api" to RPC handler
- core/corehttp/gateway.go: add server.domain="libp2p" to libp2p handler
- docs/changelogs/v0.40.md: add changelog highlight
- docs/metrics.md: document server_domain label and server_address drop

(cherry picked from commit 6a082f6)
lidel added a commit that referenced this pull request Feb 25, 2026
#11208 dropped server.address via an OTel SDK View, but the SDK passes
View-dropped attributes into exemplars as FilteredAttributes. CID
subdomain hostnames (~85 runes for key+value) push exemplar labels past
the Prometheus 128-rune limit, causing per-scrape log spam.

Replace the View+server.domain approach with otelhttp.WithServerName,
which sets server.address to a fixed short value at the source. Nothing
leaks into exemplars because the attribute is never dropped by a View.

- cmd/ipfs/kubo/daemon.go: remove WithView and attribute import
- core/corehttp/gateway.go: remove server.domain logic (~90 lines),
  use WithServerName("gateway") and WithServerName("libp2p")
- core/corehttp/commands.go: use WithServerName("api")
- docs/metrics.md, docs/changelogs/v0.40.md: update accordingly
lidel added a commit that referenced this pull request Feb 25, 2026
the OTel SDK View from #11208 drops server.address from http.server.*
metric labels, but the OTel spec requires filtered attributes to be
carried as exemplar FilteredAttributes. on subdomain gateways the
server.address value (e.g. "CID.ipfs.dweb.link") combined with
trace_id and span_id exceeds the 128-rune prometheus exemplar limit.

- cmd/ipfs/kubo/daemon.go: add exemplar.AlwaysOffFilter to MeterProvider
- docs/changelogs/v0.40.md: document exemplar disable in metrics section
lidel added a commit that referenced this pull request Feb 25, 2026
* fix: disable otel exemplars to prevent prometheus rune overflow

the OTel SDK View from #11208 drops server.address from http.server.*
metric labels, but the OTel spec requires filtered attributes to be
carried as exemplar FilteredAttributes. on subdomain gateways the
server.address value (e.g. "CID.ipfs.dweb.link") combined with
trace_id and span_id exceeds the 128-rune prometheus exemplar limit.

- cmd/ipfs/kubo/daemon.go: add exemplar.AlwaysOffFilter to MeterProvider
- docs/changelogs/v0.40.md: document exemplar disable in metrics section
lidel added a commit that referenced this pull request Feb 25, 2026
* fix: disable otel exemplars to prevent prometheus rune overflow

the OTel SDK View from #11208 drops server.address from http.server.*
metric labels, but the OTel spec requires filtered attributes to be
carried as exemplar FilteredAttributes. on subdomain gateways the
server.address value (e.g. "CID.ipfs.dweb.link") combined with
trace_id and span_id exceeds the 128-rune prometheus exemplar limit.

- cmd/ipfs/kubo/daemon.go: add exemplar.AlwaysOffFilter to MeterProvider
- docs/changelogs/v0.40.md: document exemplar disable in metrics section

(cherry picked from commit 221741e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant