Skip to content

Validate low cardinality keys - #6713

Merged
jonatan-ivanov merged 1 commit into
micrometer-metrics:mainfrom
jonatan-ivanov:validate-keyvalues
Sep 19, 2025
Merged

jonatan-ivanov merged 1 commit into
micrometer-metrics:mainfrom
jonatan-ivanov:validate-keyvalues

Conversation

@jonatan-ivanov

@jonatan-ivanov jonatan-ivanov commented Sep 9, 2025 •

Copy link
Copy Markdown
Member

Observations with the same name should have the same set of low cardinality keys.
Example 1:

observation{name=test, lowCardinalityKeyValues=[color=red]}
observation{name=test, lowCardinalityKeyValues=[color=green]}
observation{name=test, lowCardinalityKeyValues=[color=blue]}

Example 1 is valid since all the observations with the same name ("test") has the same set of low cardinality keys ("color").

Example 2:

observation{name=test, lowCardinalityKeyValues=[color=red]}
observation{name=test, lowCardinalityKeyValues=[]}
observation{name=test, lowCardinalityKeyValues=[status=ok]}

Example 2 is invalid since the second observation is missing a key the first one has ("color") and the third observation is not only missing a key the first one has ("color") but it also has an extra key the first one does not have ("status").

If the validator detects this, it will give you an error like this:

Metrics backends may require that all observations with the same name have the same set of low cardinality keys.
There is already an existing observation named 'test' containing keys [color].
The observation you are attempting to register has keys [status].

In order to fix it, you should always add the same set of keys to the Observation. If some data is missing, you can signal it with a "none"/"unknown"/etc. value:

observation{name=test, lowCardinalityKeyValues=[color=red, status=unknown]}
observation{name=test, lowCardinalityKeyValues=[color=unknown, status=unknown]}
observation{name=test, lowCardinalityKeyValues=[color=unknown, status=ok]}

In your codebase, this means if you attach keyvalues conditionally:

if (color != null) {
    observation.lowCardinalityKeyValue("color", color);
}

you should only apply the condition to the value only:

observation.lowCardinalityKeyValue("color", color != null ? color : "unknown");

If you want to turn this validation off (not recommended), you can do it this way:

TestObservationRegistry registry = TestObservationRegistry.builder()
        .validateObservationsWithTheSameNameHavingTheSameSetOfLowCardinalityKeys(false)
        .build();

@jonatan-ivanov jonatan-ivanov added this to the 1.16.0-RC1 milestone Sep 9, 2025
@jonatan-ivanov jonatan-ivanov added enhancement A general enhancement module: micrometer-test An issue that is related to our TCK labels Sep 9, 2025
@jonatan-ivanov jonatan-ivanov added module: micrometer-observation-test and removed module: micrometer-test An issue that is related to our TCK labels Sep 9, 2025

@shakuzen shakuzen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I'd validate this by default. Observations are multiple layers abstracted away from Prometheus, which is the only system I'm aware of that has this limitation, and even then, the limitation is only in the Prometheus Java client. I guess we're trying to help people writing instrumentation where they don't know what ObservationHandler might be in use at runtime. Logging a warning probably isn't very helpful when running tests because it's likely to go unnoticed. Given that, I'm not sure what alternative there is. Maybe this is the best we can do.

@jonatan-ivanov

jonatan-ivanov commented Sep 12, 2025 •

Copy link
Copy Markdown
Member Author

I added a component to be able to turn features on/off.
Right now I disabled this validation by default but now that I just pushed the changes I think maybe it should be enabled by default and if users have issues they can manually disable it by passing an empty array to the create method:

TestObservationRegistry.create(new TestObservationRegistry.Capability[]{});

or maybe adding an EMPTY (or ALL_OFF?) capability or array constant so that:

TestObservationRegistry.create(EMPTY);

@shakuzen

Copy link
Copy Markdown
Member

I'd be fine with it being checked by default if there's a straightforward way to disable it. These kinds of configurations options tend to get complicated easily, though.

@jonatan-ivanov

Copy link
Copy Markdown
Member Author

I enabled by default and added a constant to disable it easily:

TestObservationRegistry.create(ALL_CAPABILITIES_DISABLED);

I guess another alternative to make these things configurable is boolean flags with setters or a builder which I think they can get complicated the same way as the enums, not sure what better options we have.

@shakuzen shakuzen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A builder is most future-proof, I think, but we can try with this for now and switch to a builder later if it makes more sense.

Observations with the same name should have the same set of
low cardinality keys
@jonatan-ivanov

Copy link
Copy Markdown
Member Author

Let's use a builder from the beginning since it can prevent making the enum public.
I still kept the enum as an implementation detail to prevent using a bunch of booleans in the future. By default the feature is still enabled but using a builder you can disable it, see the tests.

@shakuzen shakuzen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jonatan-ivanov
jonatan-ivanov merged commit 3beeee1 into micrometer-metrics:main Sep 19, 2025
@jonatan-ivanov
jonatan-ivanov deleted the validate-keyvalues branch September 19, 2025 21:03
@izeye izeye mentioned this pull request Jan 15, 2026
izeye added a commit to izeye/micrometer that referenced this pull request Jan 15, 2026
Signed-off-by: Johnny Lim <izeye@naver.com>
jonatan-ivanov pushed a commit that referenced this pull request Jan 15, 2026
Signed-off-by: Johnny Lim <izeye@naver.com>
allnightlong pushed a commit to allnightlong/micrometer that referenced this pull request Jan 19, 2026
Signed-off-by: Johnny Lim <izeye@naver.com>
@JoostK

JoostK commented Mar 16, 2026

Copy link
Copy Markdown

We've had to disable this validator as it falsely triggers when MeterRegistry#remove(MeterID) is being used to clear any prior tags.

@jonatan-ivanov

Copy link
Copy Markdown
Member Author

I would argue that it is a false trigger. The name of the method to enable/disable this is validateObservationsWithTheSameNameHavingTheSameSetOfLowCardinalityKeys and its javadoc says:

Enables/disables validating that Observations with the same name should have the same set of low cardinality keys.

Even though this is most useful from the point of view of metrics but you don't need a MeterRegistry to trigger this validation (you can have other outputs using low cardinality key-values).

Also, even if you remove metrics from the registry, you might still have this issue if that data is published to your metrics backend. Unless you also remove the data from your backend too, I would say the validation is not a false trigger, the problem it validates still exists.

@shakuzen

Copy link
Copy Markdown
Member

We've had to disable this validator as it falsely triggers when MeterRegistry#remove(MeterID) is being used to clear any prior tags.

I'm lacking context around what you're doing and why. It sounds like you're working around an issue, in which case it probably makes sense you need to disable this validation. If you are using the Prometheus registry, the limitation on registering meters with the same name but a different set of tag keys has been lifted in 1.17.0 (see the migration guide). That said, it is still best practice to avoid that. If you have a use case that you think is an exception to that, we'd be interested to learn more about it.

@JoostK

JoostK commented Mar 17, 2026 •

Copy link
Copy Markdown

Thanks for the responses! I can see how there's a layering distinction at play here and MeterRegistry is just one potential consumer of the general observation registry, so not a source of truth in terms of the captured observations.

This is indeed in the context of a Prometheus registry, so the upcoming changes in 1.17 are interesting. We allow for observation keys to be extended through configuration, such that application refreshes may result in a different bag of observation keys;. These changes are expected to be non-frequent and a refresh listener explicitly purges the meter registry for affected meters to avoid the unsupported change in keys in Prometheus' Java client. The test that verifies this behavior therefore was affected by this validation, as the TestObservationRegistry wasn't aware of the purging action (and since only the MeterRegistry is being purged, that can be considered WAI as Jonatan mentioned). We do call TestObservationRegistry#clear() during the refresh, but I can see why that doesn't also result in clearing the "history" that has been captured in the validator (#clear() merely resets the captured observations, not necessarily any facts that have been established for prior observations)

So perhaps it is expected to use the option to disable this validation in this use-case, acknowledging the "risks" that observations with changing keys may have for any handler. So please see this as a data point as to how this validation may affect real world tests and the relevancy of the opt-out flag for future consideration.

Thanks for all your effort in maintaining Micrometer!

@jonatan-ivanov

Copy link
Copy Markdown
Member Author

We do call TestObservationRegistry#clear() during the refresh, but I can see why that doesn't also result in clearing the "history" that has been captured in the validator (#clear() merely resets the captured observations, not necessarily any facts that have been established for prior observations)

I think this can be a valid feature: when you call clear on TestObservationRegistry, we could clear the "memory" of the registry and the memory of the "validator" (the latter is not happening today). Do you want to create a pull request for this?

jonatan-ivanov added a commit to jonatan-ivanov/micrometer that referenced this pull request Mar 25, 2026
In micrometer-metricsgh-7295 a ThreadLocal was introduced and elements are only added to
it, never removed.

Also, it was reported in micrometer-metricsgh-6713 that in some cases reusing
TestObservationRegistry can trigger validation issues since the
validator remembers all the low cardinality key-values that were
recorded previously.

Both of the issues can be worked around by creating a new
TestObservationRegistry instance, however in some cases clearing its
memory might be more convenient.

See micrometer-metricsgh-7295
See micrometer-metricsgh-6713
@jonatan-ivanov

jonatan-ivanov commented Mar 25, 2026 •

Copy link
Copy Markdown
Member Author

@JoostK This might help: #7337

jonatan-ivanov added a commit that referenced this pull request Mar 25, 2026
In gh-7295 a ThreadLocal was introduced and elements are only added to
it, never removed.

Also, it was reported in gh-6713 that in some cases reusing
TestObservationRegistry can trigger validation issues since the
validator remembers all the low cardinality key-values that were
recorded previously.

Both of the issues can be worked around by creating a new
TestObservationRegistry instance, however in some cases clearing its
memory might be more convenient.

See gh-7295
See gh-6713
@JoostK

JoostK commented Mar 25, 2026

Copy link
Copy Markdown

Thanks, and apologies for the late reply; this was still pending in my inbox. Thanks for #7337, that ought to do it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants