Repository navigation
Validate low cardinality keys - #6713
Conversation
shakuzen
left a comment
There was a problem hiding this comment.
I'm not sure I'd validate this by default. Observations are multiple layers abstracted away from Prometheus, which is the only system I'm aware of that has this limitation, and even then, the limitation is only in the Prometheus Java client. I guess we're trying to help people writing instrumentation where they don't know what ObservationHandler might be in use at runtime. Logging a warning probably isn't very helpful when running tests because it's likely to go unnoticed. Given that, I'm not sure what alternative there is. Maybe this is the best we can do.
|
I added a component to be able to turn features on/off. TestObservationRegistry.create(new TestObservationRegistry.Capability[]{});or maybe adding an TestObservationRegistry.create(EMPTY); |
|
I'd be fine with it being checked by default if there's a straightforward way to disable it. These kinds of configurations options tend to get complicated easily, though. |
|
I enabled by default and added a constant to disable it easily: TestObservationRegistry.create(ALL_CAPABILITIES_DISABLED);I guess another alternative to make these things configurable is boolean flags with setters or a builder which I think they can get complicated the same way as the enums, not sure what better options we have. |
shakuzen
left a comment
There was a problem hiding this comment.
A builder is most future-proof, I think, but we can try with this for now and switch to a builder later if it makes more sense.
Observations with the same name should have the same set of low cardinality keys
fa7d59b to
d6de427
Compare
|
Let's use a builder from the beginning since it can prevent making the enum public. |
Signed-off-by: Johnny Lim <izeye@naver.com>
Signed-off-by: Johnny Lim <izeye@naver.com>
|
We've had to disable this validator as it falsely triggers when |
|
I would argue that it is a false trigger. The name of the method to enable/disable this is
Even though this is most useful from the point of view of metrics but you don't need a Also, even if you remove metrics from the registry, you might still have this issue if that data is published to your metrics backend. Unless you also remove the data from your backend too, I would say the validation is not a false trigger, the problem it validates still exists. |
I'm lacking context around what you're doing and why. It sounds like you're working around an issue, in which case it probably makes sense you need to disable this validation. If you are using the Prometheus registry, the limitation on registering meters with the same name but a different set of tag keys has been lifted in 1.17.0 (see the migration guide). That said, it is still best practice to avoid that. If you have a use case that you think is an exception to that, we'd be interested to learn more about it. |
|
Thanks for the responses! I can see how there's a layering distinction at play here and This is indeed in the context of a Prometheus registry, so the upcoming changes in 1.17 are interesting. We allow for observation keys to be extended through configuration, such that application refreshes may result in a different bag of observation keys;. These changes are expected to be non-frequent and a refresh listener explicitly purges the meter registry for affected meters to avoid the unsupported change in keys in Prometheus' Java client. The test that verifies this behavior therefore was affected by this validation, as the So perhaps it is expected to use the option to disable this validation in this use-case, acknowledging the "risks" that observations with changing keys may have for any handler. So please see this as a data point as to how this validation may affect real world tests and the relevancy of the opt-out flag for future consideration. Thanks for all your effort in maintaining Micrometer! |
I think this can be a valid feature: when you call |
In micrometer-metricsgh-7295 a ThreadLocal was introduced and elements are only added to it, never removed. Also, it was reported in micrometer-metricsgh-6713 that in some cases reusing TestObservationRegistry can trigger validation issues since the validator remembers all the low cardinality key-values that were recorded previously. Both of the issues can be worked around by creating a new TestObservationRegistry instance, however in some cases clearing its memory might be more convenient. See micrometer-metricsgh-7295 See micrometer-metricsgh-6713
In gh-7295 a ThreadLocal was introduced and elements are only added to it, never removed. Also, it was reported in gh-6713 that in some cases reusing TestObservationRegistry can trigger validation issues since the validator remembers all the low cardinality key-values that were recorded previously. Both of the issues can be worked around by creating a new TestObservationRegistry instance, however in some cases clearing its memory might be more convenient. See gh-7295 See gh-6713
|
Thanks, and apologies for the late reply; this was still pending in my inbox. Thanks for #7337, that ought to do it. |
Observations with the same name should have the same set of low cardinality keys.
Example 1:
Example 1 is valid since all the observations with the same name ("test") has the same set of low cardinality keys ("color").
Example 2:
Example 2 is invalid since the second observation is missing a key the first one has ("color") and the third observation is not only missing a key the first one has ("color") but it also has an extra key the first one does not have ("status").
If the validator detects this, it will give you an error like this:
In order to fix it, you should always add the same set of keys to the Observation. If some data is missing, you can signal it with a
"none"/"unknown"/etc. value:In your codebase, this means if you attach keyvalues conditionally:
you should only apply the condition to the value only:
If you want to turn this validation off (not recommended), you can do it this way: