Repository navigation
Description
Is your feature request related to a problem? Please describe.
Right now, the collector reloads configuration whenever it gets a notification from any of its configuration sources. In practice, this never happens, as watching is not implemented in any of the existing config providers, but it will the moment that is added.
I don't think this is behaviour users would universally want. In many contexts, you only want to reload the configuration on-demand. Kubernetes is one example of this, where mounted ConfigMaps can be remounted if they change, but it's not desirable behaviour for an application to reload the config without the Pod having been recreated. Running as a systemd unit is another example.
I'd like to figure out what is the best way of letting the user control how the collector reloads configuration. I'd like to explore different approaches - for example, many applications do this on receipt of a SIGHUP signal.
Describe the solution you'd like
I don't know what the right set of solutions is, but I think it should allow:
- the user to decide if they want automatic configuration reload based on provider notification, maybe per provider
- the user to be able to reload configuration on demand without restarting the collector process
In particular, I'm not sure what the default should be.
Describe alternatives you've considered
Some possible solutions:
- Let automatic reloading be controlled by command line options
for example:otelcol --config config.yaml --enable-hot-reload
- Let automatic reloading be controlled by a section in the configuration
service: config: hotReload: true
- unconditionally reload configuration on receipt of a SIGHUP signal
I'm open to other ideas.
Additional context
This originally came up because I wanted to add notifications to the fileprovider in #5945 . @TylerHelmuth suggested it should start off behind a feature flag, and I reconsidered whether it should even ultimately be enabled for all users. I looked at how this is done in some other agents, and this issue is the result.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
bogdandrutu commentedon Aug 29, 2022
I am confused about this, I don't think the remounted happens for the existing pods. Imagine an app crash, then restart will pick the new version? Can you investigate a bit more?
I think it is almost impossible in case of a restart for example, also it becomes impossible to know what version of the config etc. For use cases like this we should create a provider to a proper "configuration" system that deals with all these complications.
Where is this coming from? SIGHUP seem to mean other things, but I don't have that much experience with this signal.
What does it mean? Is this what we have right now?
Can you please re-work/clarify the goals section. Then we can clarify the options, because right now the "possible solutions" section is not clear that I can pick one of that and will solve this problem, it feels more or less a random list of possible things to do and combine that may get us to the "solution".
swiatekm commentedon Aug 30, 2022
I was also surprised, but it does absolutely happen: https://kubernetes.io/docs/concepts/configuration/configmap/#mounted-configmaps-are-updated-automatically. To make the thing even more confusing, whether this happens depends on the mount type as well. All in all, it's kind of a mess.
Either way, this was just an example of a situation where the user may explicitly not want to reload their config file when it changes.
It's conventionally used for configuration reloading by daemons, which aren't attached to a terminal and therefore have no use for it otherwise. Some examples from similar projects:
This is actually something I think we could easily and safely add. We already catch signals in the main collector loop and configuration reloading is already implemented.
swiatekm commentedon Aug 30, 2022
What we have right now is automatic reload based on notifications from the config provider. I want to let users have more control over this. Right now, there's not even a way to opt-out of it.
Sorry about that, this is not a very streamlined issue, and I'm not sure what the right set of solutions is. I want to give users more control over configuration reloading, and I'd like to see what options are on the table for that.
portertech commentedon Aug 30, 2022
@swiatekm-sumo you thinking something like this? sensu@995cba5
portertech commentedon Aug 30, 2022
Works https://gist.github.com/portertech/63a33aca67bc0468120abc924eda49e3
bogdandrutu commentedon Aug 30, 2022
@portertech I would prefer to not jump and add more "legacy" things to the collector. We should define the "solution" first then add more functionality.
bogdandrutu commentedon Sep 9, 2022
@swiatekm-sumo I thought more also about this, I think we should do the following:
codeboten commentedon Sep 9, 2022
+1 to @bogdandrutu's suggestions. And from the examples stated above, I do believe using SIGHUP is appropriate here. See the following excerpt from the wikipedia entry https://en.wikipedia.org/wiki/Signal_(IPC)#SIGHUP:
swiatekm commentedon Sep 19, 2022
Technically it can, as we read the config normally before starting the main collector loop, unless I'm misunderstanding something. Command-line switch is probably the better choice in general though.
I can try to add this if we have agreement on the mechanism.
Do we need this to be configurable? I can't think of any circumstance where you'd need to disable it. There's no reason to send this signal to the collector other than this functionality.
bogdandrutu commentedon Sep 19, 2022
I think it is fine, since if it causes troubles we can have a flag/config to disable it.
swiatekm commentedon Sep 20, 2022
Cool, in that case we have an implementation ready in #6000. CC @portertech
bogdandrutu commentedon Sep 21, 2022
@swiatekm-sumo I think that is just the initial part. There are more to clarify, e.g. what is the behavior/interaction with watchers?
swiatekm commentedon Sep 21, 2022
I'd just go with the naive solution for now - we reload on every reload event (be it from a watcher or a signal). If we get an event during a reload, we finish the reload and then reload again. And we don't care if the config actually changed or not.
We could try to be clever about it and implement some kind of throttling, but that seems like premature optimization to me. Ultimately all of these events should be controlled by the user, and it should be on them not to overload the collector with signals or configuration changes. At the very least I'd wait until someone complains before thinking about mitigations for hypothetical problems.
The other benefit is that the naive implementation (as seen in #6000) is both simple and easy to understand.
swiatekm commentedon Mar 15, 2023
Summary of recent sig discussion re: config reloading:
@tigrannajaryan I was told you had some thoughts on this as well. @dmitryax you mentioned you wanted to make hot-reloading configurable per config source - how would that work in practice?
TylerHelmuth commentedon Mar 15, 2023
I was referring to this issue, but it is fixed now.
MaciejGrynda commentedon Nov 3, 2023
Hello,
Last update for this was over a year ago, did something change? SIGHUP is still supported, but it would be great if we had OS independent functionality.
Something like exposing REST API to force reload or definig a file to watch for modification timestamp change
swiatekm commentedon Nov 3, 2023
@MaciejGrynda I think the consensus was that automatically reloading whenever config files changed wasn't something we wanted to do, but mechanisms where the user explicitly requests the reload (like SIGHUP) were fine. I suggest you open a new issue if you want one of those. I know some other CNCF projects take the REST API approach, so that would be an argument in favor.
vikrambe commentedon Dec 17, 2023
Hello, Do we have a plan to support dynamic reload capability soon ? I see a lot of interest and dangling threads on this topic. Here are few question i have
@bogdandrutu @tigrannajaryan Do suggest ?
agardnerIT commentedon Nov 11, 2024
Reading through this and other threads and I'm not 100% sure on the status here, but one more option is feature flagging as changing config at runtime is precisely what feature flags are designed to do.
flagd is already part of the OpenTelemetry demo app so is well understood (and as a CNCF project & part of OpenFeature, trusted).
flagd has a Go in-process evaluator which means the collector could simply read a JSON / YAML file to achieve this
If anyone is at Kubecon this week, ping me on the CNCF slack and we can catch up about this.
[RFC] Partial reload support (open-telemetry#14640)
[RFC] Partial reload support (#14640)
[RFC] Partial reload support (open-telemetry#14640)
[service] Skip nodes without a built component in Graph.ShutdownAll (o…
[service] Rebuild all nodes of a receiver that shares one instance ac…