You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Enhance sending queue multi batching to be bounded. #14526
In current sending_queue multi partitioner. No of partitions we can create is not bounded which can be an issue in cases of high cardinality partition keys and can result in OOM in collector. Wanted to make sure size does not grow unbounded.
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
How are you partitioning? What would you expect to happen if the maximum number of partitions is reached, and a new partition would be created?
The config to multi batching is not yet exposed. One of the goals is to expose it and part of the config will be max number of partitions/key allowed. And When this limit is reached we can have eviction strategy depending on it we decide how to evict a key before adding to new key.
Activity
axw commentedon Feb 4, 2026
How are you partitioning?
What would you expect to happen if the maximum number of partitions is reached, and a new partition would be created?
vengalraoguttha commentedon Feb 4, 2026
The config to multi batching is not yet exposed. One of the goals is to expose it and part of the config will be max number of partitions/key allowed. And When this limit is reached we can have eviction strategy depending on it we decide how to evict a key before adding to new key.
axw commentedon Feb 5, 2026
FYI it is possible through an experimental interface. Here's how elasticsearchexporter uses it: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/1c8c65499049c7ea058aabb563b23c1b90236ac2/exporter/elasticsearchexporter/factory.go#L111-L116
That sounds sensible to me, so long as the evicted partition is flushed. Do you have an idea in mind for what the eviction strategy should be? LRU?
[exporterhelper][queuebatcher] Implement limiting No of partition in …
[exporterhelper][queuebatcher] Implement limiting No of partition in …
[exporterhelper][queuebatcher] Remove unused partitions from memory a…
[exporterhelper][queuebatcher] Implement limiting No of partition in …
[exporterhelper][queuebatcher] Remove unused partitions from memory a…
[exporterhelper][queuebatcher] Implement limiting No of partition in …
[exporterhelper][queuebatcher] Remove unused partitions from memory a…
[exporterhelper][queuebatcher] Add config for batching cache size and…
[exporterhelper] Move partition config defaults to NewDefaultBatchCon…