Skip to content

Enhance sending queue multi batching to be bounded. #14526

Closed

Description

@vengalraoguttha

Component(s)

No response

Describe the issue you're reporting

In current sending_queue multi partitioner. No of partitions we can create is not bounded which can be an issue in cases of high cardinality partition keys and can result in OOM in collector. Wanted to make sure size does not grow unbounded.

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Activity

  1. axw commented on Feb 4, 2026

    @axw
    Contributor

    How are you partitioning?
    What would you expect to happen if the maximum number of partitions is reached, and a new partition would be created?

  2. vengalraoguttha commented on Feb 4, 2026

    @vengalraoguttha
    ContributorAuthor

    How are you partitioning? What would you expect to happen if the maximum number of partitions is reached, and a new partition would be created?

    The config to multi batching is not yet exposed. One of the goals is to expose it and part of the config will be max number of partitions/key allowed. And When this limit is reached we can have eviction strategy depending on it we decide how to evict a key before adding to new key.

  3. axw commented on Feb 5, 2026

    @axw
    Contributor

    The config to multi batching is not yet exposed.

    FYI it is possible through an experimental interface. Here's how elasticsearchexporter uses it: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/1c8c65499049c7ea058aabb563b23c1b90236ac2/exporter/elasticsearchexporter/factory.go#L111-L116

    And When this limit is reached we can have eviction strategy depending on it we decide how to evict a key before adding to new key.

    That sounds sensible to me, so long as the evicted partition is flushed. Do you have an idea in mind for what the eviction strategy should be? LRU?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions