vllm.utils ¶

Modules:

• argparse_utils –

  Argument parsing utilities for vLLM.

• async_utils –

  Contains helpers related to asynchronous code.

• b12x –

  Accessors for the optional b12x package.

• cache –
• collection_utils –

  Contains helpers that are applied to collections.

• counter –
• cpu_resource_utils –
• cpu_triton_utils –

  Contains replacement functions to fallback Triton usages in CPU backend

• deep_gemm –

  Compatibility wrapper for DeepGEMM API changes.

• flashinfer –

  Compatibility wrapper for FlashInfer API changes.

• flashinfer_moe_ep –

  FlashInfer moe_ep helpers for DeepSeek V4 vLLM integration.

• func_utils –

  Contains helpers that are applied to functions.

• gc_utils –
• gpu_sync_debug –

  Detect unintended GPU<->CPU syncs in the hot path.

• hashing –
• hpc –

  Compatibility wrapper for HPC API changes.

• humming –

  Lazy facade for the optional humming package.

• import_utils –

  Contains helpers related to importing modules.

• jit_monitor –

  Monitor unexpected kernel JIT compilation during inference.

• jsontree –

  Helper functions to work with nested JSON structures.

• math_utils –

  Math utility functions for vLLM.

• mem_constants –
• mem_utils –
• mistral –

  Provides lazy import of the vllm.tokenizers.mistral module.

• multi_stream_utils –
• nccl –
• network_utils –
• numa_utils –

  NUMA binding utilities for vLLM worker processes.

• nvtx_pytorch_hooks –
• ompmultiprocessing –

  OMP Aware Multiprocessing manager for running multiprocessing.Process()

• platform_utils –
• registry –
• serial_utils –
• sparse_utils –

  Thread-safe wrapper for sparse tensor invariant validation.

• system_utils –
• tensor_schema –
• torch_utils –

Functions:

• length_from_prompt_token_ids_or_embeds –

  Calculate the request length (in number of tokens) give either

15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

def length_from_prompt_token_ids_or_embeds(
    prompt_token_ids: list[int] | torch.Tensor | None,
    prompt_embeds: torch.Tensor | None,
) -> int:
    """Calculate the request length (in number of tokens) give either
    prompt_token_ids or prompt_embeds.
    """
    prompt_token_len = None if prompt_token_ids is None else len(prompt_token_ids)
    prompt_embeds_len = None if prompt_embeds is None else len(prompt_embeds)

    if prompt_token_len is None:
        if prompt_embeds_len is None:
            raise ValueError("Neither prompt_token_ids nor prompt_embeds were defined.")
        return prompt_embeds_len
    else:
        if prompt_embeds_len is not None and prompt_embeds_len != prompt_token_len:
            raise ValueError(
                "Prompt token ids and prompt embeds had different lengths"
                f" prompt_token_ids={prompt_token_len}"
                f" prompt_embeds={prompt_embeds_len}"
            )
        return prompt_token_len