Skip to content

[llm_bench] Add per-stage performance metrics for WhisperPipeline and sampling duration for LLMPipeline - #3669

Merged
sbalandi merged 38 commits into
openvinotoolkit:masterfrom
goyaladitya05:feature/whisper-sampling-perf-metrics
May 29, 2026
Merged

sbalandi merged 38 commits into
openvinotoolkit:masterfrom
goyaladitya05:feature/whisper-sampling-perf-metrics

Conversation

@goyaladitya05

@goyaladitya05 goyaladitya05 commented Apr 8, 2026 •

Copy link
Copy Markdown
Contributor

Description

Enables llm_bench to report separate latencies for each WhisperPipeline stage and adds sampling duration tracking across all LLM pipeline backends.

Changes

  • PerfMetrics: added m_sampling_durations and get_sampling_duration(), collected for WhisperPipeline, static LLM, continuous batching, and SDPA pipelines
  • WhisperPerfMetrics: added encode_inference_durations / decode_inference_durations and get_encode_inference_duration / get_decode_inference_duration
  • llm_bench: reports per-stage Whisper latencies for GenAI pipelines:
    1. First token: tokenization, feature extraction, encode, decode
    2. Other tokens: decode avg
    3. Sampling: single avg across all tokens
    4. Detokenization
  • llm_bench hooks: added tm_sample_list to hook_greedy_search.py and all llm_hook_sample/hook_sample_v*.py files to time the token selection step.
  • test_perf_metrics in test_whisper_pipeline.py and test_llm_pipeline.py asserts new metrics are collected and consistent with their counters

Built locally and verified.

Closes: #3320

Checklist:

  • This PR follows GenAI Contributing guidelines.
  • Tests have been updated or added to cover the new code.
  • This PR fully addresses the ticket.
  • I have made corresponding changes to the documentation. NA, could not find any documentation for this. Updated the docstrings wherever applicable

Copilot AI review requested due to automatic review settings April 8, 2026 11:50
@github-actions github-actions Bot added category: llm_bench Label for tool/llm_bench folder category: visual language Visual language pipeline category: continuous batching Continuous batching category: LLM LLM pipeline (stateful, static) category: whisper Whisper pipeline category: Python API Python API for GenAI category: CPP API Changes in GenAI C++ public headers no-match-files category: GGUF GGUF file reader labels Apr 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends GenAI performance instrumentation by adding sampling-stage duration tracking to common PerfMetrics, and adds encoder/decoder inference durations to WhisperPerfMetrics, then updates llm_bench and Python bindings/tests to surface and validate these metrics.

Changes:

  • Added RawPerfMetrics::m_sampling_durations + PerfMetrics::get_sampling_duration() and collected sampling timings across static LLM, SDPA, continuous batching, and Whisper pipelines.
  • Added WhisperRawPerfMetrics::{encode_inference_durations, decode_inference_durations} + corresponding WhisperPerfMetrics getters/statistics.
  • Updated llm_bench to print per-stage Whisper latencies and extended Python tests to validate the new metrics.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tools/llm_bench/task/speech_to_text_generation.py Extracts Whisper per-stage metrics (tokenization/features/encode/decode/sampling/detokenization) for reporting.
tools/llm_bench/llm_bench_utils/metrics_print.py Adds whisper_genai reporting path and a new per-stage Whisper latency printer.
tests/python_tests/test_whisper_pipeline.py Extends Whisper perf metrics test to assert encode/decode/sampling metrics are present and consistent with raw counters.
tests/python_tests/test_llm_pipeline.py Extends LLM perf metrics test to validate sampling duration statistics vs raw counters.
src/python/py_whisper_pipeline.cpp Exposes new Whisper raw metrics fields + new WhisperPerfMetrics getters to Python.
src/python/py_perf_metrics.cpp Exposes sampling_durations and get_sampling_duration() to Python.
src/python/openvino_genai/py_openvino_genai.pyi Updates Python stubs for new perf metrics APIs/properties.
src/cpp/src/whisper/whisper_utils.hpp Adds helpers to record extra per-infer durations and to filter additional per-token metrics.
src/cpp/src/whisper/whisper_utils.cpp Implements new helpers (but currently contains a duplicate function definition causing a compile error).
src/cpp/src/whisper/pipeline_static.cpp Collects encode/decode inference durations and sampling durations in Whisper static pipeline.
src/cpp/src/whisper/perf_metrics.cpp Computes mean/std for new Whisper encode/decode inference duration metrics and merges them in operator+.
src/cpp/src/perf_metrics.cpp Computes sampling duration statistics and concatenates sampling durations in PerfMetrics::operator+.
src/cpp/src/lm_encoding.cpp Tracks sampling duration around sampler.sample() in SDPA backend.
src/cpp/src/llm/pipeline_static.cpp Tracks sampling duration around m_sampler.sample() in static LLM pipeline.
src/cpp/src/continuous_batching/pipeline_impl.cpp Records sampling duration per step and stores it into raw perf counters.
src/cpp/include/openvino/genai/whisper_pipeline.hpp Adds new Whisper raw perf counters and WhisperPerfMetrics getters/fields.
src/cpp/include/openvino/genai/perf_metrics.hpp Adds sampling durations to raw metrics and exposes get_sampling_duration().
src/cpp/include/openvino/genai/continuous_batching_pipeline.hpp Extends pipeline metrics with per-step sampling duration for continuous batching.

Comment thread src/cpp/src/whisper/whisper_utils.cpp Resolved
Copilot AI review requested due to automatic review settings April 8, 2026 12:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Comment thread src/cpp/src/whisper/pipeline_static.cpp Resolved
@goyaladitya05
goyaladitya05 marked this pull request as ready for review April 8, 2026 12:21
Copilot AI review requested due to automatic review settings April 8, 2026 12:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 7 comments.

Comment thread src/python/py_perf_metrics.cpp Resolved
Comment thread src/python/py_perf_metrics.cpp Resolved
Comment thread src/python/py_whisper_pipeline.cpp Resolved
Comment thread src/python/py_whisper_pipeline.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 7 comments.

Comment thread src/cpp/src/lm_encoding.cpp Resolved
Comment thread src/cpp/src/lm_encoding.cpp Resolved
Comment thread src/cpp/src/llm/pipeline_static.cpp Resolved
Comment thread src/cpp/src/llm/pipeline_static.cpp Resolved
Comment thread src/python/py_whisper_pipeline.cpp Resolved
@goyaladitya05
goyaladitya05 requested a review from Copilot April 8, 2026 19:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.

@goyaladitya05

goyaladitya05 commented May 15, 2026 •

Copy link
Copy Markdown
Contributor Author

@eshiryae @sbalandi, I've updated the changes.

@eshiryae eshiryae left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whisper part - LGTM

@sbalandi

Copy link
Copy Markdown
Contributor

build_jenkins

Comment thread src/cpp/src/llm/pipeline_static.cpp Resolved
Comment thread src/cpp/src/whisper/whisper_utils.cpp Resolved
@goyaladitya05
goyaladitya05 force-pushed the feature/whisper-sampling-perf-metrics branch from ec5ef1e to 81474c3 Compare May 19, 2026 18:48
Copilot AI review requested due to automatic review settings May 19, 2026 18:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.

@goyaladitya05
goyaladitya05 requested a review from as-suvorov May 19, 2026 19:03
@goyaladitya05

goyaladitya05 commented May 19, 2026 •

Copy link
Copy Markdown
Contributor Author

@as-suvorov @sbalandi I've updated the changes, please review

@goyaladitya05

Copy link
Copy Markdown
Contributor Author

Hey @sbalandi, all of these failures are unreleted to my changes.

Copilot AI review requested due to automatic review settings May 26, 2026 11:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@sbalandi

Copy link
Copy Markdown
Contributor

build_jenkins

@goyaladitya05 goyaladitya05 changed the title Add per-stage performance metrics for WhisperPipeline and sampling duration for LLMPipeline [llm_bench] Add per-stage performance metrics for WhisperPipeline and sampling duration for LLMPipeline May 26, 2026
@sbalandi

Copy link
Copy Markdown
Contributor

build_jenkins

@sbalandi
sbalandi added this pull request to the merge queue May 29, 2026
Merged via the queue into openvinotoolkit:master with commit cac68b9 May 29, 2026
172 checks passed
@sbalandi

Copy link
Copy Markdown
Contributor

@goyaladitya05 the pull request is merged. Thanks for your contribution and patient iterative work to improve the commit !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: continuous batching Continuous batching category: CPP API Changes in GenAI C++ public headers category: GGUF GGUF file reader category: llm_bench Label for tool/llm_bench folder category: LLM LLM pipeline (stateful, static) category: Python API Python API for GenAI category: visual language Visual language pipeline category: whisper Whisper pipeline no-match-files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand WhisperPipeline performance metrics and add support in llm_bench

5 participants