Repository navigation
[llm_bench] Add per-stage performance metrics for WhisperPipeline and sampling duration for LLMPipeline - #3669
Merged
sbalandi merged 38 commits intoMay 29, 2026
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends GenAI performance instrumentation by adding sampling-stage duration tracking to common PerfMetrics, and adds encoder/decoder inference durations to WhisperPerfMetrics, then updates llm_bench and Python bindings/tests to surface and validate these metrics.
Changes:
- Added
RawPerfMetrics::m_sampling_durations+PerfMetrics::get_sampling_duration()and collected sampling timings across static LLM, SDPA, continuous batching, and Whisper pipelines. - Added
WhisperRawPerfMetrics::{encode_inference_durations, decode_inference_durations}+ correspondingWhisperPerfMetricsgetters/statistics. - Updated
llm_benchto print per-stage Whisper latencies and extended Python tests to validate the new metrics.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/llm_bench/task/speech_to_text_generation.py | Extracts Whisper per-stage metrics (tokenization/features/encode/decode/sampling/detokenization) for reporting. |
| tools/llm_bench/llm_bench_utils/metrics_print.py | Adds whisper_genai reporting path and a new per-stage Whisper latency printer. |
| tests/python_tests/test_whisper_pipeline.py | Extends Whisper perf metrics test to assert encode/decode/sampling metrics are present and consistent with raw counters. |
| tests/python_tests/test_llm_pipeline.py | Extends LLM perf metrics test to validate sampling duration statistics vs raw counters. |
| src/python/py_whisper_pipeline.cpp | Exposes new Whisper raw metrics fields + new WhisperPerfMetrics getters to Python. |
| src/python/py_perf_metrics.cpp | Exposes sampling_durations and get_sampling_duration() to Python. |
| src/python/openvino_genai/py_openvino_genai.pyi | Updates Python stubs for new perf metrics APIs/properties. |
| src/cpp/src/whisper/whisper_utils.hpp | Adds helpers to record extra per-infer durations and to filter additional per-token metrics. |
| src/cpp/src/whisper/whisper_utils.cpp | Implements new helpers (but currently contains a duplicate function definition causing a compile error). |
| src/cpp/src/whisper/pipeline_static.cpp | Collects encode/decode inference durations and sampling durations in Whisper static pipeline. |
| src/cpp/src/whisper/perf_metrics.cpp | Computes mean/std for new Whisper encode/decode inference duration metrics and merges them in operator+. |
| src/cpp/src/perf_metrics.cpp | Computes sampling duration statistics and concatenates sampling durations in PerfMetrics::operator+. |
| src/cpp/src/lm_encoding.cpp | Tracks sampling duration around sampler.sample() in SDPA backend. |
| src/cpp/src/llm/pipeline_static.cpp | Tracks sampling duration around m_sampler.sample() in static LLM pipeline. |
| src/cpp/src/continuous_batching/pipeline_impl.cpp | Records sampling duration per step and stores it into raw perf counters. |
| src/cpp/include/openvino/genai/whisper_pipeline.hpp | Adds new Whisper raw perf counters and WhisperPerfMetrics getters/fields. |
| src/cpp/include/openvino/genai/perf_metrics.hpp | Adds sampling durations to raw metrics and exposes get_sampling_duration(). |
| src/cpp/include/openvino/genai/continuous_batching_pipeline.hpp | Extends pipeline metrics with per-step sampling duration for continuous batching. |
goyaladitya05
marked this pull request as ready for review
April 8, 2026 12:21
goyaladitya05
requested review from
Wovchena,
as-suvorov,
pavel-esir,
popovaan,
sbalandi and
sgonorov
as code owners
April 8, 2026 12:21
Contributor
Author
Contributor
|
build_jenkins |
as-suvorov
requested changes
May 19, 2026
goyaladitya05
force-pushed
the
feature/whisper-sampling-perf-metrics
branch
from
May 19, 2026 18:48
ec5ef1e to
81474c3
Compare
Contributor
Author
|
@as-suvorov @sbalandi I've updated the changes, please review |
Contributor
Author
|
Hey @sbalandi, all of these failures are unreleted to my changes. |
as-suvorov
approved these changes
May 26, 2026
sbalandi
approved these changes
May 26, 2026
Contributor
|
build_jenkins |
Contributor
|
build_jenkins |
Contributor
|
@goyaladitya05 the pull request is merged. Thanks for your contribution and patient iterative work to improve the commit ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Enables llm_bench to report separate latencies for each WhisperPipeline stage and adds sampling duration tracking across all LLM pipeline backends.
Changes
Built locally and verified.
Closes: #3320
Checklist: