Skip to content

[wwb] Add text generation in chat mode - #3472

Merged
Wovchena merged 7 commits into
openvinotoolkit:masterfrom
sbalandi:chat_mode_wwb
Mar 24, 2026
Merged

Wovchena merged 7 commits into
openvinotoolkit:masterfrom
sbalandi:chat_mode_wwb

Conversation

@sbalandi

@sbalandi sbalandi commented Mar 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

Added possibility to run wwb in chat mode.

example to run:

optimum-cli export openvino -m TinyLlama/TinyLlama-1.1B-Chat-v1.0 ./model/TinyLlama-1.1B-Chat-v1.0
wwb --base-model ./model/TinyLlama-1.1B-Chat-v1.0  --model-type text-chat --gt-data chat-model.csv 
wwb --target-model ./model/TinyLlama-1.1B-Chat-v1.0 --model-type text-chat --gt-data chat-model.csv --output chat_mode

CVS-166618

Checklist:

  • This PR follows GenAI Contributing guidelines.
  • Tests have been updated or added to cover the new code.
  • This PR fully addresses the ticket.
  • I have made corresponding changes to the documentation.

Copilot AI review requested due to automatic review settings March 10, 2026 18:01
@github-actions github-actions Bot added the category: WWB PR changes WWB label Mar 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new WWB task (text-chat-mode) intended to evaluate multi-turn chat-style text generation, integrating it into the CLI/model loader flow and introducing a dedicated evaluator and CLI test.

Changes:

  • Add text-chat-mode to WWB CLI/model loading and evaluator selection.
  • Introduce ChatTextEvaluator for generating/scoring chat transcripts.
  • Add a CLI test covering the new mode.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
tools/who_what_benchmark/whowhatbench/wwb.py Adds new text-chat-mode option and generation hook; modifies GenAI text generation helper.
tools/who_what_benchmark/whowhatbench/whowhat_metrics.py Adjusts similarity evaluation loop (currently with an extra, redundant encode call).
tools/who_what_benchmark/whowhatbench/model_loaders.py Allows text-chat-mode to load via the text model loader.
tools/who_what_benchmark/whowhatbench/chat_text_evaluator.py New evaluator implementing chat transcript generation and scoring.
tools/who_what_benchmark/whowhatbench/init.py Exposes ChatTextEvaluator from the package.
tools/who_what_benchmark/tests/test_cli_text.py Adds a CLI test for text-chat-mode.

You can also share your feedback on Copilot code review. Take the survey.

Copilot AI review requested due to automatic review settings March 11, 2026 15:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.


You can also share your feedback on Copilot code review. Take the survey.

Comment thread tools/who_what_benchmark/README.md Resolved
Copilot AI review requested due to automatic review settings March 12, 2026 19:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.


You can also share your feedback on Copilot code review. Take the survey.

Copilot AI review requested due to automatic review settings March 12, 2026 21:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.


You can also share your feedback on Copilot code review. Take the survey.

Comment thread tools/who_what_benchmark/README.md Resolved
Comment on lines +281 to +285
def test_text_chat_model(model_id, tmp_path):
temp_file_name = tmp_path / "gt.csv"
chat_model_path = convert_text_model(model_id, model_id.split("/")[1], _convert_base)

run_wwb(

Copilot AI Mar 12, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most WWB CLI tests guard macOS with if sys.platform == 'darwin': pytest.xfail("Ticket 173169"), but this new chat-mode test does not, which may reintroduce known Darwin CI failures. Consider adding the same xfail (and while here, remove the trailing whitespace on the blank line after convert_text_model(...)).

Copilot uses AI. Check for mistakes.
Copilot AI review requested due to automatic review settings March 13, 2026 11:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.


You can also share your feedback on Copilot code review. Take the survey.

Comment thread tools/who_what_benchmark/README.md Resolved
@github-actions github-actions Bot added the category: GGUF GGUF file reader label Mar 13, 2026
Copilot AI review requested due to automatic review settings March 19, 2026 13:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.


You can also share your feedback on Copilot code review. Take the survey.

Comment thread tools/who_what_benchmark/README.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Comment thread tools/who_what_benchmark/README.md Resolved
answers_prediction = data_prediction["answers"].values

# in question mode - gold, prediction are string
# in chat mode - gold, prediction are list of answers

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Annotate the types with Pyrhon's feature

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added, but not sure if I understood your comment correctly, please correct me if I'm wrong: https://github.com/openvinotoolkit/openvino.genai/pull/3472/changes#diff-8c43fe52d7939c562560dbe8d27c842899b0865b625df9b5dd45aa58a04dd34dR37

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

Copilot AI review requested due to automatic review settings March 23, 2026 11:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Copilot AI review requested due to automatic review settings March 23, 2026 18:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

Copilot AI review requested due to automatic review settings March 24, 2026 12:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comment on lines +1 to +3
from typing import Any, Union

import os

Copilot AI Mar 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new module is missing the standard repository copyright/SPDX header that appears at the top of other whowhatbench/*.py files (e.g., text_evaluator.py). Please add the # Copyright ... and # SPDX-License-Identifier: Apache-2.0 header for licensing consistency.

Copilot uses AI. Check for mistakes.
from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM, OVWeightQuantizationConfig

from test_cli_image import get_similarity

Copilot AI Mar 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Importing get_similarity from test_cli_image couples this test module to another test file and executes test_cli_image module-level code (e.g., tempfile.mkdtemp()), creating side effects during collection/import. Please move get_similarity into conftest.py (or a small test utility module) and import it from there instead.

Suggested change
from test_cli_image import get_similarity

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in another pr

@Wovchena
Wovchena added this pull request to the merge queue Mar 24, 2026
Merged via the queue into openvinotoolkit:master with commit 9c52f3f Mar 24, 2026
121 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: WWB PR changes WWB

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants