Repository navigation
[wwb] Add text generation in chat mode - #3472
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a new WWB task (text-chat-mode) intended to evaluate multi-turn chat-style text generation, integrating it into the CLI/model loader flow and introducing a dedicated evaluator and CLI test.
Changes:
- Add
text-chat-modeto WWB CLI/model loading and evaluator selection. - Introduce
ChatTextEvaluatorfor generating/scoring chat transcripts. - Add a CLI test covering the new mode.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/who_what_benchmark/whowhatbench/wwb.py | Adds new text-chat-mode option and generation hook; modifies GenAI text generation helper. |
| tools/who_what_benchmark/whowhatbench/whowhat_metrics.py | Adjusts similarity evaluation loop (currently with an extra, redundant encode call). |
| tools/who_what_benchmark/whowhatbench/model_loaders.py | Allows text-chat-mode to load via the text model loader. |
| tools/who_what_benchmark/whowhatbench/chat_text_evaluator.py | New evaluator implementing chat transcript generation and scoring. |
| tools/who_what_benchmark/whowhatbench/init.py | Exposes ChatTextEvaluator from the package. |
| tools/who_what_benchmark/tests/test_cli_text.py | Adds a CLI test for text-chat-mode. |
You can also share your feedback on Copilot code review. Take the survey.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.
You can also share your feedback on Copilot code review. Take the survey.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
You can also share your feedback on Copilot code review. Take the survey.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.
You can also share your feedback on Copilot code review. Take the survey.
| def test_text_chat_model(model_id, tmp_path): | ||
| temp_file_name = tmp_path / "gt.csv" | ||
| chat_model_path = convert_text_model(model_id, model_id.split("/")[1], _convert_base) | ||
|
|
||
| run_wwb( |
There was a problem hiding this comment.
Most WWB CLI tests guard macOS with if sys.platform == 'darwin': pytest.xfail("Ticket 173169"), but this new chat-mode test does not, which may reintroduce known Darwin CI failures. Consider adding the same xfail (and while here, remove the trailing whitespace on the blank line after convert_text_model(...)).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
You can also share your feedback on Copilot code review. Take the survey.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.
You can also share your feedback on Copilot code review. Take the survey.
| answers_prediction = data_prediction["answers"].values | ||
|
|
||
| # in question mode - gold, prediction are string | ||
| # in chat mode - gold, prediction are list of answers |
There was a problem hiding this comment.
Annotate the types with Pyrhon's feature
There was a problem hiding this comment.
added, but not sure if I understood your comment correctly, please correct me if I'm wrong: https://github.com/openvinotoolkit/openvino.genai/pull/3472/changes#diff-8c43fe52d7939c562560dbe8d27c842899b0865b625df9b5dd45aa58a04dd34dR37
There was a problem hiding this comment.
It shouldn't be a comment. https://stackoverflow.com/a/41641489/25242963
| from typing import Any, Union | ||
|
|
||
| import os |
There was a problem hiding this comment.
This new module is missing the standard repository copyright/SPDX header that appears at the top of other whowhatbench/*.py files (e.g., text_evaluator.py). Please add the # Copyright ... and # SPDX-License-Identifier: Apache-2.0 header for licensing consistency.
| from transformers import AutoTokenizer | ||
| from optimum.intel.openvino import OVModelForCausalLM, OVWeightQuantizationConfig | ||
|
|
||
| from test_cli_image import get_similarity |
There was a problem hiding this comment.
Importing get_similarity from test_cli_image couples this test module to another test file and executes test_cli_image module-level code (e.g., tempfile.mkdtemp()), creating side effects during collection/import. Please move get_similarity into conftest.py (or a small test utility module) and import it from there instead.
| from test_cli_image import get_similarity |
Description
Added possibility to run wwb in chat mode.
example to run:
CVS-166618
Checklist: