Skip to content

[llm bench] Add mem info on initial/compilation phase to json/csv - #2741

Merged
sbalandi merged 3 commits into
openvinotoolkit:masterfrom
sbalandi:start_mem_json
Sep 19, 2025
Merged

sbalandi merged 3 commits into
openvinotoolkit:masterfrom
sbalandi:start_mem_json

Conversation

@sbalandi

@sbalandi sbalandi commented Sep 17, 2025 •

Copy link
Copy Markdown
Contributor

PR related to CVS-171619 and #2686 . Initial memory info was added to output, but wasn't added to csv/json output. Therefore, collecting the information is added as part of this PR.

@github-actions github-actions Bot added the category: llm_bench Label for tool/llm_bench folder label Sep 17, 2025
@sbalandi
sbalandi force-pushed the start_mem_json branch 2 times, most recently from 3c91043 to fefc7bc Compare September 17, 2025 12:30
@as-suvorov
as-suvorov requested a review from Copilot September 17, 2025 14:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds compilation phase memory information to CSV and JSON output formats for the LLM benchmark tool. The changes extend memory monitoring capabilities to include initial memory state and memory consumption during model compilation/initialization phases.

  • Refactored memory monitoring from memory_monitor to memory_data_collector for better organization
  • Added compilation phase memory data collection with initial memory state tracking
  • Extended CSV and JSON output formats to include compilation memory metrics

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tools/llm_bench/llm_bench_utils/pt_utils.py Updated parameter names from memory_monitor to memory_data_collector across all model creation functions
tools/llm_bench/llm_bench_utils/ov_utils.py Updated parameter names from memory_monitor to memory_data_collector across all model creation functions
tools/llm_bench/llm_bench_utils/output_json.py Added compilation memory metrics to JSON output including initial memory state and compilation phase memory consumption
tools/llm_bench/llm_bench_utils/output_csv.py Added compilation memory metrics to CSV output with updated headers and comments
tools/llm_bench/llm_bench_utils/memory_monitor.py Introduced MemoryDataCollector class and MemStatus class for better memory data organization
tools/llm_bench/benchmark.py Updated to use MemoryDataCollector instead of direct memory_monitor usage

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@sbalandi
sbalandi force-pushed the start_mem_json branch 3 times, most recently from 6e368d8 to cb1b238 Compare September 18, 2025 10:24
@Wovchena
Wovchena requested a review from Copilot September 18, 2025 13:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

self.sys = sys


class MemoryDataCollector():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Collector seems to be too general name. Every nested Memory* class name could have that word. Maybe Summarizer is a better word to give a hint about hierarchy

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed

@Wovchena
Wovchena requested a review from Copilot September 18, 2025 17:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.


def write_result(report_file, model, framework, device, model_args, iter_data_list, pretrain_time, model_precision, iter_timestamp):
def write_result(report_file, model, framework, device, model_args, iter_data_list, pretrain_time, model_precision, iter_timestamp, memory_data_collector):
mem_unit = memory_data_collector.memory_monitor.memory_unit if memory_data_collector else MemoryDataSummarizer.DEF_MEM_UNIT

Copilot AI Sep 18, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line accesses nested attributes deeply (memory_data_collector.memory_monitor.memory_unit). Consider adding a method to MemoryDataSummarizer to get the memory unit directly for better encapsulation.

Suggested change
mem_unit = memory_data_collector.memory_monitor.memory_unit if memory_data_collector else MemoryDataSummarizer.DEF_MEM_UNIT
mem_unit = memory_data_collector.get_memory_unit() if memory_data_collector else MemoryDataSummarizer.DEF_MEM_UNIT

Copilot uses AI. Check for mistakes.
@sbalandi
sbalandi enabled auto-merge September 19, 2025 10:24
@sbalandi
sbalandi added this pull request to the merge queue Sep 19, 2025
Merged via the queue into openvinotoolkit:master with commit 806e812 Sep 19, 2025
AsyaPronina pushed a commit to eshiryae/openvino.genai that referenced this pull request Oct 22, 2025
…envinotoolkit#2741)

PR related to
[CVS-171619](https://jira.devtools.intel.com/browse/CVS-171619) and
openvinotoolkit#2686 . Initial
memory info was added to output, but wasn't added to csv/json output.
Therefore, collecting the information is added as part of this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: llm_bench Label for tool/llm_bench folder

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants