Repository navigation
[llm bench] Add mem info on initial/compilation phase to json/csv - #2741
Conversation
3c91043 to
fefc7bc
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR adds compilation phase memory information to CSV and JSON output formats for the LLM benchmark tool. The changes extend memory monitoring capabilities to include initial memory state and memory consumption during model compilation/initialization phases.
- Refactored memory monitoring from
memory_monitortomemory_data_collectorfor better organization - Added compilation phase memory data collection with initial memory state tracking
- Extended CSV and JSON output formats to include compilation memory metrics
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/llm_bench/llm_bench_utils/pt_utils.py | Updated parameter names from memory_monitor to memory_data_collector across all model creation functions |
| tools/llm_bench/llm_bench_utils/ov_utils.py | Updated parameter names from memory_monitor to memory_data_collector across all model creation functions |
| tools/llm_bench/llm_bench_utils/output_json.py | Added compilation memory metrics to JSON output including initial memory state and compilation phase memory consumption |
| tools/llm_bench/llm_bench_utils/output_csv.py | Added compilation memory metrics to CSV output with updated headers and comments |
| tools/llm_bench/llm_bench_utils/memory_monitor.py | Introduced MemoryDataCollector class and MemStatus class for better memory data organization |
| tools/llm_bench/benchmark.py | Updated to use MemoryDataCollector instead of direct memory_monitor usage |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
6e368d8 to
cb1b238
Compare
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
cb1b238 to
321207a
Compare
| self.sys = sys | ||
|
|
||
|
|
||
| class MemoryDataCollector(): |
There was a problem hiding this comment.
Collector seems to be too general name. Every nested Memory* class name could have that word. Maybe Summarizer is a better word to give a hint about hierarchy
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
|
||
| def write_result(report_file, model, framework, device, model_args, iter_data_list, pretrain_time, model_precision, iter_timestamp): | ||
| def write_result(report_file, model, framework, device, model_args, iter_data_list, pretrain_time, model_precision, iter_timestamp, memory_data_collector): | ||
| mem_unit = memory_data_collector.memory_monitor.memory_unit if memory_data_collector else MemoryDataSummarizer.DEF_MEM_UNIT |
There was a problem hiding this comment.
This line accesses nested attributes deeply (memory_data_collector.memory_monitor.memory_unit). Consider adding a method to MemoryDataSummarizer to get the memory unit directly for better encapsulation.
| mem_unit = memory_data_collector.memory_monitor.memory_unit if memory_data_collector else MemoryDataSummarizer.DEF_MEM_UNIT | |
| mem_unit = memory_data_collector.get_memory_unit() if memory_data_collector else MemoryDataSummarizer.DEF_MEM_UNIT |
680dca1 to
4a1b5b0
Compare
…envinotoolkit#2741) PR related to [CVS-171619](https://jira.devtools.intel.com/browse/CVS-171619) and openvinotoolkit#2686 . Initial memory info was added to output, but wasn't added to csv/json output. Therefore, collecting the information is added as part of this PR.
PR related to CVS-171619 and #2686 . Initial memory info was added to output, but wasn't added to csv/json output. Therefore, collecting the information is added as part of this PR.