Repository navigation
feat(benchmark): add CVTG-2K and LongText-Bench text-rendering evaluation - #2103
yinhe-likehong wants to merge 1 commit into
Conversation
…tion (modelscope#1929) Add the Text-Rendering slice of X2I evaluation support: - CVTG-2K (TextCrafter, arXiv:2503.23461): Word Accuracy and word-level NED against the prompt's quoted text regions, scored via PaddleOCR - LongText-Bench (X-Omni, arXiv:2507.22058): Text Score as a Counter intersection of VLM-transcribed text vs the ground truth, word-level for English and character-level for Chinese Both scoring cores are pure functions mirroring the official evaluators, with lazy OCR/VLM backends and a stub mode for pipeline tests. Also fix a double-index bug in Text2ImageAdapter.match_score that scored any non-image-pair metric as an error. Tests: 16 new tests covering the scoring cores against official results, adapter mapping, and a full pipeline run (local jsonl + local image service + stub backends). Verified PaddleOCR scoring and real dataset field mapping on a CPU box; the VLM path needs a GPU environment.
Real-environment verification dataBeyond the CI tests, the scoring paths and dataset mappings were verified against real data on a CPU box (openEuler 24.03, Python 3.11.6, offline wheels: 1. Real PaddleOCR scoring (CVTG-2K backend)
The partial-score case confirms the metric degrades gracefully instead of saturating, matching the official evaluator behavior. 2. Real dataset field mapping
The list-valued 3. Environment notes
|
Yunnglin
left a comment
There was a problem hiding this comment.
Requesting changes because the real LongText scoring path fails and several scoring rules differ from the official evaluators.
-
[P1] Fix the non-stub LongText transcription path.
Inmetrics.py:536,evalscope.utils.utilsdoes not exist. Entering_transcribe()with model loading bypassed reproducesModuleNotFoundError; with the model loaded normally, execution would reach the same invalid import. The adapter catches this exception and records a zero score. Please remove the unused import and add coverage that exercises the non-stub transcription path. -
[P1] Preserve the official aggregation rules.
The new scorers return per-image ratios and inherit equal-weight image averaging. The official CVTG evaluator aggregates Word Accuracy and NED over all ground-truth words; the official LongText evaluator divides total matched tokens by total ground-truth tokens. With one fully correct one-word sample and one completely incorrect three-word sample, the current aggregation/report/serialization path produces 0.5, whereas the official result is approximately 0.25. Please retain the underlying counts/similarity sums and aggregate them according to each benchmark's protocol, without changing unrelated T2I metrics. Add a report-level regression with unequal text lengths and partial matches. -
[P2] Restore CVTG's official repeated-word matching.
text_render.py:53–55uses a Counter intersection, while the official evaluator checks each ground-truth word for membership in the recognized words without consuming occurrences. For ground truth['hello', 'hello']and recognition['hello'], the official score is 1.0, but this implementation returns 0.5. This affects real data: 135 of the 2,000 prompts contain repeated ground-truth words. Please match the official behavior and update the test that currently asserts the differing result. -
[P1] Select LongText tokenization per language and match Chinese normalization.
LongTextScoredefaults to a single globallang='en', and the adapter does not switch it for thezhsubset. For ground truth出口A换乘2号线and recognition出口A换乘2号, the official Chinese score is 0.875, but the default implementation returns 0. Even withlang='zh', normalization retains whitespace that the official evaluator removes:出口 Aversus出口Ascores 0.75 instead of 1.0. Please select the language from the current sample/subset, support both languages in the same run, and use the official Chinese preprocessing.
The 16 new tests pass, but the pipeline test uses local fixtures and ground-truth-echoing backends, so it does not cover these failures. This review did not independently run PaddleOCR or GPU VLM inference.
Before merge, please also declare an explicit evaluation_version for both new benchmarks, complete the required description sections (Key Features and the task Domain field), and fix the Ruff formatting failure in aliases.py. Maintainers can handle the generated metadata/documentation step.
Maintainer-owned follow-up: the configured datasets are currently unavailable on ModelScope. Maintainers will handle dataset publication/mirroring; this is not an author action item. We should verify the real dataset-loading path against those sources before merging.
Summary
Implements the Text-Rendering slice of X2I evaluation support from #1929:
TextCrafter_Eval/unified_metrics_eval.py.textbench/summary_scores.py.Implementation
evalscope/metrics/vision/text_render.py) aligned with the official evaluators, so they are unit-testable without heavy dependencies.backend='stub'echoes the ground truth so pipeline tests run on CPU-only CI.LongTextScorereuses the image-pair metric channel to pass the ground-truth text (the datasettextfield, space-joined when stored as a list) to the scorer.textfield is a list of rendered fragments in the real data; the adapter joins it with spaces as the official evaluator does.Bug fix
Text2ImageAdapter.match_scoredouble-indexedmetric_func(image_path, prompt)[0]althoughT2IMetric.__call__already reducesapply()'s list to the single-sample value; every non-image-pair metric hitTypeError/KeyErrorand was silently recorded aserror/0. Removed the extra index.Tests
16 new tests (
tests/benchmark/test_x2i_text_rendering.py): 13 scoring-core unit tests checked against official evaluator results, 2 adapter mapping tests, and a full-pipeline test (local jsonl + local OpenAI-images service + stub backends) asserting report scores and dimension aggregation.Additionally verified outside CI on a CPU box (openEuler 24.03, paddlepaddle 2.6.2-cpu + paddleocr 2.7.3 with official PP-OCRv4/v3 models):
The Qwen2.5-VL transcription backend needs a GPU environment and was not exercised end-to-end.
Closes #1929 (partially: GenEval2, Qwen-Image-Bench, OneIG, DPG-Bench, MJHQ-30K remain as follow-ups)
🤖 Generated with kscc