Skip to content

feat(benchmark): add CVTG-2K and LongText-Bench text-rendering evaluation - #2103

Open
yinhe-likehong wants to merge 1 commit into
modelscope:mainfrom
yinhe-likehong:feat/x2i-text-rendering
Open

yinhe-likehong wants to merge 1 commit into
modelscope:mainfrom
yinhe-likehong:feat/x2i-text-rendering

Conversation

@yinhe-likehong

Copy link
Copy Markdown

Summary

Implements the Text-Rendering slice of X2I evaluation support from #1929:

  • CVTG-2K (TextCrafter, arXiv:2503.23461): 2,000 prompts with 2-5 quoted text regions per prompt. Scores the generated image with Word Accuracy (exact word match) and word-level NED (best-match normalized Levenshtein distance) via PaddleOCR, mirroring TextCrafter_Eval/unified_metrics_eval.py.
  • LongText-Bench (X-Omni, arXiv:2507.22058): 160 en + 160 zh prompts across 8 text-rich scenes. Text Score as a Counter intersection of VLM-transcribed text vs ground truth, word-level for English and character-level for Chinese, mirroring textbench/summary_scores.py.

Implementation

  • Scoring cores are pure functions (evalscope/metrics/vision/text_render.py) aligned with the official evaluators, so they are unit-testable without heavy dependencies.
  • OCR/VLM backends load lazily (PaddleOCR, Qwen2.5-VL via transformers); backend='stub' echoes the ground truth so pipeline tests run on CPU-only CI.
  • LongTextScore reuses the image-pair metric channel to pass the ground-truth text (the dataset text field, space-joined when stored as a list) to the scorer.
  • The LongText-Bench dataset text field is a list of rendered fragments in the real data; the adapter joins it with spaces as the official evaluator does.

Bug fix

Text2ImageAdapter.match_score double-indexed metric_func(image_path, prompt)[0] although T2IMetric.__call__ already reduces apply()'s list to the single-sample value; every non-image-pair metric hit TypeError/KeyError and was silently recorded as error/0. Removed the extra index.

Tests

16 new tests (tests/benchmark/test_x2i_text_rendering.py): 13 scoring-core unit tests checked against official evaluator results, 2 adapter mapping tests, and a full-pipeline test (local jsonl + local OpenAI-images service + stub backends) asserting report scores and dimension aggregation.

Additionally verified outside CI on a CPU box (openEuler 24.03, paddlepaddle 2.6.2-cpu + paddleocr 2.7.3 with official PP-OCRv4/v3 models):

  • Real PaddleOCR scoring: fully-rendered text scores 1.0/1.0 (WordAccuracy/NED); partially-rendered text scores 0.5/0.55.
  • Real dataset mapping: all 2,000 CVTG-2K prompts yield non-empty GT word extraction (16,206 GT words); all 320 LongText-Bench rows self-consistent at 1.0.

The Qwen2.5-VL transcription backend needs a GPU environment and was not exercised end-to-end.

Closes #1929 (partially: GenEval2, Qwen-Image-Bench, OneIG, DPG-Bench, MJHQ-30K remain as follow-ups)

🤖 Generated with kscc

…tion (modelscope#1929)

Add the Text-Rendering slice of X2I evaluation support:

- CVTG-2K (TextCrafter, arXiv:2503.23461): Word Accuracy and word-level
  NED against the prompt's quoted text regions, scored via PaddleOCR
- LongText-Bench (X-Omni, arXiv:2507.22058): Text Score as a Counter
  intersection of VLM-transcribed text vs the ground truth, word-level
  for English and character-level for Chinese

Both scoring cores are pure functions mirroring the official evaluators,
with lazy OCR/VLM backends and a stub mode for pipeline tests. Also fix
a double-index bug in Text2ImageAdapter.match_score that scored any
non-image-pair metric as an error.

Tests: 16 new tests covering the scoring cores against official results,
adapter mapping, and a full pipeline run (local jsonl + local image
service + stub backends). Verified PaddleOCR scoring and real dataset
field mapping on a CPU box; the VLM path needs a GPU environment.
@yinhe-likehong

Copy link
Copy Markdown
Author

Real-environment verification data

Beyond the CI tests, the scoring paths and dataset mappings were verified against real data on a CPU box (openEuler 24.03, Python 3.11.6, offline wheels: paddlepaddle 2.6.2-cpu + paddleocr 2.7.3 with the official PP-OCRv4/v3 inference models).

1. Real PaddleOCR scoring (CVTG-2K backend)

# Scenario OCR result (words) WordAccuracy NED Verdict
1 Rendered image containing all ground-truth text (HELLO WORLD + OPEN 24H) hello world open 24h (4/4) 1.0000 1.0000 ✅ perfect score
2 Partially rendered image (only HELLO WORLD drawn) hello world (2/4) 0.5000 0.5500 ✅ partial score in (0, 1)

The partial-score case confirms the metric degrades gracefully instead of saturating, matching the official evaluator behavior.

2. Real dataset field mapping

Dataset Real structure vs. implementation Verification
CVTG-2K (dnkdnk/CVTG-2K) CVTG/ + CVTG-Style/ × 2/3/4/5.json; records {index, prompt, carrier_list, sentence_list}, GT quoted in prompt matches (official evaluator also reads only index + prompt) 2,000/2,000 prompts yield non-empty GT extraction (16,206 GT words total)
LongText-Bench (X-Omni/LongText-Bench) text_prompts{,_zh}.jsonl, 160 rows each; fields {prompt_id, prompt, text, category, length, text_length}; text is a list of fragments mismatch found and fixed: the adapter space-joins the list like the official evaluator 320/320 rows self-consistent at Text Score = 1.0 (en + zh)

The list-valued text field was discovered during this verification and is handled in LongTextBenchAdapter.record_to_sample.

3. Environment notes

  • Paddle import segfaulted on openEuler 24.03 until OMP_NUM_THREADS=1 was set; the verification scripts pin it.
  • The Qwen2.5-VL transcription backend (LongText-Bench) requires a GPU environment and was not exercised end-to-end; the box used here has no NVIDIA GPU and 2 GB RAM.

@Yunnglin Yunnglin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes because the real LongText scoring path fails and several scoring rules differ from the official evaluators.

  1. [P1] Fix the non-stub LongText transcription path.
    In metrics.py:536, evalscope.utils.utils does not exist. Entering _transcribe() with model loading bypassed reproduces ModuleNotFoundError; with the model loaded normally, execution would reach the same invalid import. The adapter catches this exception and records a zero score. Please remove the unused import and add coverage that exercises the non-stub transcription path.

  2. [P1] Preserve the official aggregation rules.
    The new scorers return per-image ratios and inherit equal-weight image averaging. The official CVTG evaluator aggregates Word Accuracy and NED over all ground-truth words; the official LongText evaluator divides total matched tokens by total ground-truth tokens. With one fully correct one-word sample and one completely incorrect three-word sample, the current aggregation/report/serialization path produces 0.5, whereas the official result is approximately 0.25. Please retain the underlying counts/similarity sums and aggregate them according to each benchmark's protocol, without changing unrelated T2I metrics. Add a report-level regression with unequal text lengths and partial matches.

  3. [P2] Restore CVTG's official repeated-word matching.
    text_render.py:53–55 uses a Counter intersection, while the official evaluator checks each ground-truth word for membership in the recognized words without consuming occurrences. For ground truth ['hello', 'hello'] and recognition ['hello'], the official score is 1.0, but this implementation returns 0.5. This affects real data: 135 of the 2,000 prompts contain repeated ground-truth words. Please match the official behavior and update the test that currently asserts the differing result.

  4. [P1] Select LongText tokenization per language and match Chinese normalization.
    LongTextScore defaults to a single global lang='en', and the adapter does not switch it for the zh subset. For ground truth 出口A换乘2号线 and recognition 出口A换乘2号, the official Chinese score is 0.875, but the default implementation returns 0. Even with lang='zh', normalization retains whitespace that the official evaluator removes: 出口 A versus 出口A scores 0.75 instead of 1.0. Please select the language from the current sample/subset, support both languages in the same run, and use the official Chinese preprocessing.

The 16 new tests pass, but the pipeline test uses local fixtures and ground-truth-echoing backends, so it does not cover these failures. This review did not independently run PaddleOCR or GPU VLM inference.

Before merge, please also declare an explicit evaluation_version for both new benchmarks, complete the required description sections (Key Features and the task Domain field), and fix the Ruff formatting failure in aliases.py. Maintainers can handle the generated metadata/documentation step.

Maintainer-owned follow-up: the configured datasets are currently unavailable on ModelScope. Maintainers will handle dataset publication/mirroring; this is not an author action item. We should verify the real dataset-loading path against those sources before merging.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] X2I task eval support

2 participants