[mohammadaaftabv]

Description

Adds WhisperHallucinationStage, a CPU-only transcript filter for common ASR hallucination patterns. No model, GPU, or network is used; the stage applies four lexical checks to text produced by an upstream ASR stage.

The branch is rebased onto current main at 223f771.

Detection checks

Any single hit flags the row.

Check	Behavior
Repeated n-grams	Flags when the unique-word ratio is at or below unique_words_threshold.
Long word	Flags on the absolute long_word_threshold or the relative long_word_rel_threshold.
Phrase match	Matches the cleaned transcript against the bundled common-hallucination phrase corpus.
High char rate	Flags when word characters per second exceed max_char_rate.

For Finnish, Hungarian, Estonian, German, Dutch, Danish, Swedish, and Norwegian, the relative long-word check is skipped. The absolute long_word_threshold is now the single threshold for every language; there is no separate agglutinative_long_word_threshold.

The individual detector functions and setup() are AST-identical to nithinraok/Curator:nkoluguri/integration-test at 2963bf3. process() retains the reference self-setup fallback. As a simple CPU-only audio stage, it follows current Curator convention and defines only direct single-task processing.

YAML configuration and example

The runnable example is in tutorials/audio/whisper_hallucination/pipeline.yaml. From the Curator repository root:

python nemo_curator/config/run.py \
  --config-path ../../tutorials/audio/whisper_hallucination \
  --config-name pipeline \
  manifest_path=tests/fixtures/audio/text_filtering/whisper_hallucination_input.jsonl \
  output_path=/tmp/whisper_hallucination_output.jsonl

The YAML exposes:

• unique_words_threshold
• long_word_threshold
• long_word_rel_threshold
• max_char_rate
• overwrite
• recovery_value

_PREFIX_MATCH_MIN_LEN = 8 remains a ClassVar, not a YAML parameter. Lowering this safety boundary can make common short phrases such as you, Ja, or Да prefix-match unrelated transcripts.

The bundled phrases.txt is byte-identical to tutorials/audio/granary_v2_postprocessing/phrases.txt on the reference branch:

SHA-256 34ba2fcd7756f193e80ba4ac34a6b5db0dab92adeb0750beb796b1bf57f6bc42
size    1535 bytes

The pre-commit exclusions are scoped only to this frozen fixture because the reference bytes intentionally contain trailing spaces and no final newline.

Flag ownership and recovery

The stage writes "Hallucination:{name}" only when the row is unflagged or already has a hallucination flag, so a different filter's reason is preserved.

With overwrite=true, clean second-pass text clears an existing hallucination flag. recovery_value is a caller-selected note:

overwrite: true
recovery_value: "Recovered:ASR"

For a clean row previously marked Hallucination:PrimaryASR, the result is:

• _skipme: ""
• additional_notes.WhisperHallucination: "recovered:asr"

An empty recovery_value records "recovered".

Validation

• 39 focused tests pass.
• The refreshed paired local comparison at e0edecb produced byte-identical target/reference JSONL across all 21 frozen rows with zero failures.
• The stage keeps no per-worker processed/flagged aggregate counters.
• Pre-commit passes for all changed files.
• Detector helpers plus setup() are AST-identical to the reference implementation.
• Phrase files match under both cmp and SHA-256.
• The exact documented command completed a local Ray Data run over the checked-in two-row fixture: the clean row passed and Thank you. was flagged with phrase_match.

Checklist

• I am familiar with the Contributing Guide.
• New and existing tests cover the changes.
• The example YAML and executable command are included.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

The PR adds a CPU-only audio stage that flags likely Whisper hallucinations using repetition, long-word, phrase-corpus, and character-rate checks.

• Exposes the stage through the audio package and a runnable Hydra pipeline.
• Bundles a multilingual hallucination phrase corpus and focused detector, overwrite, recovery, and configuration tests.
• The remaining asymmetric hyphen handling misses an exact space-separated variant of a bundled phrase.

Confidence Score: 4/5

The PR should not merge until exact space-separated variants of hyphenated corpus phrases are detected without broadening continuation matching.

The new phrase matcher only derives a space alias from hyphenated transcripts, so the bundled Fifty-four phrase does not recognize an exact Fifty four transcript and silently leaves that known hallucination unflagged.

Files Needing Attention: nemo_curator/stages/audio/text_filtering/whisper_hallucination.py

Important Files Changed

Filename	Overview
nemo_curator/stages/audio/text_filtering/whisper_hallucination.py	Adds the detector stage and recovery behavior, but transcript-side hyphen aliases remain asymmetric for exact matches.
tests/stages/audio/text_filtering/test_whisper_hallucination.py	Adds broad focused coverage, including punctuation and prefix regressions, but omits the exact Fifty four variant.
tutorials/audio/whisper_hallucination/pipeline.yaml	Adds a runnable Ray Data pipeline exposing the stage's supported thresholds and recovery controls.
tutorials/audio/whisper_hallucination/phrases.txt	Adds the multilingual phrase corpus, including the hyphenated Fifty-four entry affected by asymmetric exact matching.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[AudioTask transcript] --> B[Repeated-word check]
  A --> C[Long-word check]
  A --> D[Phrase normalization and matching]
  A --> E[Character-rate check]
  B --> F{Any detector hit?}
  C --> F
  D --> F
  E --> F
  F -->|Yes| G[Set hallucination flag and note]
  F -->|No| H[Pass or recover prior hallucination flag]

Loading

_{Reviews (11): Last reviewed commit: "refactor(audio): use single-task halluci..." | Re-trigger Greptile}

[greptile-apps]

Phrase normalization misses common variants

The cleaned transcript remains case-sensitive and removes only periods, question marks, and exclamation marks, so equivalent ASR outputs such as lowercase thank you or comma-terminated Merci, do not match the corpus. Normalizing casing and punctuation consistently would prevent these common variants from reducing the filter's coverage.

Knowledge Base Used: Audio Stages

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[sarahyurick]

@mohammadaaftabv wdyt?

[mohammadaaftabv]

@nithinraok, could you also clarify the intended case and punctuation normalization for the hallucination phrase corpus?

The current [nkoluguri/integration-test](https://github.com/nithinraok/Curator/blob/nkoluguri/integration-test) implementation:

• stores corpus entries using only line.strip();
• remains case-sensitive;
• removes ., ?, and ! from the transcript;
• does not remove commas or hyphens.

As a result:

• Thank you. matches Thank you;
• thank you does not match Thank you;
• THANK YOU does not match Thank you;
• Merci, does not match Merci;
• Thank-you does not match Thank you.

One proposed alternative is to normalize both corpus phrases and transcripts by:

1. replacing Unicode punctuation with spaces;
2. applying Unicode-aware casefold();
3. collapsing whitespace;
4. requiring equality of the complete normalized transcript.

That alternative would make all of these equivalent:

• Thank you
• thank you
• THANK YOU!
• Thank-you

It would also make Merci, match Merci.

However, it changes the reference behavior and reduces the 101 raw entries to 99 unique normalized phrases because some entries collapse to the same normalized value.

Could you advise whether the case-sensitive and limited-.?! cleanup in the reference is deliberate based on observed Whisper output? Should this PR preserve exact reference parity, or should both implementations move to broader normalization?

For now, the PR preserves the reference behavior. The cleaned and reference text files load the same 101 stripped phrases, and fresh local execution shows 100% phrase-decision parity.

[nithinraok]

does not remove commas or hyphens.

Hi Aaftab, nice catch. lets also consider commas or hyphens and also update in nkoluguri/integration-test branch

[greptile-apps]

The current line needs two more .replace() calls. Here's the update:

Suggested change

	cleaned = text.strip().replace(".", "").replace("?", "").replace("!", "")
	if cleaned in self._phrases:
	cleaned = text.strip().replace(".", "").replace("?", "").replace("!", "").replace(",", "").replace("-", "")

This keeps parity with the limited-punctuation approach in the reference while closing the gaps for comma-terminated (Merci,) and hyphenated (Thank-you) variants. Once this is merged, the same change should land on nkoluguri/integration-test so both implementations stay in sync.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

[mohammadaaftabv]

@nithinraok , following up on the normalization discussion here: while implementing the requested comma/hyphen handling, I found one remaining asymmetry for ., ?, and !.
The current PR normalizes incoming transcripts with:

text.strip().replace(".", "").replace("?", "").replace("!", "").replace(",", "")

but corpus entries are normalized with only:

text.strip().replace(",", "")

Consequently, these bundled corpus phrases remain unreachable:

Това е първият въпрос. is stored with the final ., while the incoming candidate loses it.
¿Y qué tal? is stored with the final ?, while the incoming candidate loses it.

Direct execution currently allows both transcripts to pass instead of flagging them as phrase_match.
I propose the following constrained corpus-side change:
@staticmethod
def _normalize_corpus_phrase(text: str) -> str:
return (
text.strip()
.replace(".", "")
.replace("?", "")
.replace("!", "")
.replace(",", "")
)
This is intentionally narrower than the broader normalization originally discussed:
no casefold();
no Unicode-wide punctuation replacement;
no internal whitespace collapsing;
no change to prefix matching;
corpus hyphens remain preserved.

Preserving corpus hyphens is important because normalizing Fifty-four to Fifty four would make the prefix rule incorrectly flag valid text such as Fifty four dollars. The incoming side can continue checking both the original hyphenated transcript and a hyphen-to-space candidate, so Fifty-four and Thank-you remain handled without broadening corpus prefixes.
I would add regression coverage showing:
Това е първият въпрос. → flagged;
¿Y qué tal? → flagged;
Bonjour, je m'appelle Marie → still flagged;
Fifty-four → still flagged;
Fifty four dollars → still passes;
lowercase/case variants remain case-sensitive.

Would you approve this narrow .?!, corpus normalization, and should we apply the same change to nkoluguri/integration-test? Or should the PR retain the current reference behavior even though those two bundled phrases cannot be matched?

[greptile-apps]

The asymmetry is confirmed. The current _normalize_corpus_phrase only strips ,, while _transcript_candidates strips .?!,. Any corpus entry that ends with . or ? is unreachable.

Your proposed fix is correct and well-scoped. Here's the concrete suggestion:

Suggested change

	cleaned = text.strip().replace(".", "").replace("?", "").replace("!", "")
	if cleaned in self._phrases:
	@staticmethod
	def _normalize_corpus_phrase(text: str) -> str:
	return (
	text.strip()
	.replace(".", "")
	.replace("?", "")
	.replace("!", "")
	.replace(",", "")
	)

This closes the asymmetry for .?!, while intentionally leaving hyphens in corpus phrases, which is the right call — normalizing Fifty-four → Fifty four in the corpus would cause the prefix rule to incorrectly flag Fifty four dollars. The incoming side already handles hyphens via _transcript_candidates.

The two previously unreachable phrases (Това е първият въпрос. and ¿Y qué tal?) will now match correctly. The same change should go to nkoluguri/integration-test to keep both in sync, as @nithinraok noted.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

[nithinraok]

looks good to me, make same changes to nkoluguri/integration-test

[sarahyurick]

Suggested change

	common_hall_file: str = ""
	common_hall_file: str

[sarahyurick]

Bumping this.

[sarahyurick]

@mohammadaaftabv wdyt?

[sarahyurick]

Let's remove this block. Setup will always be called if the user is running the pipeline correctly.

[sarahyurick]

Suggested change

	# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
	# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.

[sarahyurick]

I think some of these tests are overkill.

[sarahyurick]

Can we remove these changes? I would rather the txt file not have trailing whitespaces, etc.

[mohammadaaftabv]

/ok to test e713723

[greptile-apps]

Hyphen normalization broadens prefix matches

When a valid transcript begins with the space-separated form of a hyphenated corpus phrase, such as Fifty four dollars, normalization converts Fifty-four to the prefix Fifty four. The unrestricted prefix check then marks the valid transcript as Hallucination:WhisperHallucination, allowing it to be excluded from the curated dataset.

[greptile-apps]

Space-separated phrase variant bypasses detection

When a transcript contains the exact text Fifty four, _transcript_candidates() creates no hyphenated candidate, so it cannot match the bundled Fifty-four phrase and the known hallucination remains unflagged.

[mohammadaaftabv]

/ok to test 78a295e