Skip to content

docs: correct fuzzy deduplication strictness guidance - #2287

Merged
ayushdg merged 2 commits into
NVIDIA-NeMo:mainfrom
nightcityblade:fix/issue-2285
Aug 13, 2026
Merged

ayushdg merged 2 commits into
NVIDIA-NeMo:mainfrom
nightcityblade:fix/issue-2285

Conversation

@nightcityblade

Copy link
Copy Markdown
Contributor

Description

Closes #2285.

Correct the inverted MinHash-LSH tuning guidance for num_bands and minhashes_per_band. The stricter and looser examples now follow the documented threshold approximation, and the math tutorial uses the same directions.

Validation:

  • npm run check -- --warnings (0 errors; the unauthenticated redirects check is the only warning)
  • pre-commit run --files fern/versions/main/pages/curate-text/process-data/deduplication/fuzzy.mdx tutorials/math/README.md
  • Threshold assertion: strict 0.835 > default 0.794 > loose 0.725

Usage

N/A — documentation-only correction.

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

Signed-off-by: nightcityblade <nightcityblade@gmail.com>
@nightcityblade
nightcityblade requested a review from a team as a code owner August 2, 2026 15:44
@nightcityblade
nightcityblade requested review from abhinavg4 and removed request for a team August 2, 2026 15:44
@copy-pr-bot

copy-pr-bot Bot commented Aug 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@greptile-apps

greptile-apps Bot commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

Corrects the fuzzy-deduplication tuning guidance so stricter and looser MinHash-LSH configurations align with the documented threshold approximation.

  • Reverses the num_bands and minhashes_per_band strictness directions in the fuzzy-deduplication guide.
  • Updates configuration examples and the math tutorial to use consistent guidance.

Confidence Score: 5/5

The documentation-only correction appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
fern/versions/main/pages/curate-text/process-data/deduplication/fuzzy.mdx Corrects the strictness guidance, adds the threshold approximation, and updates examples consistently.
tutorials/math/README.md Aligns the tutorial's LSH tuning directions with the corrected threshold behavior.

Reviews (2): Last reviewed commit: "Merge branch 'main' into fix/issue-2285" | Re-trigger Greptile

@sarahyurick
sarahyurick requested a review from ayushdg August 3, 2026 16:41
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Aug 4, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@ayushdg

ayushdg commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

/ok to test 59e6b38

@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Aug 11, 2026
@nightcityblade

Copy link
Copy Markdown
Contributor Author

Thank you for the review and approval. All required checks are now green, and the PR is ready to merge when convenient.

@ayushdg
ayushdg merged commit 35a255e into NVIDIA-NeMo:main Aug 13, 2026
99 of 101 checks passed
tromanski-nv pushed a commit to tromanski-nv/Curator that referenced this pull request Sep 14, 2026
Signed-off-by: nightcityblade <nightcityblade@gmail.com>
Co-authored-by: nightcityblade <nightcityblade@gmail.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>

This branch was previously deployed

3 inactive deployments
nemo-ci — 59e6b38f Deployed Aug 11, 2026 by copy-pr-bot[bot] via L0_Unit_Test_GPU-text #5192
public — 59e6b38f Deployed Aug 11, 2026 by copy-pr-bot[bot] via release / finalize / notify #771
test — 59e6b38f Deployed Aug 11, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #5192
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: fuzzy-dedup tuning guidance for num_bands / minhashes_per_band appears inverted

3 participants