Skip to content

Add more custom metrics to minhash stage - #2274

Merged
ayushdg merged 1 commit into
NVIDIA-NeMo:mainfrom
ayushdg:minhash-metrics
Jul 28, 2026
Merged

ayushdg merged 1 commit into
NVIDIA-NeMo:mainfrom
ayushdg:minhash-metrics

Conversation

@ayushdg

@ayushdg ayushdg commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds more metrics to track the read, minhash and write time that can provide more complete stats in benchmarks

Closes NMCUR-187

Usage

# Add snippet demonstrating usage

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>
@ayushdg
ayushdg requested a review from praateekmahajan as a code owner July 28, 2026 15:54
@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds granular custom timing metrics to the MinHash stage.

  • Records input read, MinHash computation, and output write durations independently.
  • Extends the MinHash stage test to verify all three timing metrics are recorded.

Confidence Score: 5/5

The PR appears safe to merge with no actionable defects identified.

The new timing contexts use the existing stage metric lifecycle, and the backend consumes and aggregates these custom metrics after each task without altering MinHash inputs, outputs, or computation.

Important Files Changed

Filename Overview
nemo_curator/stages/deduplication/fuzzy/minhash.py Wraps the existing read, MinHash computation, and parquet write operations in established custom-metric timing contexts without changing processing behavior.
tests/stages/deduplication/fuzzy/test_minhash_stage.py Verifies that each newly introduced timing metric is populated with a positive duration.

Reviews (1): Last reviewed commit: "Add more custom metrics to minhash stage" | Re-trigger Greptile

@sarahyurick sarahyurick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ayushdg
ayushdg merged commit d00ebfc into NVIDIA-NeMo:main Jul 28, 2026
61 checks passed
@ayushdg
ayushdg deleted the minhash-metrics branch July 28, 2026 16:27
tromanski-nv pushed a commit to tromanski-nv/Curator that referenced this pull request Sep 14, 2026
Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>

This branch was previously deployed

3 inactive deployments
public — 0aff3ce4 Deployed Jul 28, 2026 by copy-pr-bot[bot] via release / finalize / notify #699
nemo-ci — 0aff3ce4 Deployed Jul 28, 2026 by copy-pr-bot[bot] via L0_Unit_Test_GPU-sdg #5106
test — 0aff3ce4 Deployed Jul 28, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #5106
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants