Skip to content

Migrate MS MARCO examples to hf datasets - #3783

Merged
tomaarsen merged 5 commits into
huggingface:mainfrom
omkar-334:migrate-msmarco-to-hf-datasets
Jun 2, 2026
Merged

tomaarsen merged 5 commits into
huggingface:mainfrom
omkar-334:migrate-msmarco-to-hf-datasets

Conversation

@omkar-334

@omkar-334 omkar-334 commented May 24, 2026 •

Copy link
Copy Markdown
Contributor

MS MARCO examples downloaded raw files from msmarco.z22.web.core.windows.net (a 1GB tarball
plus qrels/queries) and parsed them by hand. This replaces those raw downloads with the maintained
Hf datasets

Files

  • training/ms_marco/eval_msmarco.py
  • training/ms_marco/multilingual/translate_queries.py
  • training/ms_marco/eval_cross_encoder_trec_dl.py

Data verification

  • corpus: byte-identical (500/500 sampled passages match the azure collection.tsv; row count
    matches exactly at 8,841,823).
  • query coverage: dev 6980/6980 and train 502939/502939 query ids are 100% present in the HF
    queries; qrels are equivalent

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates the MS MARCO example scripts away from downloading/parsing raw MS MARCO tarballs/qrels and instead uses maintained Hugging Face datasets for qrels/queries (and a HF-hosted corpus for the SBERT dev evaluator).

Changes:

  • Replace MS MARCO raw file downloads/parsing with datasets.load_dataset() in the MS MARCO dev evaluation and multilingual query translation scripts.
  • Switch TREC-DL 2019 qrels loading to the mteb/msmarco dataset while keeping the candidate top1000 download flow.
  • Simplify/modernize the scripts by removing tar/http download logic where no longer needed.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
examples/sentence_transformer/training/ms_marco/eval_msmarco.py Uses HF datasets for dev qrels + queries and a HF-hosted MS MARCO corpus to run IR evaluation.
examples/sentence_transformer/training/ms_marco/multilingual/translate_queries.py Uses HF datasets for train qrels + queries to build the set of queries to translate.
examples/cross_encoder/training/ms_marco/eval_cross_encoder_trec_dl.py Loads TREC-DL 2019 qrels from HF datasets; derives queries from the top1000 candidate file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

@tomaarsen
tomaarsen merged commit a8f2bfb into huggingface:main Jun 2, 2026
17 checks passed
@tomaarsen

Copy link
Copy Markdown
Member

Hello!

Thank you, I did a slight cleanup, but this was already looking strong.

  • Tom Aarsen

@omkar-334
omkar-334 deleted the migrate-msmarco-to-hf-datasets branch August 3, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants