Evaluate open-source LLMs on Swiss legal benchmarks (SLDS, SwiLTra-Bench, LEXam) using lighteval with paper-grounded LLM-as-judge metrics.
Default install for API-based evaluation, aggregation, plotting, blog figures, and tests:
git clone https://github.com/JoelNiklaus/SwissLegalEvals.git
cd SwissLegalEvals
uv venv --python 3.13
GIT_LFS_SKIP_SMUDGE=1 uv sync --extra dev # skip lighteval fixture blobs; install pytest, ruff, kaleidoOptional local vLLM support on a normal machine:
uv sync --extra local
uv pip install vllmCluster patch for our Hopper environment only:
uv sync --extra local
mkdir -p scripts/logs/slurm
sbatch scripts/setup_vllm.shscripts/setup_vllm.sh is not the general installation path. It is a cluster-specific setup for our Hopper nodes: it creates a Python 3.12 .venv-apertus because the Apertus dependency stack has no CPython 3.13 wheel, installs the pinned Swiss AI vLLM/Transformers revisions with the matching precompiled cu129 extension, optionally applies glibc-fix when /admin/opt/glibc-2.38 is present (skipped on images that only have system glibc 2.35), and installs a tiny nvJitLink preload hook so the wheel does not accidentally pick up older system CUDA libraries. scripts/launch_eval.sh uses this environment for all local vLLM models; the project .venv remains for API-based runs and development. Other users should first try the normal uv pip install vllm path.
Provider API keys are loaded automatically from .env in the project root. Shell exports still work and take precedence over .env values.
Example .env:
cp .env.example .env
# Then fill in the values:
HF_TOKEN=...
OPENAI_API_KEY=...
OPENROUTER_API_KEY=...| Provider | Variable |
|---|---|
openai |
OPENAI_API_KEY |
openrouter |
OPENROUTER_API_KEY |
hf-inference-providers |
HF_TOKEN |
vllm |
(local GPU) |
Optional: set HF_ORG_TO_BILL=<org-slug> to bill HF Inference Providers usage to a Team/Enterprise organization instead of your personal account. Applies to eval models and HF-backed judges (SLDS, LEXam OQ).
Useful overrides:
| Variable | Purpose |
|---|---|
SWISSLEGALEVALS_JUDGE_CONFIG |
Absolute path to an alternate judges.yaml. |
SWISSLEGALEVALS_TASKS_CONFIG |
Task config path passed to lighteval subprocesses. |
SWISSLEGALEVALS_TASKS_PROFILE |
Active task profile passed to lighteval subprocesses. |
Use --env-file /path/to/.env to load a different dotenv file. Default HF judges use novita because the currently configured DeepSeek judge IDs are not served by together on HF inference providers.
Dry-run all configured models (prints lighteval commands, no API keys required):
uv run swiss-legal-evals --dry-runRun all models (evaluate is an alias for swiss-legal-evals):
uv run evaluateRun one configured model by provider:
# HF inference providers (large MoE models)
uv run swiss-legal-evals --models deepseek-v4-pro
# Local vLLM on the cluster (after setup_vllm.sh)
scripts/launch_model.sh gemma-4-31b-it
scripts/launch_model.sh qwen3.5-35b-a3b
scripts/launch_model.sh apertus-v1.5-70b
scripts/launch_model.sh mistral-medium-3.5-128b
# SwiLTra-only translation specialist on the cluster
scripts/launch_model.sh hy-mt2-30bOpenRouter is supported by the runner, but no OpenRouter models are part of the published 2026 run. Add one to configs/models.yaml before selecting it:
- name: my-openrouter-model
provider: openrouter
model: provider/model-idLocal vllm runs use tensor_parallel_size and data_parallel_size from configs/models.yaml. Current local models use DP1 after Ray data-parallel deadlocks on this cluster: most use TP4 (4 H100s), Apertus 1.5 uses TP4, Mistral Medium 3.5 uses TP8, and lfm2.5-8b uses a single GPU. scripts/launch_all.sh requests data_parallel_size * tensor_parallel_size GPUs per vLLM job. HF Inference Provider models request no GPUs, only CPUs for orchestration and judging.
Full SwiLTra-Bench (all granularity levels):
uv run swiss-legal-evals --profile swiltrabench_full
# or keep default profile and add the group:
uv run swiss-legal-evals --groups slds swiltrabench_full lexamDebug cap:
uv run swiss-legal-evals --models gemma-4-31b-it --max-samples 5Single-task smoke test with an API model:
uv run swiss-legal-evals \
--models deepseek-v4-flash \
--task-string 'slds:de_de|0' \
--max-samples 1 \
--output-dir results_smoke/hfUse an alternate model config for infrastructure smoke tests:
uv run swiss-legal-evals \
--models-config configs/models_smoke.yaml \
--models qwen2.5-0.5b-vllm \
--max-samples 1 \
--output-dir results_smoke/vllmOther useful flags:
| Flag | Purpose |
|---|---|
--models-config |
Use a different model list, such as configs/models_smoke.yaml. |
--tasks-config |
Use a different task file, such as configs/tasks_cap_compare.yaml. |
--skip-env-check |
Skip API-key validation before launching jobs. Useful only for debug. |
--verbose |
Enable more detailed runner logs. |
Run the same checks as CI before opening a pull request or release:
uv run ruff check src tests scripts/analyze_cap_compare.py
uv run pytest tests -qRun the evaluation, aggregate raw JSON, then render Plotly family charts:
uv run swiss-legal-evals
uv run swiss-legal-evals-aggregate
uv run swiss-legal-evals-plotOutput files:
| Path | Produced by | Purpose |
|---|---|---|
results/results/<model>/results_*.json |
swiss-legal-evals / lighteval |
Raw model-level result JSON. |
results/details/**/details_*.parquet |
swiss-legal-evals / lighteval |
Per-sample generations and scores. |
results/summary_long.csv |
swiss-legal-evals-aggregate |
Tidy task/metric table. |
results/summary_family_mean.csv |
swiss-legal-evals-aggregate |
Model/family/metric means, used by plot. |
results/summary.csv |
swiss-legal-evals-plot |
Model x family primary-metric table. |
plots/<family>.html |
swiss-legal-evals-plot |
Interactive Plotly family charts. |
The completed 2026 full-run outputs are mirrored in the public Hugging Face bucket joelniklaus/SwissLegalEvals:
# Download the published results into ./results
hf buckets sync hf://buckets/joelniklaus/SwissLegalEvals results
# Maintainers: mirror local ./results back to the bucket
hf buckets sync results hf://buckets/joelniklaus/SwissLegalEvals --deleteLocal result directories (results/, results_smoke*, results_cap_compare_*) are gitignored to keep the repository lightweight.
blog/swiss-legal-evals-2026.md is a short write-up of the 2026 full run (state of the art of open models on Swiss legal tasks). Regenerate its static PNG charts from the published summaries with:
uv sync --extra dev
hf buckets sync hf://buckets/joelniklaus/SwissLegalEvals/summary.csv results
hf buckets sync hf://buckets/joelniklaus/SwissLegalEvals/summary_long.csv results
uv run python blog/make_figures.py # writes blog/figures/*.pngblog/make_figures.py writes nine static PNGs and uses cached Hugging Face org avatars from blog/logos/. Two of them cover the LEXam IDK calibration, mcq_idk_score.png (penalty-adjusted score) and mcq_abstention.png (how often each model picks the reserved "I don't know" letter), and read the per-task idk_score / idk_freq values from summary_long.csv. The first run downloads missing logos; later runs use the checked-in cache.
configs/tasks.yaml selects which benchmark families to run and sets generation_size (max new tokens) under each task block. Example:
profiles:
default:
slds:
generation_size: 32768
swiltrabench:
scope: scoped
generation_size: 32768
lexam:
generation_size: 32768
open_question: true
mcq: trueAll task families use the same 32k generation cap for simplicity. SwiLTra still applies max(level default, generation_size) so paragraph-level tasks respect the configured floor. LEXam MCQ and open questions share lexam.generation_size.
run.py passes the active profile to lighteval via SWISSLEGALEVALS_TASKS_CONFIG and SWISSLEGALEVALS_TASKS_PROFILE.
Model entries can narrow the global groups with a tasks: field. The translation specialist hy-mt2-30b uses tasks: [swiltrabench], so it never runs SLDS or LEXam.
Temporary cap-comparison profiles live in configs/tasks_cap_compare.yaml (cap16k, cap32k). Count default-profile samples with:
uv run python scripts/count_task_samples.pyThe scripts in scripts/ are mostly for the Hopper Slurm environment used for the published run. They are useful references, but they are not a portable cloud deployment layer.
| Script | Purpose |
|---|---|
scripts/launch_all.sh |
Submit one Slurm job per configured benchmark model. |
scripts/launch_model.sh |
Submit a single model with the configured GPU count. |
scripts/launch_eval.sh |
Slurm worker that loads modules and runs the evaluator. |
scripts/setup_vllm.sh |
Hopper-specific vLLM/CUDA/glibc setup patch. |
scripts/count_task_samples.py |
Print per-task and per-group sample counts. |
scripts/run_cap_compare.sh |
Run 16k vs 32k generation-cap experiments. |
scripts/analyze_cap_compare.py |
Summarize cap-comparison outputs. |
scripts/check_mcq_integrity.py |
Verify MCQ runs scored the prompts they generated from. |
scripts/bootstrap_composite.py |
95% intervals for the composite, by resampling tasks. |
These choices explain the shape of the repo if you are getting oriented:
- Generation is scored by paper-grounded judges. SLDS, SwiLTra-Bench, and LEXam open questions use the judge prompts/models from their papers rather than BLEU, ROUGE, or embedding metrics. MCQ tasks use extractive scoring (
trad_score,idk_score,extract_fail) instead of a judge. - Judge configuration is separate from model configuration.
configs/judges.yamlcontrols the scorer models and providers;configs/models.yamlcontrols the models being evaluated. Both can use API providers, while local evaluation models usevllm. - lighteval is pinned to a git revision.
pyproject.tomluseshuggingface/lighteval@4d470292936b9ec5523cb495b4165cc4f77bcc77because these Swiss legal tasks were not in the PyPI release when this repo was built. - The default SwiLTra-Bench profile is scoped. The normal run uses one granularity per translation dataset (
text_level,paragraph_level,press_release) to keep the full benchmark feasible. Useswiltrabench_fullfor all levels. - LEXam MCQ with IDK stops at 16 choices.
mcq_32with IDK would need labels beyondA-Z, so it is disabled until the prompt/scorer supports wider labels. - Reasoning-model outputs are normalized before scoring.
postprocess.pystrips common chain-of-thought wrappers and final-answer markers before judges or MCQ extraction see the text. - API generations stream. HF inference-provider model calls stream chat completions so long generations keep the router connection alive instead of timing out.
- Transient provider failures are isolated per sample. Model and judge calls retry independently; exhausted calls are logged as missing judgments or generations so one provider outage does not discard an entire batch.
- Existing sample caches are reused across restarts. If lighteval assigns an equivalent task a different in-process hash, the fullest previously loaded cache is selected and only missing samples are regenerated.
- Generation is chunked so the cache survives a crash. lighteval writes its sample cache only after a whole batch finishes, so a run that dies mid-batch saves nothing.
cache_patch.pycalls generation in chunks of 500 docs, making every chunk boundary a cache checkpoint; a crash costs at most one chunk and the relaunch resumes. SetSWISS_LEGAL_EVALS_CACHE_CHUNK_SIZEto trade throughput against the size of that loss window. - LEXam MCQ option order is derived from the question. lighteval shuffles the answer choices from a single module-level RNG, so a question's letters depend on how many questions were shuffled before it in that process. Scoring cached generations in a later process then re-rolled the letters and marked correct answers wrong, which invalidated the MCQ scores of every run that resumed from a cache. The order is now seeded from the choice texts, so it is identical in every process. Use
scripts/check_mcq_integrity.pyto confirm a run scored the prompts it generated from. - API calls have a request timeout. lighteval's inference-providers client defaults to no timeout, so a stream the provider stops feeding blocks forever. Runs default to 1800 s, overridable per model with
timeoutinconfigs/models.yaml. Note that changing it alters the model-config hash, so a model's existing sample cache is not reused.
configs/
judges.yaml judge providers and model IDs
models.yaml published benchmark model list
models_smoke.yaml tiny infra smoke models
tasks.yaml default/full task groups and generation caps
tasks_cap_compare.yaml 16k vs 32k cap profiles
src/swiss_legal_evals/
aggregate.py lighteval JSON -> summary CSVs
cuda_preload.py nvJitLink preload helper for Hopper vLLM setup
plot.py Plotly family charts
postprocess.py reasoning-output cleanup and MCQ extraction helpers
providers.py provider/API-key validation
run.py CLI entry point that builds and launches lighteval
task_lists.py YAML profiles -> lighteval task strings
tasks.py custom Swiss legal TASKS_TABLE and judge metrics
scripts/
launch_all.sh Slurm submitter for the full model set
launch_model.sh Slurm submitter for one model
launch_eval.sh Slurm worker
setup_vllm.sh Hopper vLLM setup patch
count_task_samples.py sample-count utility
run_cap_compare.sh generation-cap comparison runner
analyze_cap_compare.py cap-comparison analysis
blog/
swiss-legal-evals-2026.md
make_figures.py
figures/ static PNGs used by the post
logos/ cached Hugging Face org avatars
tests/ pytest coverage for runner, configs, tasks, aggregation, postprocessing
.github/workflows/ci.yml ruff + pytest CI
If you use SwissLegalEvals, cite the repository:
@misc{niklaus2026swisslegalevals,
author = {Joel Niklaus},
title = {The state-of-the-art in open-source AI for Swiss legal tasks},
year = {2026},
howpublished = {\url{https://github.com/JoelNiklaus/SwissLegalEvals}},
note = {SLDS, SwiLTra-Bench, and LEXam evaluations with lighteval}
}MIT - see LICENSE.