Datasets:
Submission: skamalj/signpost-d-9b (9.7B, general, zero-shot), 0.7345 accuracy
Hi, submitting skamalj/signpost-d-9b for the zero-shot table. It's
the 9B version of signpost-d-4b (#61): same data, same recipe, on Qwen3.5-9B.
Test split (2,000 decisions), benchmark scorer:
| Accuracy | KL | Brier | ECE |
|---|---|---|---|
| 0.7345 [0.713, 0.756] | 0.159 | 0.082 | 0.101 |
By type: noul 0.818, choice 0.735, score 0.670.
- Predictions: all 2,000, in
typed_decisions_test_predictions.json(case index, question id, type, probabilities). - Serving: stock vLLM nightly (
0.34.0rc1.dev535),/v1/systemone, the benchmark's flags; one request per case, no temperature, no fitting. - Latency: median 214 ms per case on one L40S.
- Model: Qwen3.5-9B with three LoRA runs averaged and merged (open weights, Apache-2.0); no extra head.
- Training: only our own generated data (2,575 records in 163 invented domains). No Typed Decisions row, label or question; none of the four workflows or their question schemas. Never trained, fitted or calibrated on
train; a sample oftrainwas analysed to understand the decisions and gold distributions, and that analysis was used to prompt the data generators. - Selection: candidates were chosen on the
trainsplit (this model: 0.7315). Fourtestruns in this line; details in the model card.
Thanks for maintaining the benchmark!
Thanks, the predictions file covers all 2,000 decisions and rescores to the same overall numbers with the dataset's scorer: 0.7345 accuracy, KL 0.159, Brier 0.082, ECE 0.101. By type it comes out as noul 0.817, choice 0.735, score 0.6725, a decision or two off your per-type figures, likely from how the 16 tied top predictions break.
Same placement as signpost-d-4b: signpost-d-9b is in the zero-shot table at #16, just below d3-turbo, with a note in the Kind cell that it was trained only on generated data and candidates were chosen on train. Card PR: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/77
Thanks for the rescore and for adding it to the table so quickly. The per-type difference makes sense: we broke the 16 ties by order, not the way the scorer does.