Submission: skamalj/signpost-d-9b (9.7B, general, zero-shot), 0.7345 accuracy

#76
by skamalj - opened

Hi, submitting skamalj/signpost-d-9b for the zero-shot table. It's
the 9B version of signpost-d-4b (#61): same data, same recipe, on Qwen3.5-9B.

Test split (2,000 decisions), benchmark scorer:

Accuracy KL Brier ECE
0.7345 [0.713, 0.756] 0.159 0.082 0.101

By type: noul 0.818, choice 0.735, score 0.670.

  • Predictions: all 2,000, in typed_decisions_test_predictions.json (case index, question id, type, probabilities).
  • Serving: stock vLLM nightly (0.34.0rc1.dev535), /v1/systemone, the benchmark's flags; one request per case, no temperature, no fitting.
  • Latency: median 214 ms per case on one L40S.
  • Model: Qwen3.5-9B with three LoRA runs averaged and merged (open weights, Apache-2.0); no extra head.
  • Training: only our own generated data (2,575 records in 163 invented domains). No Typed Decisions row, label or question; none of the four workflows or their question schemas. Never trained, fitted or calibrated on train; a sample of train was analysed to understand the decisions and gold distributions, and that analysis was used to prompt the data generators.
  • Selection: candidates were chosen on the train split (this model: 0.7315). Four test runs in this line; details in the model card.

Thanks for maintaining the benchmark!

LocalLLaMA org

Thanks, the predictions file covers all 2,000 decisions and rescores to the same overall numbers with the dataset's scorer: 0.7345 accuracy, KL 0.159, Brier 0.082, ECE 0.101. By type it comes out as noul 0.817, choice 0.735, score 0.6725, a decision or two off your per-type figures, likely from how the 16 tied top predictions break.

Same placement as signpost-d-4b: signpost-d-9b is in the zero-shot table at #16, just below d3-turbo, with a note in the Kind cell that it was trained only on generated data and candidates were chosen on train. Card PR: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/77

Thanks for the rescore and for adding it to the table so quickly. The per-type difference makes sense: we broke the 16 ties by order, not the way the scorer does.

Sign up or log in to comment