The dataset viewer is not available for this split.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
engine: string
engine_options: struct<model: string>
child 0, model: string
model_source: struct<kind: string, temperature: double, quantization: string, loaded_seconds: int64, policy: strin (... 2 chars omitted)
child 0, kind: string
child 1, temperature: double
child 2, quantization: string
child 3, loaded_seconds: int64
child 4, policy: string
device: string
vllm: string
quantization: string
loaded_seconds: double
frozen_corpus_sha256: null
rows_path: string
runner_sha256: string
latency: string
note: string
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6296 chars omitted)
child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6284 chars omitted)
child 0, catalog_id: int64
child 1, dataset: string
child 2, requests: int64
child 3, answered: int64
child 4, unsupported: int64
child 5, errors: int64
child 6, abstained: int64
child 7, pending: int64
child 8, scored_requests: int64
child 9, metric: string
child 10, score: double
child 11, reference_same_cases: null
child 12, median_ms: double
child 13, detail: struct<field_accuracy: double, case_exact_accuracy: double, custom_metrics: struct<ndcg_at_10: doubl (... 5290 chars omitted)
child 0, field_accuracy: double
child 1, case_exact_accuracy: double
child 2, custom
...
ull, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
edition: string
counts: struct<ok: int64, unsupported: int64>
child 0, ok: int64
child 1, unsupported: int64
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 149, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 129, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 544, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 3052, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2559, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2584, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 558, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 440, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 361, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2381, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2309, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
engine: string
engine_options: struct<model: string>
child 0, model: string
model_source: struct<kind: string, temperature: double, quantization: string, loaded_seconds: int64, policy: strin (... 2 chars omitted)
child 0, kind: string
child 1, temperature: double
child 2, quantization: string
child 3, loaded_seconds: int64
child 4, policy: string
device: string
vllm: string
quantization: string
loaded_seconds: double
frozen_corpus_sha256: null
rows_path: string
runner_sha256: string
latency: string
note: string
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6296 chars omitted)
child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6284 chars omitted)
child 0, catalog_id: int64
child 1, dataset: string
child 2, requests: int64
child 3, answered: int64
child 4, unsupported: int64
child 5, errors: int64
child 6, abstained: int64
child 7, pending: int64
child 8, scored_requests: int64
child 9, metric: string
child 10, score: double
child 11, reference_same_cases: null
child 12, median_ms: double
child 13, detail: struct<field_accuracy: double, case_exact_accuracy: double, custom_metrics: struct<ndcg_at_10: doubl (... 5290 chars omitted)
child 0, field_accuracy: double
child 1, case_exact_accuracy: double
child 2, custom
...
ull, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
edition: string
counts: struct<ok: int64, unsupported: int64>
child 0, ok: int64
child 1, unsupported: int64
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Cerno Decision Index 0.3 public runs
Complete, untouched public-suite runs of the Cerno models on Decision Index
edition 0.3 (kit commit 9eb2dbe), produced with the kit's own runner (decision_index run, 8 round-robin shards, one process per H200
GPU, one request at a time; vLLM for v1.0, v1.1 and the 27B v1.1, transformers for v1.2) through the public inference code, and scored with decision_index score --edition 0.3.
Each runs/<name>/ holds results.jsonl.gz, scores.json, index.json, benchmark-summary.json, environment.json,
status.json.
| run | model | public index |
|---|---|---|
runs/cerno-27b-v1.0 |
apertixai/cerno-27b (FP8) through evals.decision_index.engine_vllm:VllmEngine, temperature 1.0 |
65.19, complete (140,178 scoreable: 140,176 ok, 2 declared unsupported) |
runs/cerno-27b-v1.1 |
apertixai/cerno-27b-v1.1 (FP8, read twice) through evals.decision_index.engine_vllm:VllmEngine, temperature 1.059 |
69.73, complete (140,176 ok, 2 declared unsupported) |
runs/cerno-9b-v1.0 |
apertixai/cerno-9b (FP8), v1.0, superseded by v1.1, temperature 1.675 |
58.05, complete (140,176 ok, 2 declared unsupported) |
runs/cerno-4b-v1.0 |
apertixai/cerno-4b (FP8), v1.0, superseded by v1.1, temperature 1.715 |
56.58, complete (140,176 ok, 2 declared unsupported) |
runs/cerno-9b-v1.1 |
apertixai/cerno-9b (FP8, v1.1), temperature 1.539 |
58.59, complete (140,176 ok, 2 declared unsupported) |
runs/cerno-4b-v1.1 |
apertixai/cerno-4b (FP8, v1.1), temperature 1.534 |
57.60, complete (140,176 ok, 2 declared unsupported) |
runs/cerno-9b-v1.2 |
apertixai/cerno-9b-v1.2 (bf16, block attention, read twice) through evals.decision_index.engine:CernoEngine, temperature 1.256 |
66.83, complete (140,178 ok) |
runs/cerno-4b-v1.2 |
apertixai/cerno-4b-v1.2 (bf16, block attention, read twice) through evals.decision_index.engine:CernoEngine, temperature 1.236 |
65.05, complete (140,178 ok) |
Provenance: each package's status.json names the inference-repository commit it was produced with. The v1.0 packages name the
repository's first single commit (1d33cb3844ebe4eba2fc36acb197238c13b5e103); for v1.1 the repository was republished as one commit
(75f98a19b28eff71855d92e955c47a745c153ff8) whose vLLM engine and decider subset are byte-identical to that first commit (the
llama.cpp path, README, examples and reference samples changed). The v1.2 packages name the commit that added the v1.2
transformers engine (153f35e5273f3a2382dd53b1af65a5608019dde5); the v1.0 and v1.1 engines in it are unchanged. The 27B v1.1
package names the commit that added read twice to the vLLM engine (3ae33a49774aa5ad29bc8977c2677856ce702696); checkpoints that
declare no read_twice_tokens (v1.0, v1.1) read once there as before.
Not affiliated with TypeSafe AI or with the Decision Index maintainers.
- Downloads last month
- 325