Dataset Viewer
Duplicate
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code:   StreamingRowsError
Exception:    CastError
Message:      Couldn't cast
engine: string
engine_options: struct<model: string>
  child 0, model: string
model_source: struct<kind: string, temperature: double, quantization: string, loaded_seconds: int64, policy: strin (... 2 chars omitted)
  child 0, kind: string
  child 1, temperature: double
  child 2, quantization: string
  child 3, loaded_seconds: int64
  child 4, policy: string
device: string
vllm: string
quantization: string
loaded_seconds: double
frozen_corpus_sha256: null
rows_path: string
runner_sha256: string
latency: string
note: string
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6296 chars omitted)
  child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6284 chars omitted)
      child 0, catalog_id: int64
      child 1, dataset: string
      child 2, requests: int64
      child 3, answered: int64
      child 4, unsupported: int64
      child 5, errors: int64
      child 6, abstained: int64
      child 7, pending: int64
      child 8, scored_requests: int64
      child 9, metric: string
      child 10, score: double
      child 11, reference_same_cases: null
      child 12, median_ms: double
      child 13, detail: struct<field_accuracy: double, case_exact_accuracy: double, custom_metrics: struct<ndcg_at_10: doubl (... 5290 chars omitted)
          child 0, field_accuracy: double
          child 1, case_exact_accuracy: double
          child 2, custom
...
ull, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
              child 0, metric: string
              child 1, score: double
              child 2, scored_requests: int64
          child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
              child 0, metric: string
              child 1, score: double
              child 2, scored_requests: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
  child 0, median: double
  child 1, p95: double
  child 2, mean: double
edition: string
counts: struct<ok: int64, unsupported: int64>
  child 0, ok: int64
  child 1, unsupported: int64
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/utils.py", line 149, in get_rows_or_raise
                  return get_rows(
                      dataset=dataset,
                  ...<4 lines>...
                      column_names=column_names,
                  )
                File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
                  return func(*args, **kwargs)
                File "/src/services/worker/src/worker/utils.py", line 129, in get_rows
                  rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
                File "/src/services/worker/src/worker/utils.py", line 544, in safe_iter
                  yield from ds.decode(False) if ds.features else ds
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 3052, in __iter__
                  for key, example in ex_iterable:
                                      ^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2559, in __iter__
                  for key, pa_table in self._iter_arrow():
                                       ~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2584, in _iter_arrow
                  for key, pa_table in self.ex_iterable._iter_arrow():
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 558, in _iter_arrow
                  for key, pa_table in iterator:
                                       ^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 440, in _iter_arrow
                  for key, pa_table in self.generate_tables_fn(**gen_kwags):
                                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 361, in _generate_tables
                  self._cast_table(pa_table, json_field_paths=json_field_paths),
                  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
                  pa_table = table_cast(pa_table, self.info.features.arrow_schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2381, in table_cast
                  return cast_table_to_schema(table, schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2309, in cast_table_to_schema
                  raise CastError(
                  ...<3 lines>...
                  )
              datasets.table.CastError: Couldn't cast
              engine: string
              engine_options: struct<model: string>
                child 0, model: string
              model_source: struct<kind: string, temperature: double, quantization: string, loaded_seconds: int64, policy: strin (... 2 chars omitted)
                child 0, kind: string
                child 1, temperature: double
                child 2, quantization: string
                child 3, loaded_seconds: int64
                child 4, policy: string
              device: string
              vllm: string
              quantization: string
              loaded_seconds: double
              frozen_corpus_sha256: null
              rows_path: string
              runner_sha256: string
              latency: string
              note: string
              benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6296 chars omitted)
                child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6284 chars omitted)
                    child 0, catalog_id: int64
                    child 1, dataset: string
                    child 2, requests: int64
                    child 3, answered: int64
                    child 4, unsupported: int64
                    child 5, errors: int64
                    child 6, abstained: int64
                    child 7, pending: int64
                    child 8, scored_requests: int64
                    child 9, metric: string
                    child 10, score: double
                    child 11, reference_same_cases: null
                    child 12, median_ms: double
                    child 13, detail: struct<field_accuracy: double, case_exact_accuracy: double, custom_metrics: struct<ndcg_at_10: doubl (... 5290 chars omitted)
                        child 0, field_accuracy: double
                        child 1, case_exact_accuracy: double
                        child 2, custom
              ...
              ull, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: double
                            child 2, scored_requests: int64
                        child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: double
                            child 2, scored_requests: int64
              successful_request_latency_ms: struct<median: double, p95: double, mean: double>
                child 0, median: double
                child 1, p95: double
                child 2, mean: double
              edition: string
              counts: struct<ok: int64, unsupported: int64>
                child 0, ok: int64
                child 1, unsupported: int64
              to
              {'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
              ...
              Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
              because column names don't match

Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.

Cerno decision models

Cerno Decision Index 0.3 public runs

Complete, untouched public-suite runs of the Cerno models on Decision Index edition 0.3 (kit commit 9eb2dbe), produced with the kit's own runner (decision_index run, 8 round-robin shards, one process per H200 GPU, one request at a time; vLLM for v1.0, v1.1 and the 27B v1.1, transformers for v1.2) through the public inference code, and scored with decision_index score --edition 0.3. Each runs/<name>/ holds results.jsonl.gz, scores.json, index.json, benchmark-summary.json, environment.json, status.json.

run model public index
runs/cerno-27b-v1.0 apertixai/cerno-27b (FP8) through evals.decision_index.engine_vllm:VllmEngine, temperature 1.0 65.19, complete (140,178 scoreable: 140,176 ok, 2 declared unsupported)
runs/cerno-27b-v1.1 apertixai/cerno-27b-v1.1 (FP8, read twice) through evals.decision_index.engine_vllm:VllmEngine, temperature 1.059 69.73, complete (140,176 ok, 2 declared unsupported)
runs/cerno-9b-v1.0 apertixai/cerno-9b (FP8), v1.0, superseded by v1.1, temperature 1.675 58.05, complete (140,176 ok, 2 declared unsupported)
runs/cerno-4b-v1.0 apertixai/cerno-4b (FP8), v1.0, superseded by v1.1, temperature 1.715 56.58, complete (140,176 ok, 2 declared unsupported)
runs/cerno-9b-v1.1 apertixai/cerno-9b (FP8, v1.1), temperature 1.539 58.59, complete (140,176 ok, 2 declared unsupported)
runs/cerno-4b-v1.1 apertixai/cerno-4b (FP8, v1.1), temperature 1.534 57.60, complete (140,176 ok, 2 declared unsupported)
runs/cerno-9b-v1.2 apertixai/cerno-9b-v1.2 (bf16, block attention, read twice) through evals.decision_index.engine:CernoEngine, temperature 1.256 66.83, complete (140,178 ok)
runs/cerno-4b-v1.2 apertixai/cerno-4b-v1.2 (bf16, block attention, read twice) through evals.decision_index.engine:CernoEngine, temperature 1.236 65.05, complete (140,178 ok)

Provenance: each package's status.json names the inference-repository commit it was produced with. The v1.0 packages name the repository's first single commit (1d33cb3844ebe4eba2fc36acb197238c13b5e103); for v1.1 the repository was republished as one commit (75f98a19b28eff71855d92e955c47a745c153ff8) whose vLLM engine and decider subset are byte-identical to that first commit (the llama.cpp path, README, examples and reference samples changed). The v1.2 packages name the commit that added the v1.2 transformers engine (153f35e5273f3a2382dd53b1af65a5608019dde5); the v1.0 and v1.1 engines in it are unchanged. The 27B v1.1 package names the commit that added read twice to the vLLM engine (3ae33a49774aa5ad29bc8977c2677856ce702696); checkpoints that declare no read_twice_tokens (v1.0, v1.1) read once there as before.

Not affiliated with TypeSafe AI or with the Decision Index maintainers.

Downloads last month
325