Instructions to use h2oai/h2o-lightning-4b-nemotron-nano with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use h2oai/h2o-lightning-4b-nemotron-nano with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="h2oai/h2o-lightning-4b-nemotron-nano", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("h2oai/h2o-lightning-4b-nemotron-nano", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("h2oai/h2o-lightning-4b-nemotron-nano", trust_remote_code=True, device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use h2oai/h2o-lightning-4b-nemotron-nano with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "h2oai/h2o-lightning-4b-nemotron-nano" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "h2oai/h2o-lightning-4b-nemotron-nano", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/h2oai/h2o-lightning-4b-nemotron-nano
- SGLang
How to use h2oai/h2o-lightning-4b-nemotron-nano with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "h2oai/h2o-lightning-4b-nemotron-nano" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "h2oai/h2o-lightning-4b-nemotron-nano", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "h2oai/h2o-lightning-4b-nemotron-nano" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "h2oai/h2o-lightning-4b-nemotron-nano", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use h2oai/h2o-lightning-4b-nemotron-nano with Docker Model Runner:
docker model run hf.co/h2oai/h2o-lightning-4b-nemotron-nano
H2O-Lightning-4B-Nemotron-Nano
H2O-Lightning-4B-Nemotron-Nano is a 4B-parameter decision model from H2O.ai, built on nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16. It answers typed decision questions about a record (a document, a ticket, a policy, a conversation). It returns a probability for every option:
- choice: picks one of a set of named options;
- yes/no (
noul): gives the probability that a statement is true; - score: gives an ordinal level on a stated scale, with its probability distribution.
It reads text only. It runs on unmodified vLLM 0.30.0 with a small shim in front (lightning_nemo4b_shim.py).
Each decision is one forward pass and one output token, so the cost is input tokens only.
Sibling model: H2O-Lightning-4B is built on Qwen3.5-4B and has its own card. Every measurement on this card is this model's own.
Measured
Measured with serve.sh as shipped (vLLM 0.30.0; one temperature, 0.75, for every
question type), on one NVIDIA L40S.
Text: JevBench's public set
From JevBench's own CLI, with --adapter typesafe, on the 231 public items (datasets/public/easy.jsonl,
original.jsonl and hard.jsonl; dataset hash dc3995d8β¦) at JevBench commit bb05a33, with the commands below.
| 1x L40S | |
|---|---|
| correct | 192 / 231 (83.1 %) |
| by tier: easy / standard / hard | 48/48 Β· 71/72 Β· 73/111 |
| answered and valid | 231 / 231 |
| yes/no answers with P(yes) strictly between 0.2 and 0.8 | 0 of 74 |
| mean input tokens per decision | 699 |
| latency p50 / p95, standard tier, serial | 0.027 s / 0.027 s |
| latency p50 / p95, all 231 items, serial | 0.028 s / 0.124 s |
Latency depends on the GPU.
Memory. vLLM loads the weights into 7.47 GiB of GPU memory. serve.sh passes vLLM's --gpu-memory-utilization 0.90
(GPU_UTIL), and most of the rest of that share becomes the KV cache: on one L40S, 31.64 GiB, 1,611,093 tokens, or
39.33 requests of the full 40,960 tokens at once. Lower GPU_UTIL to share the GPU with other processes.
Decision Index 0.3 public suite
Our own runs of the public Decision Index 0.3 kit (decision-index at commit f8552a1, edition 0.3, its official
scorer) on the public text suite (140,178 scored rows; the model reads no images, so the vision half does not
apply), served with this release's serve.sh:
| Decision Index 0.3 public suite, our runs | DI index | raw index |
|---|---|---|
as shipped (serve.sh: 40,960-token context, 12,000-token state budget) |
53.20 | 65.46 |
no state trimming (MAX_LEN=131072, SHIM_MAX_STATE_TOKENS=1000000) |
53.22 | 65.47 |
The DI index is the kit's chance-corrected index (0-100), not percent correct. These are our runs, not a leaderboard result; the leaderboard's private tests were not run. No row ended in an error in either run.
Routing thresholds
When a decision is uncertain you may want to hand it to a stronger model or a person. This table shows that trade-off: at a confidence threshold, how many questions get escalated (top probability below the threshold) and how many slip through confidently wrong (top probability at or above it, but wrong), as a share of all questions.
Measured on 3,474 held-out questions from two public third-party decision sets, with no overlap with our training data; text temperature as shipped (0.75). The thresholds were fixed before the analysis was run.
| question type (n) | threshold | escalated | accuracy on the rest | confidently wrong |
|---|---|---|---|---|
| choice (1,854) | 0.8 | 43.6 % | 94.3 % | 3.2 % |
| choice | 0.9 | 56.9 % | 96.9 % | 1.3 % |
| yes/no (1,367) | 0.8 | 47.3 % | 93.5 % | 3.4 % |
| yes/no | 0.9 | 64.3 % | 98.2 % | 0.7 % |
| score (253) | 0.8 | 54.9 % | 89.5 % | 4.7 % |
| score | 0.9 | 77.5 % | 96.5 % | 0.8 % |
| all (3,474) | 0.8 | 45.9 % | 93.7 % | 3.4 % |
| all (3,474) | 0.9 | 61.3 % | 97.3 % | 1.0 % |
Without a threshold, accuracy is 75.4 % (choice), 79.4 % (yes/no) and 68.4 % (score).
These numbers use routing mode (the yes/no floor off), for deployments behind a fallback. As shipped (the evaluated configuration), yes/no answers are reported at no less than 0.801, so none would be escalated below 0.8.
Run it
Hardware measured: one L40S; the weights are 7.95 GB. Install vLLM 0.30.0 in a fresh virtual environment
(for example python3 -m venv .venv && . .venv/bin/activate):
pip install "vllm==0.30.0+cu129" --extra-index-url https://wheels.vllm.ai/0.30.0/cu129 --extra-index-url https://download.pytorch.org/whl/cu129
The shim runs in the same environment: it is standard-library Python apart from the tokenizers package, which vLLM
installs.
hf download h2oai/h2o-lightning-4b-nemotron-nano --local-dir h2o-lightning-4b-nemotron-nano
1. Serve the model. Keep config.json as shipped: its "head_dtype": "float32" makes vLLM compute the logits in
fp32. Keep vLLM's default --chat-template-content-format. A long record is never refused for its length: a state over
12,000 tokens keeps its head and its tail around a ... line.
vllm serve ./h2o-lightning-4b-nemotron-nano --served-model-name h2oai/h2o-lightning-4b-nemotron-nano --host 127.0.0.1 --port 8000 \
--dtype bfloat16 --max-model-len 40960 --gpu-memory-utilization 0.90
2. Start the shim on the same machine:
python3 h2o-lightning-4b-nemotron-nano/lightning_nemo4b_shim.py --config h2o-lightning-4b-nemotron-nano/serve_config.json --vllm http://127.0.0.1:8000 --port 8741 --tokenizer h2o-lightning-4b-nemotron-nano/tokenizer.json
Run the shim as shipped for evaluations and benchmarks (including JevBench): serve_config.json is the evaluated
configuration. For production routing, see Optional: routing mode.
bash h2o-lightning-4b-nemotron-nano/serve.sh runs steps 1 and 2 together; the measurements above used it. Before
answering, the shim checks that vLLM serves h2oai/h2o-lightning-4b-nemotron-nano, that vLLM's chat rendering of a
probe question is token for token the prompt the model expects, that the tokenizer it counts with is vLLM's, and that
the letter mask is applied; if not, it answers 503, so a misconfigured server stops a run instead of scoring it.
3. Warm up until this returns HTTP 200:
curl -s http://127.0.0.1:8741/v1/systemone -H 'Content-Type: application/json' -d '{"state": "warm-up",
"questions": {"decision": {"type": "noul", "instructions": "Is this a warm-up?",
"criteria": {"true": "yes", "false": "no"}}}}'
4. Run JevBench's harness:
cd <JEVBENCH> && python3 -m jevbench.cli run \
--tasks datasets/public/easy.jsonl,datasets/public/original.jsonl,datasets/public/hard.jsonl \
--adapter typesafe --endpoint http://127.0.0.1:8741 --key-env '' --model h2oai/h2o-lightning-4b-nemotron-nano \
--cost-basis self_hosted_gpu --reserve-usd 0 --results <OUT>/results.jsonl --raw-dir <OUT>/raw \
--ledger <OUT>/ledger.jsonl --manifest <OUT>/manifest.json --run-label h2o-lightning-4b-nemotron-nano --delay-s 0
The API. Send requests to POST /v1/systemone (/api/alpha/decisions is an alias). One request can carry several
questions about the same state:
{"state": "Ticket 4471: since 09:10 the checkout page returns HTTP 500 for every customer; no orders have gone through in 40 minutes. Severity guide: P1 = a revenue path fully down; P2 = degraded but working; P3 = cosmetic.",
"questions": {
"priority": {"type": "choice", "instructions": "Which priority does the severity guide assign?",
"criteria": {"p1": "Revenue path fully down", "p2": "Degraded but working", "p3": "Cosmetic"}},
"all_users": {"type": "noul", "instructions": "The problem affects every customer.",
"criteria": {"true": "Affects everyone", "false": "Affects only some"}},
"urgency": {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["Low", "Medium", "High"]}}}
The response (rounded here):
{"model": "h2oai/h2o-lightning-4b-nemotron-nano",
"answers": {"priority": {"type": "choice", "choice": "p1",
"probabilities": {"p1": 0.931, "p2": 0.046, "p3": 0.024}, "confidence": 0.896},
"all_users": {"type": "noul", "noul": 0.949},
"urgency": {"type": "score", "score": 1.855, "legend": {"0": "Low", "1": "Medium", "2": "High"},
"probabilities": {"0": 0.02, "1": 0.105, "2": 0.875}, "confidence": 0.875}},
"probability_source": "native", "effort_used": {"tier": "fast", "passes_per_decision": 1},
"usage": {"input_tokens": 268, "processed_input_tokens": 268, "output_tokens": 3, "latency_ms": 58.3}}
usage.input_tokens counts the prompt head that the questions share once; usage.output_tokens is one label
token read per answered question. A record cut or selected to fit the state budget is still counted whole
in usage.input_tokens; usage.processed_input_tokens is what vLLM read.
Text only. The model has no image input. Send records as text; nothing in state is read as an image.
Status codes. An input the shim cannot take gets HTTP 422, which the runner scores as one wrong answer. That covers:
- instructions and options alone over the context limit;
- more than 255 options (26 lettered, then the trained wide labels);
- an unknown question type or a malformed question.
A request with no questions gets 400; a body that is not an object with state and questions gets 422. vLLM's 401, 403 and 429 pass through. Any other vLLM failure is a 502, and a misconfigured
vLLM is a 503; both count toward the runner's three-consecutive-failures stop.
OpenAI's Decisions API. The shim also serves POST /v1/decisions for OpenAI's Python client (pip install "openai>=3.26", then OpenAI(base_url="http://127.0.0.1:8741/v1", api_key="unused").decisions.create(model="h2o-lightning-4b-nemotron-nano", input=..., questions=[...]); h2oai/h2o-lightning-4b-nemotron-nano works too). Each request is asked through the same decision as
/v1/systemone, in the option format the model was trained on (a choice's description is the option's text, its
value when it has none; a score level is its label; a predicate is a yes/no question with the trained texts); the
answers and usage come back in OpenAI's shape. This model reads text only: a request with an input_image is refused (HTTP 422).
Tests (no GPU): python3 h2o-lightning-4b-nemotron-nano/test_shim.py and python3 h2o-lightning-4b-nemotron-nano/test_security.py.
Image references and the Box question
This model reads text only: a request with image assets or a box question gets 400.
Log-probabilities the shim needs
Every answer is a probability distribution over all of the question's options. Up to 20 options are read in one
request from vLLM's top 20 log-probabilities under the option-letter mask (--max-logprobs 20, vLLM's default, which
serve.sh sets explicitly; do not lower it). From this release on, a question with 21 to 26 options is read by the
option letters' own token ids (logprob_token_ids, one request), so every option gets its probability; earlier
releases read those questions from the top 20 too, and a letter outside them read the 1e-12 floor.
"letter_read": "masked" in serve_config.json (or LIGHTNING_LETTER_READ=masked) restores the earlier read. Questions with more than 26 options use the wide labels and are always read by token id. A yes/no question without a "true" option is asked as true / false, as the model was trained. At start-up the shim checks both readouts on a 26-option probe and
answers 503 with the reason when the server cannot give them (it needs vLLM 0.30 with structured outputs).
Optional: routing mode
Not for benchmark or leaderboard runs. Those use the shipped serve_config.json unchanged.
As shipped, the shim reports each yes/no answer at no less than 0.801 on its chosen side ("noul_floor": 0.801 in
serve_config.json). That is the evaluated configuration, and it is why no shipped yes/no answer falls between 0.2 and
0.8. If you put the model in front of a fallback, and escalate uncertain answers to a larger model or a person, you
usually want the raw yes/no probability instead, so uncertain yes/no answers can be flagged. In that deployment only,
start the shim with the floor off:
SHIM_NOUL_FLOOR=0 python3 h2o-lightning-4b-nemotron-nano/lightning_nemo4b_shim.py --config h2o-lightning-4b-nemotron-nano/serve_config.json --vllm http://127.0.0.1:8000 --port 8741 --tokenizer h2o-lightning-4b-nemotron-nano/tokenizer.json
The floor never changes which answer is chosen, and choice and score answers are unaffected. See Routing thresholds for what each threshold escalates with the floor off.
Weights
model.safetensors is the fine-tuned model merged into one bf16 file (sha256 94c7be97a7622a1c66da630f289ee67a506d03319af7bd636a5c2b0305ebc8f6), built on
nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 at revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f. No adapter is
shipped. This release replaces the weights of the previous one with a model trained longer, on more data, in three
stages (below); the serving files (serve.sh, the shim, serve_config.json, config.json and the tokenizer) are
unchanged, so a deployment of the previous release serves these weights with the same commands and memory.
What it was trained on
H2O-Lightning-4B-Nemotron-Nano was fine-tuned from NVIDIA-Nemotron-3-Nano-4B in three stages, from easier to harder, with one mixed distribution in every stage; the final stage also mixes in additional multi-question decision data. Every training example is a record (a document, ticket, policy, conversation, game state...) with one or more typed questions about it and graded answers: 1,121,833 examples in all. Documents, policies and rules make up the largest share.
| Use case | Share of examples |
|---|---|
| Documents, policies & rules (contracts, policies, multi-step reading, judging answers) | 27% |
| Everyday language & knowledge | 15% |
| Numbers, dates & probability | 14% |
| Support & operations (tickets, routing, workflows) | 13% |
| Coding & tool calls | 12% |
| Games, simulations & music (state tracking, planning, control, music) | 10% |
| Safety & guardrails | 7% |
| Finance & commerce | 1% |
Use cases are assigned per data family, so the split is approximate.
Question types. About two thirds of the examples are choice questions, a little under a quarter are yes/no and the rest are score questions.
| Question type | Share of examples |
|---|---|
| Choice | 67% |
| Yes / no | 23% |
| Score | 10% |
Text only. All fine-tuning examples were text. About 98% were in English; the rest are across 22 other languages.
Mamba-2 output projections. The fine-tune adds LoRA adapters to the Mamba-2, attention and MLP projections. In the first two stages and the earlier part of the third, the fused Mamba-2 training kernel applied the mixer's output projection (out_proj) inside itself, so the adapters on it received no gradient and stayed at zero. The last part of the third stage applied them outside the kernel, so they were trained there only. In the released weights every out_proj differs from the base, by at most 0.0056 per weight, a smaller change than in the input projections (in_proj, at most 0.077). All the measurements on this card are of these released weights.
Limitations
- Evaluated mostly in English. At most 255 options (A-Z, then the trained wide labels); a request with more gets 422.
- Text only: no image or video input.
- The probabilities are calibrated for decision questions of this kind, not for open-ended text.
- Not a chat model: it answers one decision per question and does not generate explanations.
Disclosures
- No JevBench or ImageJevBench data, public or otherwise, was used for training.
- No rules keyed to any benchmark item's wording, id or answer; no network calls other than the local vLLM server.
- Price basis for an evaluator: NVIDIA-Nemotron-3-Nano-4B at bf16, one output token per decision; 699 mean input tokens on the public text set.
License
The weights are a derivative of NVIDIA-Nemotron-3-Nano-4B and are distributed under the
NVIDIA Nemotron Open Model License (LICENSE; v. December 15, 2025,
nvidia.com).
Licensed by NVIDIA Corporation under the NVIDIA Nemotron Model License. Built on
nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 (revision
dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f) by NVIDIA.
The shim (lightning_nemo4b_shim.py, with state_cut.py and selector_v2.py), test_shim.py, serve.sh,
serve_config.json and this documentation are Apache-2.0 (LICENSE-Apache-2.0). The prompt layout and the one-token
option-letter readout the shim implements come from
Cygnet (commit 74f9555, MIT; LICENSE-cygnet). Served with
unmodified vLLM (Apache-2.0). See NOTICE.
- Downloads last month
- 1,191
Model tree for h2oai/h2o-lightning-4b-nemotron-nano
Base model
nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base

