[danbev]

This commit adds support for the NVIDIA Nemotron Nano 3 model, enabling the conversion and running of this model.

Tech blog: https://developer.nvidia.com/blog/inside-nvidia-nemotron-3-techniques-tools-and-data-that-make-it-efficient-and-accurate/

[danielhanchen]

GGUFs work great! I converted them via the PR at https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF

Note <think> and </think> are separate tokens, so folks might need to use --special if needed.

[arch-btw]

@danielhanchen please wait until it's merged next time.

[danielhanchen]

@danielhanchen please wait until it's merged next time.

We were launch partners with Nvidia and so we supported finetuning out of the gate and announced the GGUFs with it.

The GGUFs also already work in LMStudio as this llama.cpp PR was merged.

I will need to change the instructions for llama.cpp in our guide.

[cmp-nct]

It's a very good model, happy to see it supported so quickly on release.
Sadly it's very bad in detailed replicating, so visual puzzles are not going to work with it but that's a model architectural flaw I guess - same on openrouter

[ggerganov]

I think we have some issue with parsing the reasoning tokens:

Hoping that we'll get support from the community to fix this. Pinging @aldehir

Let's merge after the CI is green.

[engrtipusultan]

I think we have some issue with parsing the reasoning tokens:

Does it have something to do with this, feom unsloth:

Nemotron 3 chat template format:

Nemotron 3 uses <think> with token id 12 and </think> with id 13 for reasoning. Use --special to see the tokens.

https://docs.unsloth.ai/models/nemotron-3

[dinerburger]

Even with --special it doesn't appear to emit the thinking tags. (OpenWebUI does not render them for example). You do get <|im_end however.

[aldehir]

Hoping that we'll get support from the community to fix this. Pinging @aldehir

I will take a look.

[pwilkin]

LMStudio seems to support it and support the thinking, so there has to be some way to make it work, but yeah, during normal generation on server WebUI I didn't see the closing thinking tag (the opening one I assume is appended to the generation prompt, so it's a typical thinking_forced_open = true case).

[CISC]

Nits only, not required to apply any.

[danielhanchen]

@pwilkin Actually I re-checked you're correct - --reasoning-format deepseek-legacy or --reasoning-format deepseek doesn't enable <think></think> parsing. Also side note --verbose-prompt I think is broken via llama-completion / llama-cli - it doesn't print out the previous prompt and token ids anymore :(

But yes <think> is by default prepended

[ggerganov]

I'm thinking that before we fix the reasoning parsing, there is no point to merge this PR. So let's put it on hold until we figure it out.

[aldehir]

@ggerganov I have the changes ready, although they support reasoning + tool calling so they're not small. How would you like me to proceed?

I can provide a subset of the changes to only address the reasoning and then add the rest in an another PR.

[ggerganov]

@aldehir Ah great. Let's merge then and please open a PR after this with your changes. Thanks.

@danbev Merge at will

[danbev]

The models are now available on Huggingface.