You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Quantization import always writes backend: llama-cpp, ignoring the backend that produced the model #11875
POST /api/quantization/jobs/:id/import always writes backend: llama-cpp into the generated model config, regardless of the quantization type the same service just produced. For any type stock llama.cpp cannot read, the imported model is not loadable.
Reproduction
Quantize through /app/quantize with a backend whose format stock llama.cpp does not know, then import the result. The generated YAML:
Loading it fails, because the tensor types are unknown to that backend. Correcting backend: by hand makes the same file work immediately.
Concretely this happened with the rocmfp4 backend (ROCmFP4 weight types, #11636), but the shape of the problem is general: the service knows which backend served StartQuantization, and discards that knowledge at import.
Suggestion
Carry the job's Backend into the imported config instead of defaulting. The value is already on the job record (schema.QuantizationJob.Backend), so the import has it in hand.
A second, smaller point in the same place: the description is hardcoded to Fine-tuned model (GGUF) for a model that was quantized, not fine-tuned.
Impact
Small in code, awkward in practice: the one path that produces a model and the one path that registers it disagree about how to run it, and the user finds out only when the model refuses to load.
Disclosure: this report was prepared with AI assistance (Claude); the import was run on a real host (LocalAI v4.9.0-305-g8351db5de, ROCm 7.x / gfx1151) and the YAML above is the file it wrote.
Activity
fix(quantization): pin the producing backend on imported quantized mo…