Why convert GGUF to ONNX?
People want GGUF as .onnx to run a model in ONNX Runtime, DirectML or other cross-platform accelerators that don't read GGUF. Because ONNX needs a standard computation graph and GGUF is a llama.cpp-specific quantized container, you first reconstruct a normal PyTorch model and then export it.
How to convert GGUF to ONNX
Dequantize GGUF to safetensors OPEN-SOURCE
Use a converter such as the ungguf tool or a gguf-reader script to turn the .gguf back into full-precision safetensors that Transformers can load.
Export to ONNX with Optimum FREE
Point Optimum at the reconstructed model: optimum-cli export onnx --model ./model-dir onnx-out/ writes the .onnx file.
Export manually with PyTorch OPEN-SOURCE
For unsupported architectures, load the model in PyTorch and call torch.onnx.export(model, dummy_input, "model.onnx") yourself.
About these formats
A .gguf file is a GGML Universal Format File, the binary container used by llama.cpp and related tools to store a large language model as a single file. It holds the model weights,…
Open .GGUF details →A .ONNX file is a data file used by specific software.
Open .ONNX details →Quality & what to watch
- Dequantizing a quantized GGUF cannot recover the precision lost during quantization, and complex quant types (like IQ variants) may not round-trip cleanly.
- The resulting ONNX is full-precision, so it can be several times larger than the compact GGUF you started from.
- Optimum only supports certain architectures out of the box; niche or custom models may need a manual export config or fail.
