GGUF models: none of the 198 most downloaded on Hugging Face can be rebuilt identically by a third party

The signature protects the original weights, not the quantified version that is actually executed. A study shows that there is currently no way to verify that a quantified file corresponds to the advertised model.

Signing the weights of an LLM does not guarantee anything on the file that users run locally: a quantified version, produced by a transformation that no one fully documents. Two researchers measured for the first time the share of the GGUF ecosystem that a third can regenerate bit for bit. Of the 200 GGUF repositories most downloaded from Hugging Face (198), none described their compilation environment: the independent verifiability rate fell to 0 out of 198.

Reproducible, provided that all

However, files are not irreproducible. On four repositories that publish their parameters, the template file only does not contain any Q4 K M files; but by adding two missing information — the contraction setting of floating point calculations and, for hybrid models, the tensor tensor quantification recipe — 7 out of 8 files are reconstructed identically on x86 (5 out of 8 on ARM). The calibration matrix, on the other hand, must be delivered and bound by fingerprint, as recalculating it does not give the same result. A single compiler setting is sufficient to obtain an identical output on all the platforms tested, for a negligible quality cost (+0.06% perplexity).

On the safety side, the authors show that a statistical detection distinguishes normal numerical noise from gross alteration, but that a targeted modification of the bindings passes under radar: only an exact reconstruction offers a real guarantee. They publish their audit data and a tool to repeat the exercise.