FIT-GGUF 让量化按目标体积逐张量调配精度

AI 导读

FIT-GGUF 推出一种新的 GGUF 量化方式，用户可直接给定目标体积或保真档位，由它逐张量混合精度来匹配目标，OpenBMB 在 MiniCPM5-2B 上给出四档结果：Quality 1.46 GiB、Balanced 1.28 GiB、Compact 1.21 GiB、Mini 1.14 GiB。

正文

原文

当前显示原文，中文译文尚未提供。

GGUF quantization may be getting a size slider of sorts with a new FIT-GGUF quants!

Normally, when downloading a local model, you choose a quant level

Q3 → Q4 → Q5 → Q6

…and hope it fits your RAM/VRAM budget.

FIT-GGUF flips that around.

Give it a target size or fidelity tier, and it can mix precision tensor-by-tensor to find a GGUF that meets that target.

OpenBMB highlighted it on MiniCPM5-2B:

📦 Quality — 1.46 GiB

⚖️ Balanced — 1.28 GiB

🗜️ Compact — 1.21 GiB

🤏 Mini — 1.14 GiB

Those labels represent progressively looser fidelity to the BF16 model, measured primarily with KL divergence and not different skills like coding vs. reasoning.

And they run in normal llama.cpp / LM Studio.

⚠️ FIT does not claim its tensor allocation is universally optimal yet.

🔗 Link in ALT

来源：TeksEdge · x.com