MiniCPM-V 4.7 用 LES 加速端侧 MoE 推理
MiniCPM-V 4.7 提出 Lookahead Expert Scheduling（LES），通过跨层路由预测提前预取专家权重，让已加载专家的计算与其余专家的加载并行，解决 35B MoE 模型在端侧按需从 SSD 加载专家导致的生成卡顿，且无需额外训练。
⚡ Lookahead Expert Scheduling: Making On-Device MoE Inference Faster
Running a 35B MoE model on a device with limited memory presents a key challenge: expert weights can be stored on SSD, but loading them on demand can stall generation.
MiniCPM-V 4.7 addresses this with LES, which coordinates expert loading and computation rather than waiting for every required expert to load before continuing.
🔹 Expert prefetch and compute scheduling: LES predicts which experts the next layer may need through cross-layer routing prediction. It prioritizes computation for experts already loaded and runs their computation while loading the remaining required experts in parallel. Predictions guide preloading and scheduling only; the original token-level router still determines the experts used for actual computation. No additional training is required.
🔹 Optimized weight access: Fine-grained, layer-wise mixed-precision quantization reduces weight storage and transfer overhead while preserving model capability. Reorganizing expert weights into contiguous layouts also reduces scattered reads and improves sequential flash-storage access.
Lookahead Expert Scheduling: github.com/tc-mb/llama.cpp-o…
来源：OpenBMB · x.com