面壁智能 MiniCPM-V 4.7 用 LES 加速端侧 MoE 推理
面壁智能 OpenBMB 介绍 MiniCPM-V 4.7 采用的 Lookahead Expert Scheduling（LES），用于解决 35B MoE 模型在有限内存设备上按需从 SSD 加载专家权重导致生成卡顿的问题。
⚡ Lookahead Expert Scheduling: Making On-Device MoE Inference Faster
Running a 35B MoE model on a device with limited memory presents a key challenge: expert weights can be stored on SSD, but loading them on demand can stall generation.
MiniCPM-V 4.7 addresses this with LES, which coordinates expert loading and computation rather than waiting for every required expert to load before continuing.
🔹 Expert prefetch and compute scheduling: LES predicts which experts the next layer may need through cross-layer routing prediction. It prioritizes computation for experts already loaded and runs their computation while loading the remaining required experts in parallel. Predictions guide preloading and scheduling only; the original token-level router still determines the experts used for actual computation. No additional training is required.
🔹 Optimized weight access: Fine-grained, layer-wise mixed-precision quantization reduces weight storage and transfer overhead while preserving model capability. Reorganizing expert weights into contiguous layouts also reduces scattered reads and improves sequential flash-storage access.
Lookahead Expert Scheduling: github.com/tc-mb/llama.cpp-o…
来源：OpenBMB · x.com