Hugging Face H-Nets 新版本在原始字节建模中提升训练效率
Hugging Face 团队发布 H-Nets 新版本，在受控 scaling law 实验中实现最高 3-5x 效率提升，并在更大规模上以约少 10 倍的数据和计算匹配一个使用 11T tokens 训练的开源模型。该研究以原始字节语言建模为方向，探索从数据中直接学习抽象的通用构建模块。
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
来源：_albertgu · x.com