Kimi K3技术报告发布

Kimi K3 just released their technical paper. one of the most detailed and exhaustive one. A million-token context window does not create long-horizon agency if the training system cannot preserve a task across thousands of tool calls. Kimi K3 is built around that constraint. Reports that the architecture and training recipe improve overall scaling efficiency by roughly 2.5X over Kimi K2. - Most layers no longer use full softmax attention. Each block runs 3 Kimi Delta Attention (KDA) layers and one global Multi-head Latent Attention (MLA) layer; - No Position Encoding (NoPE): removes Rotary Position Embedding (RoPE) extension tricks, while bounded KDA decay makes the difficult causal tiles Tensor Core-friendly. - Their agents are trained not to depend on one fixed scaffold. Tools, prompts, memory, context management, skills, and subagents are varied during RL, while partial rollouts and resumable microVM sandboxes allowed unfinished trajectories continue across training iterations with their environment state intact. - The model attends across depth, not only across tokens. Attention Residuals let every layer selectively retrieve earlier block representations instead of squeezing the entire network history through one accumulated residual stream. - The million-token context was trained as an actual dependency problem. The curriculum grows from 8K to 64K, then 256K to 1M, using synthetic multimodal tasks whose evidence is deliberately scattered across the full sequence. - The 896-expert MoE required a complete stability redesign. Kimi K3 combines a half-width latent expert path, 16 active experts per token, RMSNorm, bounded SiTU-GLU activations, and Quantile Balancing that directly computes the routing bias needed for each expert’s target load.  > **引用原帖 Kimi.ai (@Kimi_Moonshot):** > Releasing the model weights and technical report of Kimi K3. > Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window. > New model architecture: 2.5x the intelligence per unit of compute, not just more params. > Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale. > Model weights: https://t.co/7m7eEg6Y0B > Tech report: https://t.co/yeu6cjpMCT > Tech blog: https://t.co/YTfiMSNM1f > https://x.com/Kimi_Moonshot/status/2081760186235289764 Rohan Paul (@rohanpaul_ai): https://t.co/b8ioyzQjOx