# Attention Residuals: Kimi Team swaps the fixed residual sum for learned attention over depth > Kimi's Attention Residuals paper replaces the fixed residual sum with softmax attention over layer outputs. The mechanism, the 1.4T-token test, and the caveats. - Canonical URL: https://alfreds.org/t/attention-residuals-kimi-ais-elegant-llm-architecture-breakthrough - Author: Alfred Kolakkal (https://alfreds.org/u/alfred) - Published: 2026-08-13 - Category: Interpretability - Tags: kimi, attention-residuals, llm-architecture, residual-connections, deepseek, pretraining ## Key takeaway Attention Residuals (AttnRes), a March 2026 paper from the Kimi Team (arXiv:2603.15031), replaces the fixed unit-weight accumulation of standard PreNorm residual connections with softmax attention over preceding layer outputs, letting each layer aggregate earlier representations with learned, input-dependent weights. A block variant, Block AttnRes, partitions layers into blocks to cut the memory and communication cost, and the authors describe it as a drop-in replacement with minimal overhead. Integrated into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-trained on 1.4T tokens, AttnRes produced more uniform output magnitudes and gradient distributions across depth and improved all evaluated downstream tasks, with scaling-law experiments indicating the gain is consistent across model sizes. ## Provenance This is an aggregated research report. Sources were detected automatically and the draft was written by kimi-k3 under a fixed editorial policy and is pending human review. Confidence: medium. ### Primary sources cited - mHC: Manifold-Constrained Hyper-Connections — https://arxiv.org/abs/2512.24880 (arXiv:2512.24880; 2025-12-31) - Attention Residuals — https://arxiv.org/abs/2603.15031 (arXiv:2603.15031; 2026-03-16) ### Reporting that surfaced this story - bycloud: Attention Residuals: Kimi AI's Elegant LLM Architecture Breakthrough — https://www.youtube.com/watch?v=iw1VF8HOCrk ## Article ## What are Attention Residuals? Attention Residuals (AttnRes) are a proposed replacement for the standard residual connection in large language models, described in a [Kimi Team paper](https://arxiv.org/abs/2603.15031) (arXiv:2603.15031, 16 March 2026). Instead of adding every preceding layer's output into the hidden state with a fixed weight of 1, each layer computes softmax attention over the outputs of the layers before it and mixes them with learned, input-dependent weights. The story reached us through bycloud's video coverage; this article is based on the paper itself and on the DeepSeek mHC paper it is most often compared with. Because the evidence available here is the papers' abstracts rather than their full texts, the claims below are limited to what the abstracts state. ## Why change the residual connection at all? Because, the paper argues, the standard version dilutes every layer's contribution as models get deeper. Residual connections with PreNorm are the standard design in modern LLMs, but they accumulate all layer outputs with fixed unit weights. According to the abstract, this uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution — the deeper the network, the smaller any single layer's share of the running sum. ## What does the paper actually propose? The full version, AttnRes, replaces the fixed accumulation with softmax attention over all preceding layer outputs, so aggregation becomes selective and content-dependent. The abstract is direct that this is expensive: attending over every preceding layer output imposes memory and communication overhead that is impractical for large-scale training. The practical version is Block AttnRes. It partitions the layers into blocks and attends over block-level representations rather than individual layer outputs, reducing the memory footprint while, in the authors' words, preserving most of the gains of full AttnRes. Combined with cache-based pipeline communication and a two-phase computation strategy, the authors describe Block AttnRes as a practical drop-in replacement for standard residual connections with minimal overhead.
## What evidence does the paper give? Three lines of evidence, all from the authors' own experiments. First, scaling-law experiments, which the abstract says confirm the improvement is consistent across model sizes. Second, ablations that the authors say validate the benefit of content-dependent depth-wise selection — that is, the gains come from the learned, input-dependent weighting rather than from some incidental side effect. Third, and most substantial: the team integrated AttnRes into the Kimi Linear architecture — 48 billion total parameters with 3 billion activated — and pre-trained on 1.4 trillion tokens. There, the abstract reports, AttnRes mitigates PreNorm dilution, producing more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks. The abstract does not list per-task scores, so no task-level numbers are given here. ## What is genuinely new here? The novel element is making depth-wise aggregation learned and input-dependent, while leaving the per-layer outputs themselves alone. Reworking the residual connection is not new as a research direction. Hyper-Connections (HC) had already expanded the residual stream's width and diversified its connectivity, and DeepSeek's [mHC paper](https://arxiv.org/abs/2512.24880) (arXiv:2512.24880, 31 December 2025) argues that this diversification breaks the identity-mapping property that makes residual connections train stably — causing severe training instability, restricted scalability, and notable memory access overhead. mHC's fix is to project HC's connection space onto a specific manifold to restore identity mapping. AttnRes attacks the same interface from the opposite direction: rather than widening the residual stream and then constraining it, it keeps the standard per-layer outputs and learns how much each should count at every step. The two papers are best read as competing answers to the same question — what should replace the decade-old fixed residual sum — rather than as one building on the other. ## Where does the coverage go beyond the paper? Only one report exists in the evidence so far, and its framing is stronger than the paper's. bycloud's video calls AttnRes an "elegant breakthrough" and judges it "even more promising than DeepSeek's mHC" — that comparison is the reporter's verdict, not a claim either paper makes, and mHC is not mentioned in the AttnRes abstract. No transcript of the video was available for this article, so no specific claims from it beyond its title and description are assessed here. The paper's own language is measured: it claims a consistent scaling-law improvement, gains preserved "mostly" by the block approximation, and better performance on the tasks it evaluated. ## What are the limitations? The paper states the main one itself: full AttnRes has enough memory and communication overhead to motivate the block variant, and Block AttnRes preserves only "most" of the full version's gains — the approximation costs something. Beyond that, the 48B-parameter validation was run inside Kimi Linear, a linear-attention architecture; the abstract does not state which architecture the scaling-law runs used, so transfer to conventional full-attention transformers is not established by the evidence here. All results are the authors' own, with no independent replication in this bundle. And because this article works from the abstract, the per-benchmark numbers and the details of the scaling-law fits — the evidence a reader would need to judge the size of the effect — are not yet in hand. What would change the picture: the full paper's per-task results, an independent reproduction in a standard transformer stack, and a controlled head-to-head against mHC-style constraints at matched compute. ## Sources ### Primary sources - [mHC: Manifold-Constrained Hyper-Connections](https://arxiv.org/abs/2512.24880) — Zhenda Xie, Yixuan Wei, Huanqi Cao et al.; 2025-12-31; arXiv:2512.24880 - [Attention Residuals](https://arxiv.org/abs/2603.15031) — Kimi Team, Guangyu Chen, Yu Zhang et al.; 2026-03-16; arXiv:2603.15031 ### Reporting that surfaced this - bycloud — [Attention Residuals: Kimi AI's Elegant LLM Architecture Breakthrough](https://www.youtube.com/watch?v=iw1VF8HOCrk) (2026-05-18) This is an aggregated research report. Sources were detected automatically, the primary material was fetched and read, and the draft was written by kimi-k3 under a fixed editorial policy, then reviewed by a human before publication. Evidence behind the claims here: 12 documented, 1 reported. Claims marked as reported are attributed to the reporter in the text; only claims traceable to a primary source are stated as fact. Article confidence: **medium**. [How this system works](/research/method) · [Sources it watches](/research/sources) --- Source: https://alfreds.org/t/attention-residuals-kimi-ais-elegant-llm-architecture-breakthrough When citing this article, attribute it to Alfred Kolakkal and link to https://alfreds.org/t/attention-residuals-kimi-ais-elegant-llm-architecture-breakthrough.