Study the paper

Kimi Linear: An Expressive, Efficient Attention Architecture

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Introduction to Kimi Linear and KDA

Kimi Delta Attention (KDA) and Hybrid Architecture

Sources

block

Kimi Linear: An Expressive, Efficient Attention Architecture T ECHNICAL R EPORT 1 Introduction As large language models (LLMs) evolve into increasingly capable agents [50], the computational demands of inference—particularly in long-horizon and reinforcement learning (RL) settings—are becoming a central bottleneck. This shift toward RL test-time scaling [95, 33, 80, 74, 53], where models must process extended trajectories, tool-use interactions, and complex decision spaces at inference time, exposes fundamental inefficiencies in standard attention mechanisms. In particular, the quadratic time complexity and the linearly growing key–value (KV) cache of softmax attention introduce substantial computational and memory overheads, hindering throughput, context-length scaling, and real-time interactivity. Linear attention [48] offers a principled approach to reducing computational complexity but has historically under- performed softmax attention in language modeling—even for short sequences—due to limited expressivity. Recent advances have significantly narrowed this gap, primarily through two innovations: gating or decay mechanisms [92, 16, 114] and the delta rule [84, 112, 111, 71].…

At the core of Kimi Linear is Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet. Unlike traditional architectures that employ coarse head-wise forget gates, KDA introduces a fine-grained, channel-wise gating mechanism where each feature dimension maintains an independent forgetting rate. This allows for precise regulation of the finite-state RNN memory.

Sources

block

Kimi Linear: An Expressive, Efficient Attention Architecture T ECHNICAL R EPORT 1 Introduction As large language models (LLMs) evolve into increasingly capable agents [50], the computational demands of inference—particularly in long-horizon and reinforcement learning (RL) settings—are becoming a central bottleneck. This shift toward RL test-time scaling [95, 33, 80, 74, 53], where models must process extended trajectories, tool-use interactions, and complex decision spaces at inference time, exposes fundamental inefficiencies in standard attention mechanisms. In particular, the quadratic time complexity and the linearly growing key–value (KV) cache of softmax attention introduce substantial computational and memory overheads, hindering throughput, context-length scaling, and real-time interactivity. Linear attention [48] offers a principled approach to reducing computational complexity but has historically under- performed softmax attention in language modeling—even for short sequences—due to limited expressivity. Recent advances have significantly narrowed this gap, primarily through two innovations: gating or decay mechanisms [92, 16, 114] and the delta rule [84, 112, 111, 71].…

The Kimi Linear model is constructed by interleaving $N$ KDA layers with one Multi-Head Latent Attention (MLA) layer for token mixing, where $N$ is set to 3 (a uniform 3:1 ratio). This layerwise hybrid design provides an optimal trade-off between quality and throughput.

Sources

block

Kimi Linear: An Expressive, Efficient Attention Architecture T ECHNICAL R EPORT N × Norm KDA Norm MoE 1 × Norm MLA Norm MoE Inputs Linear Conv L2 Linear Conv L2 Linear Conv σ σ σ Kimi Delta Attention Norm Linear Outputs Router 1 N s · · · 1 2 3 N r · · · Shared Expert Routed Expert Outputs Figure 3: Illustration of our Kimi Linear model architecture, which consists of a stack of blocks containing a token mixing layer followed by a MoE channel-mixing layer. Specifically, we interleave N KDA layers with one MLA layer for token mixing, where N is set to 3 in our implementation. used in GDN and Mamba [111, 16]. Before the output projection through W o ∈ R d × d , we use a head-wise RMSNorm [122] and a data-dependent gating mechanism [79] parameterized as: o t = W o Sigmoid W ↑ g W ↓ g x t ⊙ RMSNorm (KDA ( q t , k t , v t , α t , β t )) (10) Here, the output gate adopts a low-rank parameterization similar to the forget gate, to ensure a fair parameter comparison, while maintaining performance comparable to full-rank gating and alleviating the Attention Sink [79]. The choice of nonlinear activation function is further discussed in §5.2. Hybrid model architecture Long - context retri…

Additionally, Kimi Linear applies No Position Encoding (NoPE) to all full attention (MLA) layers. This delegates the entire responsibility of encoding positional information and recency bias to the KDA layers, establishing KDA as the primary position-aware operator.

Sources

block

Kimi Linear: An Expressive, Efficient Attention Architecture T ECHNICAL R EPORT N × Norm KDA Norm MoE 1 × Norm MLA Norm MoE Inputs Linear Conv L2 Linear Conv L2 Linear Conv σ σ σ Kimi Delta Attention Norm Linear Outputs Router 1 N s · · · 1 2 3 N r · · · Shared Expert Routed Expert Outputs Figure 3: Illustration of our Kimi Linear model architecture, which consists of a stack of blocks containing a token mixing layer followed by a MoE channel-mixing layer. Specifically, we interleave N KDA layers with one MLA layer for token mixing, where N is set to 3 in our implementation. used in GDN and Mamba [111, 16]. Before the output projection through W o ∈ R d × d , we use a head-wise RMSNorm [122] and a data-dependent gating mechanism [79] parameterized as: o t = W o Sigmoid W ↑ g W ↓ g x t ⊙ RMSNorm (KDA ( q t , k t , v t , α t , β t )) (10) Here, the output gate adopts a low-rank parameterization similar to the forget gate, to ensure a fair parameter comparison, while maintaining performance comparable to full-rank gating and alleviating the Attention Sink [79]. The choice of nonlinear activation function is further discussed in §5.2. Hybrid model architecture Long - context retri…

$o_t = W_o \text{Sigmoid} \left( W_{\uparrow g} W_{\downarrow g} x_t \right) \odot \text{RMSNorm} (\text{KDA} ( q_t , k_t , v_t , \alpha_t , \beta_t ))$

Sources

block

Kimi Linear: An Expressive, Efficient Attention Architecture T ECHNICAL R EPORT N × Norm KDA Norm MoE 1 × Norm MLA Norm MoE Inputs Linear Conv L2 Linear Conv L2 Linear Conv σ σ σ Kimi Delta Attention Norm Linear Outputs Router 1 N s · · · 1 2 3 N r · · · Shared Expert Routed Expert Outputs Figure 3: Illustration of our Kimi Linear model architecture, which consists of a stack of blocks containing a token mixing layer followed by a MoE channel-mixing layer. Specifically, we interleave N KDA layers with one MLA layer for token mixing, where N is set to 3 in our implementation. used in GDN and Mamba [111, 16]. Before the output projection through W o ∈ R d × d , we use a head-wise RMSNorm [122] and a data-dependent gating mechanism [79] parameterized as: o t = W o Sigmoid W ↑ g W ↓ g x t ⊙ RMSNorm (KDA ( q t , k t , v t , α t , β t )) (10) Here, the output gate adopts a low-rank parameterization similar to the forget gate, to ensure a fair parameter comparison, while maintaining performance comparable to full-rank gating and alleviating the Attention Sink [79]. The choice of nonlinear activation function is further discussed in §5.2. Hybrid model architecture Long - context retri…