#pretraining

1 article tagged pretraining.

TopicRepliesVotesActivity

Attention Residuals: Kimi Team swaps the fixed residual sum for learned attention over depth
AI Research Interpretability kimi attention-residuals llm-architecture residual-connections
Kimi's Attention Residuals paper replaces the fixed residual sum with softmax attention over layer outputs. The mechanism, the 1.4T-token test, and the caveats.
0 replies0 votes1mo
0reply
0vote
1mo