TITANS & MIRAS: real continual learning
MIRAS = a unifying theory of transformers (attention) and state space models (SSM, e.g. Mamba, RNNs)
TITANS = an optimal MIRAS implementation that’s “halfway between” SSM & transformer with a CL memory module
let’s dive in!
research.google/blog/titans-...
TITANS & MIRAS: real continual learning
View original threadTITANS
this introduces a “continual learning” module, which is a whole new neural net
the NN processes the full input while the transformer (with regular attention) also processes the full input
but the transformer also receives the output of the memory module
this introduces a “continual learning” module, which is a whole new neural net
the NN processes the full input while the transformer (with regular attention) also processes the full input
but the transformer also receives the output of the memory module
![Horizontal flow diagram showing a memory-augmented transformer across two time steps.
Top left header: “Time Step T-1 (Past State).” Below it, a pale blue rounded box reads “Processed Input Chunk (T-1).” Along the midline is a dotted horizontal axis labeled “Sequence Timeline.”
At the lower left on this past side is an orange rectangular block labeled:
“Titans Memory Module (Deep MLP) [State T-1].”
Underneath, in smaller text:
“Evolving Weights. Unlike linear SSMs, this is a multi-layer network for non-linear memory.”
A right-pointing orange arrow labeled “Retrieval (Forward Pass)” leaves this module toward the center, labeled along the side “Memory Context Vector.”
The right half is labeled at the top: “Time Step T (Current Processing).” At the top center is a pale blue box: “Current Input Chunk (T).” A vertical cyan arrow carries this down into another orange block on the upper right, also labeled “Titans Memory Module (Deep MLP) [State T].”
From the past-state memory arrow and the current input, arrows merge into a central light-blue box labeled “Combined Context [Memory + Current Input].” A downward arrow leads to a large gray block:
“Transformer Core (Standard Attention)
Fixed Weights. Uses immediate attention on combined context.”
Below this is another pale blue box: “Output Predictions (T).”
On the right side, the upper-right Titans Memory Module participates in a learning loop: a magenta arrow labeled “Surprise Calculation (Gradient)” goes downward into a magenta circle, then to another orange block at mid-right labeled again “Titans Memory Module (Deep MLP) [State T].” A magenta arrow from this circle back to the upper-right module is labeled “Update Weights (Test-Time Learning).” From the mid-right Titans block, an orange arrow points rightward labeled “To Time Step T+1.”
An orange arrow from the gray Transformer Core also feeds upward into the mid-right Titans memory module, completing the interaction between predictions and evolving memory.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:ckaz32jwl6t2cno6fmuw2nhn/bafkreic4qzc47reub6vvlihdf7kv25kezkrhh45usom5mxrwxl5usly7d4@jpeg)
2
transformers can consume input the size of entire books, and attention works astoundingly well to recall the right parts
but their capabilities drop precipitously as the input increases
TITANS chunks the input into small episodes and updates itself between them
but their capabilities drop precipitously as the input increases
TITANS chunks the input into small episodes and updates itself between them
2
the thing is, TITANS works crazy well
it’s getting recall at 10M tokens that’s considered up with SOTA for <1M tokens
it’s getting recall at 10M tokens that’s considered up with SOTA for <1M tokens
3
1
TITANS uses “surprise” as a method of deciding what to remember
TITANS is framed here as being for long context, but it really is for continual learning
yes, one long input can be chunked into episodes, or it can just be on-the-job learning day after day
yes, one long input can be chunked into episodes, or it can just be on-the-job learning day after day
3
the MIRAS paper is a theoretical breakthrough. Anything with these 4 things fits
1. updatable memory
2. attention bias
3. retention gate
4. memory algorithm
they show how both transformers & SSMs implement this framework, and it helped them discover a more optimal TITANS
arxiv.org/abs/2504.13173
1. updatable memory
2. attention bias
3. retention gate
4. memory algorithm
they show how both transformers & SSMs implement this framework, and it helped them discover a more optimal TITANS
arxiv.org/abs/2504.13173
5