The Post-Transformer Era
3 architectures competing to replace the transformer
Tranformers won the hardware lottery and have driven remarkable progress in AI capabilities. They are the basis of most text-based AI models (the ‘GPT’ in chatGPT stands for ‘Generative Pre-trained Transformer’). But some researchers think the architecture is now a bottleneck.
One of the co-inventors of the Transformer (Llion Jones) recently said that overindexing research efforts on a single architecture means the community is in a ‘local minimum’ and there is something ‘not quite right’ about current LLMs: ‘It’s unfortunate that these models work so well because it’s too easy for people to sweep these problems under the carpet.’
Similarly, Ilya Sutskiver recently commented that scaling the current architecture was not enough to get to AGI:
“One consequence of the age of scaling is that... scaling sucked out all the air in the room. And so because scaling sucked out all the air in the room, everyone started to do the same thing. We got to the point where we are in a world where there are more companies than ideas.”
Many labs are still betting that scaling up the transformer will be enough - Nvidia’s new Blackwell chips (GB200s) arriving in 2026 will push the current paradigm further. But it’s possible that this year when post-transformer text models become mainstream. Here are 3 big ideas for how to do that:
Text Diffusion Models
If transformer-based LLMs are like typists, text diffusion models are like sculptors. LLMs write text from left to right (in English) which means they look backwards and can only output one token at a time. Yann Lecun has said this constraint to output one token after another means transformer-based models can’t plan.
While (auto-regressive) transformer models generate from left to right, diffusion models work by learning to denoise whole sequences of text at the same time. This means they better account for the the structure and shape of the sentence, can plan better and can generate more quickly.
For example, if a transformer model might generate “The quick brown fox jumps” in order: (‘The’, ‘quick’, ‘brown’, ‘fox’, ‘jumps’) looking back at the previous word each time, diffusion models don’t generate tokens one-by-one. Instead they denoise an entire sequence at the same time.
Like a sculptor with a marble block, the text diffusion model chips away at embeddings initialised with random noise. It is iterative refinement, instead of sequential generation. Unlike transformers they can also parallelise generation, which means they can be much faster for longer sequences.
It’s surprising that this works. Diffusion models were developed for images originally and is based on the math of physics (fluids, heat, noise) but language is based on categories. Indeed, when you add noise to an image it blurs the image but the image is still in some sense dicepherable. If you add noise to words (by adding noise to word embeddings) the words no longer make sense. In this way words are discrete while pixellated images are not.
Continuous Thought Machines
CTMs (which got the spotlight at NEURIPS this year) introduce temporal processing to LLMs. In standard transformers information moves through the model in standardised steps, like a car on an assembly line. Every single word (token) passes through a fixed number of layers and the model spends exactly the same amount of computational energy on each step.
Rather than using activation functions (like ReLU) CTMs use time-varying interactions between neurons. Instead of just passing numbers (activations) between layers, CTMs use the synchronisation of firing patterns between neurons as a way to represent information. The representation isn’t the neuron values, but the synchronisation between the activity history of the neuron pairs. Therefore, in a CTM, the model can "pause" on a specific input and run its internal neural dynamics for as long as it needs to understand it. This is similar to how you might glance at a simple photo but stare at a complex diagram for minutes. This Neural Synchronisation allows for adaptive compute. The model can think fast for easier queries and slowly for more complex ones. In other words, the meaning is in the timing.
Nested Learning
Demis Hassabis and Sam Altman have described AI’s inability to learn on the job as a major bottleneck to AI. Researchers at Google, inspired by the human brain, believe they have a solution to this problem. Their ‘Nested learning’ approach adds more intermediate layers of memory which update at different speeds. Each of these intermediate layers is treated as a separate optimisation problem to create a hierarchy of nested learning processes. They believe this could help models continually learn on-the-fly.
Critically, in their HOPE architecture (High-order Optimization and Parameter Evolution) they enable the model to learn how to learn. The fast loop trains in a way that is similar to current architectures (Predicts the next token, computes the loss and updates its weights). It then sends information (in the form of gradients) up to the slower loop. In return the slower loop sends the fast loop back context (hyper-parameters and latent states). This same process happens between the slower loop and the slowest loop. Gradient flows go up and context flows come down.
This feedback mechanism between faster and slower learning loops enables a meta-learning system. The slow loops learn how the fast loops learn. They provide stability and learning tips (in the form of their longer-term guidance and learning tips, in the form of hyperparameters). This allows the system as a whole to continually improve as learners and not just as performers and avoid the catastrophic forgetting issue which has cursed ML models.
It’s far from certain this will work though. In the paper they prove the efficacy of the model on a small scale (~1.3b parameter model) but it would need to be proved on a much larger scale (Gemini 3 was 1 trillon parameters). The more tricky problem is how the model actually works out what to keep in long-term memory. This is still unsolved.
Optimizing the Transformer (Efficiency Hacks)
Even without a total paradigm shift, researchers are finding ways to break the transformer’s limits.
DeepSeek 3.2 & Sparse Attention: Dense attention is expensive because every new token usually has to look back at all previous tokens (quadratic cost). DeepSeek 3.2 introduced dynamic learnable sparsity, where the model learns which tokens to ignore, drastically reducing compute costs for long contexts.
Gated Attention: This approach adds gating mechanisms to standard attention layers. It solves the ‘attention sink’ problem (where models irrationally focus on the very first token) and improves training stability, allowing models to learn faster without collapsing.
Titans (Neural Long-Term Memory): Google’s new ‘Titans’ architecture (the precursure to the HOPE architecture) adds a Neural Long-Term Memory module that learns while it runs. Unlike standard models with ‘frozen’ weights, Titans can update its memory in real-time, effectively allowing it to remember context far beyond the standard window.