Why is AI shale-shaped?

The material contingency of the AI boom

The human brain is roughly a million times more energy-efficient per bit of sensory information processed than current frontier AI models.¹ A child learns language from roughly 10⁷–10⁸ words of input by age 10. GPT-4 class models were trained on ~10¹³ tokens. This is a 100,000× gap in data efficiency.

How did we end up with AI that is incredibly powerful but also clunky, messy and energy inefficient?

The AI we currently have took a very specific path that was shaped by hardware (the GPU), the energy landscape (specifically cheap fracked gas for data centres) and the financial response to 2008 (cheap credit from quantitative easing, which made fracking viable).

‘Scaling Laws’ and ‘The Bitter Lesson’ (that more compute beats human-designed algorithmic improvement) are important, but relevant mainly within this material context. Many other possible paths still remain untested because they never had such favourable conditions.

Leading AI researchers believe we have over-indexed on the transformer (currently the dominant AI architecture). One of the co-inventors of the Transformer (Llion Jones) recently said that concentrating all research efforts on a single architecture means the community is in a ‘local minimum’ and there is something ‘not quite right’ about current LLMs: ‘It’s unfortunate that these models work so well because it’s too easy for people to sweep these problems under the carpet.’

Similarly, Ilya Sutskiver recently commented that scaling the current architecture was not enough to get to AGI:

“One consequence of the age of scaling is that... scaling sucked out all the air in the room. And so because scaling sucked out all the air in the room, everyone started to do the same thing. We got to the point where we are in a world where there are more companies than ideas.” This sentiment and others point towards the fact that in many ways, engineering became more important than science in driving AI.²

The deep learning revolution (2012 onwards) maps closely to the era of cheap fracked gas and zero interest rates. While the fed balance sheet has grown there has been a 57% collapse in US natural gas prices (which is the primary fuel for data centres) from $6.87 to $2.97.

GPU-shaped AI

Did the neural net save the GPU or did the GPU save the neural net? Sarah Hooker has powerfully argued that the reason neural networks have (so far) won the battle between AI approaches is because they play well with hardware. Indeed, neural networks didn’t work very well for a long time (from ~1950s-2012). Minsky and Papert explained that neural networks would never work. Yann Lecun, Geoff Hinton and others were considered to be fringe advocates for neural networks in the 1990s. The software ecosystem at that time supported ‘symbolic’ forms of AI (this included the programming languages LISP and Prolog) which meant that ‘connectionists’ (those who supported neural networks) generally had to implement their algorithms in very low-level languages, such as C++.³ This all changed with the GPU.

The AlexNet result in 2012, 5 years after Nvidia invented CUDA, was the closest thing to AI’s ‘big bang’ moment. As part of the ‘ImageNet’ image classification competition, Geoff Hinton and two of his PhD students (Alex Krizhevsky and Ilya Sutskever) trained an image model on two NVIDIA GTX 580 GPUs. Nvidia had built their CUDA software ecosystem for gamers and graphics rendering, but it played perfectly with their AlexNet vision model which classified pictures of cats and dogs 11% more accurately than the next entry to the competition. The improvements in accuracy were astounding, and the process was also a lot faster. For comparison: in 2012, a Google paper used 16,000 CPU cores to classify cats; one year later, the same task required only 2 CPU cores and 4 GPUs.

The Transformer architecture (created in 2017) plays particularly well the GPUs. It involves ‘self-attention’ which means the model can reflexively attend to words in each sequence and weigh the relative importance of each word/ token. Critically, this means you can run attention calculations in parrallel (which was not possible with older models, for example Recurrent Neural Networks, Long-Short-Term Memory Machines or symbolic AI methods). Transformer ‘attention’ was perfectly suited to the brute force of the GPU.

Indeed, an innovation in 2022 (flash attention) knitted the transformer and GPU together even more closely. Developed by Tri Dao, FlashAttention works by exploiting the specific memory hierarchy of Nvidia GPUs. It's an algorithm designed from the ground up around the physical properties of Nvidia hardware (the relationship between HBM and SRAM on the chip). It is flash attention which makes the long context windows in modern LLMs possible.

Hooker argues that different computing paradigms could have led to very different forms of AI. ‘biological hardware, analog hardware with in-memory computation, neuromorphic computing, optical computing, and quantum computing based approaches’ might have produced very different outcomes. Neuromorphic computing (for example Intel's Loihi, IBM's TrueNorth) mimics biological neural processing and achieves intelligence-like computation at 20 watts (equivalent to an electric shaver) versus hundreds of watts for GPUs.⁴ Loihi or TrueNorth chips are around 1000x more energy efficient than an Nvidia chip.

It seems the availability of high-end GPUs and cheap energy meant US AI labs haven’t prioritised efficiency.⁵ DeepSeek R1 is perhaps the most dramatic demonstration that the brute-force scaling paradigm is not the only path to capability, even within the transformer family. It has 671 billion total parameters, but only 37 billion activated per token, a Mixture-of-Experts (MoE) architecture where the model “picks” which experts to use for each task. It was Trained for $5.6 million (2.788M H800 GPU hours), which is less than 1/10th the compute of Meta’s Llama 3.1 405B, which it outperforms. For comparison, GPT-4 is estimated to have cost $50-100 million to train.

DeepSeek achieved this through meticulous engineering: Multi-head Latent Attention (MLA) for memory efficiency, FP8 mixed-precision training, and novel pipeline parallelism algorithms. DeepSeek demonstrates that the “just scale up” approach of Western labs is partly a choice shaped by capital abundance, not a technical necessity. When constrained (by US export controls limiting DeepSeek to H800 chips rather than the more powerful H100s), Chinese researchers found a way to achieve comparable results at a fraction of the cost. Constraint bred efficiency. More money and more compute doesn’t mean inevitable progress. Deepseek’s recent innovation (sparse attention) is another step in the direction of greater efficiency.

Google also operates under slightly different hardware conditions than other major western labs. They have developed their own hardware (the TPU) which is ~10x more efficient than Nvidia GPUs (though overall less powerful). This means that they are less bought-in to the GPU-powered scaling paradigm and have consistently experimented with more approaches (text diffusion for example). Their 2022 paper Chinchilla significantly changed how the field understood scaling laws, showing that developers were focusing too much on scaling the size of models and should focus on scaling the volume of data (not just the size of the model).

This points to a disconcerting insight. We don’t really have a good theory for whyany particular scale works. Scaling laws are empirical and not theoretical. Chinchilla was one example of where the field was overindexing on a particular type of scaling (model size). The frontier race is still fundamentally about training the biggest models - GPT 5.2, Gemini 3, Opus 4.6. So it's entirely possible there are more efficient paths we're missing because the field gravitates toward whatever is most straightforward to scale.

Within our current GPU and CUDA-dominated paradigm the type of AI we get is relatively homogenous and, as Llion Jones and Ilya Sutskever have argued, alternative ideas struggle to be heard. Structured State Space Sequence model (S4), developed by Albert Gu at Carnegie Mellon, and its successor Mamba represent one of the most developed challenges to transformer dominance. Mamba has a linear computational complexity (compared to transformer quadratic complexity). This means as sequence length the transformer’s attention mechanism becomes computationally very expensive. Mamba also achieves 5x inference throughput versus transformers of similar size. While transformers process every token in relation to every other token in the attention mechanism, SSMs maintain a compressed ‘state’ that summarises previous tokens (a little more how biological memory works). S4 comes from control theory and signal processing and comes from a different (and more efficient) computational paradigm.⁶

Despite this, Mamba has actually been developed to be ‘hardware-aware’ and optimised for GPUs, showing that even alternatives to the transformer are being adapted to our hardware ecosystem. One of the developers of Mamba (Tri Dao) has admitted that the transformer is here to stay: "If you want the best performing model that runs most efficiently on my hardware that has the most support on the software side, the Transformer is a safe bet. I think it's here to stay... People tend to be conservative."

Of course the transformer has been immensely successful. But if neuromorphic computing had received the same investment and ecosystem support that GPUs got (through gaming, then crypto, then ML), we might have AI systems that look radically different (more event-driven, spiking, low-power, potentially more biologically plausible). The fact that we got the brute-force scaling paradigm (‘The bitter lesson’) rather than something more elegant isn't only a reflection of what intelligence requires. It's also a reflection of what hardware was commercially available.

Neural networks probably aren’t the only (or the best) way to get to AGI. They are certainly not the most efficient. Indeed, as co-founder of OpenAI and one of the researchers behind AlexNet (Ilya Sutskever) has argued, we may be in a local minimum where the obsession with the transformer is now blocking progress. The transformer is now infrastructural. The entire ecosystem (CUDA libraries, training frameworks, MLOps tooling, benchmark design, hiring pipelines, VC funding models) is now optimised for transformer-on-GPU. This creates enormous lock-in that goes far beyond technical considerations. It's a sociotechnical lock-in analogous to fossil fuel infrastructure.⁷

Perhaps this should be consolation for nations who think they have already lost the AI race to China and the US. There are many possible twists and turns to the AI race if new hardware (which is more efficient or more apt for neural networks) becomes developed, which would change the logic of supply chains, AI funding and applications.

The first GPU (Nvidia GeForce 256) - 1999

Shale-Shaped AI

David Edgerton has powerfully demonstrated that technologies tend to be symbiotic. The tired potted histories of the ‘Age of the steam engine’ and ‘the age of electricity’ etc. obscure the constellations of technology (and social forces) which define technical change.

Indeed, the simultaneous technological breakthrough of horizontal drilling (which allowed a single well to travel thousands of metres sideways through thin shale rock), modern hydraulic fracking (specifically slickwater fracking developed in the 1990s which meant cheap water-based fluid could be used and therefore the technique was economically viable) and precision mapping (which allowed engineers to visualise the best seams to target and therefore get more resource gain) set the stage for a more power-hungry version of AI. These changes took place in the US and ushered in the age of fracked shale gas.

The timing was important. US shale production began its exponential rise around 2008-2012 (just when the deep learning revolution was starting). By the time large-scale model training became the norm (~2015 onwards), the US had transformed from a natural gas importer to the world's largest producer. US electricity prices, particularly in regions with gas-fired generation, became globally competitive precisely when AI labs needed cheap, abundant power. Gas prices reduced by around 57% during this period.⁸

QE-Shaped AI

Fracking was initially thought to be economically unviable. Adam Tooze has argued that the shale revolution would never have happened if it wasn’t for Quantiative Easing (a feat of financial engineering in response to the 2008 financial crash). Many shale companies burned through cash for years, sustained by junk bonds and leveraged loans in a low-interest-rate environment. QE kept rates low, which kept capital flowing to unprofitable shale drillers, which kept gas cheap, which kept electricity cheap, which made brute-force GPU training economically viable.

In this sense, and others, we still live in the economic shadow of the 2008 financial crisis. Adam Tooze has detailed how the unprecedented emergency efforts of the US Federal Reserve during the financial crisis created enormous distortions in the global economy which still exist. The introduction of Quantitative Easing (dramatic asset purchases) and liquidity swap lines created a ‘financial twighlight zone’. Central banks became guarantors of financial stability. It also hit growth. As central banks ‘flooded the zone’ with money, governments were tighting spending. This created an unbalanced recovery (asset prices spiked but growth rates were hit).

Directly, Quantitative easing compressed yields from safer assets and therefore pushed capital into riskier assets which allowed for funding AI labs who were nowhere near profitability. As Tooze argues ‘no one benefited more in American business from the era of QE and zero interest rates than the Fossil Block’. While critics often dismissed fracking as a bubble or a ‘capitalist delusion’ because it struggled to make a profit, the massive influx of capital allowed these companies to create "multiple equilibria". Essentially, the cheap money kept them afloat long enough to fundamentally change the American energy landscape. Counter-intuitively, according to Tooze, fracking turned the US into an electro-state. While oil is traded globally, the gas produced by fracking stays mostly within the U.S. (due to pipeline constraints), which "fundamentally changes the logic and the price and the cost structure of electricity generation". This energy abundance filtered into the logic of the AI data centre buildout. As Tooze argues, the availability of cheap, gas-fired power (and increasingly solar) is what sustains the massive power needs of these ‘capital-light’ companies as they pivot to AI

More speculatively, QE may have shaped the kind of AI we got. Our current AI has just become much more capital heavy. A world of cheap capital favours capital-intensive approaches (bigger models, more GPUs, more data centres). It disfavours patient, efficiency-focused research. The "scaling laws" paradigm essentially formalised the observation that throwing more compute at transformers reliably improves performance. But this observation is only interesting if compute is cheap enough to throw. In a higher-rate environment, the pressure to find more compute-efficient architectures might have been far greater. We might have seen more investment in neuromorphic hardware, sparse models, or fundamentally different approaches. The brute-force scaling paradigm is in part a product of cheap money.

The financial architecture of the AI boom is itself borrowed from fossil fuels. Data centres are increasingly financed through special purpose vehicles and project finance structures. These techniques were originally developed for oil and gas exploration in 1930s Texas and scaled up for LNG terminals and pipeline infrastructure over subsequent decades. The mechanism is the same: create a legally separate entity, lend against future contracted cash flows (cloud computing revenues rather than hydrocarbon sales), and isolate risk from the parent company’s balance sheet. Meta recently structured nearly $30 billion through an SPV for a single Louisiana data centre, the largest such transaction in history. The same banks, the same structures, and increasingly the same asset managers (Blackstone, Brookfield) that financed fossil fuel infrastructure are now financing AI infrastructure.

Additionally, the 2008 crash and the resulting bailout created a ‘too-big-to-fail’ environment. The Fed's balance sheet expansion didn't just lower rates it restructured the financial system's relationship to risk. The implicit backstop to asset prices created a ‘heads I win, tails I get bailed out’ dynamic that favoured exactly the kind of speculative, long-horizon, capital-intensive bets that characterise AI investment (though OpenAI claim they do not expect a government bailout after their CFO did describe the desire for a government guarantee).

The AI Assemblage

It’s tempting to think we’re entering ‘the age of AI’ but really are we not entering an age where symbiotic technologies (energy, financial, hardware and algorithmic) feed on eachother? Would it be more accurate to say we’re in the GPU era or the gas era or the data centre era?

As a Shoggoth meme has it, AI is a sprawling monster. But, as one of the world’s most consequential AI safety researchers (and co-founder of Anthropic) Chris Olah has put it ‘We don’t program neural networks, we grow them... it’s like studying biological organisms and very different from regular software engineering.’ We provide the ‘nutrients’ (data) and the model grows based on its own internal logic. To push this metaphor further, the space in which our current version of dominant AI (transformer-based LLMs) is growing is backed by the GPU. GPUs grew up in an environment of cheap fracked gas and cheap money from quantitative easing. AI is shaped by these forces root and branch. Understanding the historical forces which have shaped AI also allows us to understand that the future of AI is also contingent, and malleable.

1. It is admittedly hard to compare the brain and an AI, given that they are quite different objects. This estimate derives from comparing energy efficiency (bits processed per joule) across two very different systems, and should be treated as an order-of-magnitude claim with significant uncertainty, probably accurate to within a factor of 100 either way. The brain consumes approximately 20 watts continuously, a figure well-established in neuroscience.

Though this estimate is imprecise it clearly points to the fact that biological neural computation operates in a radically different (and far more efficient) energy regime than current silicon-based AI, a point with significant implications for the long-term sustainability and scaling trajectory of AI systems.

For the brain: total sensory bandwidth is roughly 10⁷–10⁸ bits per second (the optic nerve alone carries ~10⁷), though meaningful integrated processing is likely lower, perhaps 10⁴–10⁶ bits per second once redundancy is accounted for. At 20 watts, this yields roughly 10³–10⁵ bits per joule.

For frontier AI (circa 2026): a single model instance serving one user draws roughly 1,000–50,000 watts, generating 30–100 tokens per second at ~10–12 bits of Shannon entropy each, roughly 0.01–1 bit per joule.

The ratio spans 10³ to 10⁸, with a central estimate around 10⁵–10⁶: hence we can say ‘roughly a million times.’ Reasonable choices in terms of methodology etc. could shift this by two orders of magnitude in either direction.

2. This was especially true after the development of Autograd. This allowed the rapid calculation of gradients within the backpropagation algorithm (the algorithm for working out in which direction to update the neural network weights). Since autograd sped up with process so much it meant researchers could try ad hoc hacks without necessarily needing the theoretical justification.

3. Hooker notes that it wasn’t until the development of Matlab in 1992 and then LUSH and TORCH that deep learning researchers had a solid software environment to support their work.

4. Just as Einstein was uncomfortable with quantum mechanics because he thought physics should describe an underlying deterministic reality, not just provide a probablistic prediction (‘God does not play dice’) so too one of the pioneers of AI (McCarthy) critiqued the AI model which beat Garry Kasparov at Chess (Deep Blue). He thought this was almost a distraction because it told us nothing about the mind. In the early decades of AI research there were fraught debates over the pathway of AI. AI’s founding generation was divided between what Roger Schank called ‘neats’ (McCarthy, Kowalski, Newell, Simon — who wanted logically precise, elegant systems) and ‘scruffies’ (Minsky, Papert, Schank, who accepted messier, more pragmatic approaches). Both camps, however, might have been horrified by modern brute-force scaling. The neats wanted mathematical beauty; the scruffies wanted cognitive plausibility. Neither wanted ‘just make it bigger.’

5. This is not to say they’ve ignored it (e.g. see Cerebras and OpenAI partnership) but they haven’t prioritised it.

6. Please note that MAMBA does have a problem - it systematically fails on copy and recall tasks. In other words, crucial details might be lost.

7. As Yann Lecun has argued, ML is more an engineering discipline than scientific.

8. According to Epoch AI’s peer-reviewed cost breakdown (Cottier et al. 2024), energy accounts for about 2–6% of frontier model training costs. Hardware depreciation dominates with 47–67%, and R&D staff at 29–49%. This means the direct energy cost impact of higher gas prices on any single training run is modest: roughly 8–15% total cost uplift when including facility and cooling effects, not the 35–40% previously modelled. But the real mechanism is systemic, not per-run. Cheap energy enabled cheap data centre construction at scale, which kept cloud compute rental prices low, which kept the cost of experimentation down. OpenAI spent $5B on R&D compute in 2024 (mostly on experimental runs, not final training). The culture of just scale it up and see depends on infrastructure that cheap energy built. And at the frontier, energy’s share is growing fast: Grok 4 was trained on the largest supercomputer cluster in the world, the Colussus data centre, and consumed 310 GWh and cost ~$490M, trained on natural gas turbines. Colussus includes 200,000 GPUs and there are plans to scale it to 1m. Colossus 2 is an even larger data centre (the first gigawatt data centre in the world) which plans to be the first data centre to bring online the Nvidia Blackwell chips (inside GB200 and GB300 chips).