This is a conversation with AI Safety Researcher Lee Sharkey about what we do and don’t know about how Large Language Models (LLMs) actually work. We explore mechanistic interpretability, the science of trying to understand what’s going on in LLM models, and why understanding the limits of our knowledge is more important now than ever.
Lee co-founded AI safety lab Apollo Research, which red-teams LLMs and works at Goodfire AI. Transcript and links below. Enjoy!
Check out the podcast on spotify.
Here are some articles Lee has written on llms:
Open Problems in Mechanistic Interpretability
Sparse Autoencoders Find Highly Interpretable Features in Language Models
Summarised Transcript:
Lee Sharkey (00:00)
One of my favorite lines from Geoffrey Hinton — in a video lecture he said:
"We’re going to imagine a 14-dimensional space. We’ll do this like everyone else does: imagine a 3-dimensional space and say 14 very loudly."
It’s a way of saying our brains aren’t good at imagining these spaces, but you can still get weak intuitions. I’m not claiming I understand 14D space like 3D, but you can get little glimmers of what it’s like.
Philip Bell (01:37)
Given [mechanistic interpretability] is basically a brand-new field, why did you decide to work in it?
Lee Sharkey (01:46)
It’s new in one sense, but also a continuation of other fields. My background is in computational neuroscience, studying artificial recurrent neural networks as models for biological networks.
The tools, concepts, and methods applied directly to AI safety. I wanted to work in AI safety, and interpretability was the intersection between what I knew and where I wanted to go. Biological and artificial networks are both systems of large numbers of nonlinear computations. They are fundamentally similar.
Philip Bell (04:03)
Was there a light bulb moment when you decided to focus on AI?
Lee Sharkey (04:32)
Yes. In 2014, during my neuroscience undergrad, I read a paper by Quoc Le and Andrew Ng. They trained an encoder–decoder network with a bottleneck layer in the middle to reconstruct images.
When you activated a single neuron in that bottleneck layer, it could generate an image of a cat — without ever being told what a cat was.
At the time, this was striking. It had abstracted concepts from random images. I remember thinking, deep learning is going somewhere.
It wasn’t until 2016–2018 that I got into it deeply. 2014 was around when DeepMind sold to Google, so things were picking up but still early.
Philip Bell (08:22)
In mechanistic interpretability, you spend most of your time looking deeply into large language models. How do you do that?
Lee Sharkey (09:15)
Neural networks have layers — matrices of numbers — that transform input lists of numbers into output lists. That’s the linear computation, followed by a nonlinear one (e.g., a threshold function: above threshold, the neuron is active; below, it’s zero).
They’re sequences of these computations. When trained, they learn algorithms inside — I call them neural algorithms, distinct from the training algorithm.
These algorithms produce representations — a cat, a cat’s ear, or lower-level features like edges. The network computes over these to produce the trained output.
My work is figuring out the fundamental computations: breaking the learned algorithm into simple steps so we can understand them. We don’t get that understanding for free.
Philip Bell (13:09)
Is it more work to understand models than to build them?
Lee Sharkey (13:18)
Yes. We know how to train them, but not how to understand them well.
Philip Bell (13:25)
With large language models, could there be a neuron that fires for a specific concept, like Barack Obama?
Lee Sharkey (14:09)
Yes. People have found such things — maybe not a single neuron, but an activation pattern. In multimodal models, there can be a pattern that responds to both images and text about, for example, Barack Obama.
Philip Bell (15:14)
Do you test many inputs and see how activations differ?
Lee Sharkey (15:33)
Yes — but you need to decide what you’re looking at: individual neurons or activation patterns.
A feature is a pattern in activation space — a vector, a direction. Neurons are fixed in the design, but possible activation patterns are infinite, and random ones may mean nothing.
We use methods like sparse dictionary learning to find patterns that recur but activate only rarely. Concepts like Barack Obama are sparse in the dataset. Training to find sparse activations helps reveal fundamental representations.
Philip Bell (18:57)
Are these methods similar to neuroscience experiments?
Lee Sharkey (19:27)
Very. Computational neuroscience inspired some of these methods. Sparse coding, for example, was used in neuroscience long before neural networks.
Philip Bell (20:48)
What are examples of interesting findings?
Lee Sharkey (21:23)
The cat example is easy to replicate. Anthropic found a Golden Gate Bridge feature — a pattern that responded to inputs about the bridge, images of it, and related concepts like San Francisco.
Almost anything you can think of might be represented. The question is how to find it.
I’m more interested in the computations that generate these patterns. For example, “induction heads” — if a model has seen A, B in the past, then sees A again, it’s more likely to predict B. This generalises to abstract concepts.
Philip Bell (24:30)
Most people see only the input and output. You’ve spent a long time looking at the inside. Has that changed how you think?
Lee Sharkey (26:01)
Neural networks are huge. You can ask a model questions across many fields, and it answers sensibly. How does it know so much?
We can think of activations as points in high-dimensional spaces — often thousands or millions of dimensions. There’s a lot of room in these spaces.
I can’t imagine a million-dimensional space, but you can develop some intuition for higher dimensions — like in Hinton’s joke about saying “14” very loudly.
Philip Bell (30:13)
Does this change how you think about your own thinking?
Lee Sharkey (30:46)
Yes. I sometimes think about my own thought patterns in similar terms to AI computations. It’s a useful way to reflect.
Philip Bell (32:21)
Are neural networks beautiful or messy?
Lee Sharkey (32:21)
Both. Like biology, they’re grown, not designed. There are elegant structures alongside messy inefficiencies — like the giraffe’s recurrent laryngeal nerve, which loops unnecessarily from the head to the chest and back.
Philip Bell (34:56)
Do humans and LLMs structure knowledge similarly?
Lee Sharkey (35:28)
Not perfectly, but yes — both are modelling the same world. In vision, model representations can resemble those in primate brains. There’s shared structure because both are learning useful representations of the same environment.
Philip Bell (39:45)
On a scale of 0–100, how much do we understand large language models?
Lee Sharkey (40:17)
If forced to give a number — 1%. We can identify concepts and patterns, but we don’t understand the step-by-step processes that produce them.
Philip Bell (41:37)
How far can mechanistic interpretability go in ensuring AI safety?
Lee Sharkey (42:24)
Current methods aren’t enough for superhuman systems. But with better methods, applied at scale, it could help verify safety — not just assume it based on correlations.
Philip Bell (44:57)
Should mechanistic interpretability stay in big labs or move elsewhere?
Lee Sharkey (45:39)
It’s already moving — governments, smaller companies, academics, and independents are increasingly involved. But it’s still small. More people, especially neuroscientists, could contribute.
Philip Bell (48:41)
Should AI users demand to know what’s going on inside models?
Lee Sharkey (49:04)
I think so. Companies often answer “we don’t know,” which is partly true — the science isn’t fully there. But they could do more with current tools to provide insights.
Philip Bell (50:04)
What’s your ideal outcome for AI in the next 10 years, and what’s most likely?
Lee Sharkey (50:31)
The ideal is AI helping solve major material and non-material problems — disease, human fulfilment, societal well-being.
The likely outcome is a mix: some areas will go well, others poorly. Even if technical safety improves, societal coordination will be a huge challenge.