Sean Ó hÉigeartaigh is Program Director of AI Futures and Responsibility at the Centre for the Future of Intelligence, University of Cambridge. I spoke to him about his article The Most Dangerous Fiction: The Rhetoric and Reality of the AI Race
This is a really important question and Sean is a leading thinker and practitioner in this field. Do you believe China is racing for AGI? Why or why not? (Comment below!)
Find it on Youtube here:
Philip Bell: Sean is the Program Director of AI Futures and Responsibility at the Centre for the Study of Existential Risk at the University of Cambridge. It’s an absolute pleasure to be speaking to you today.
Sean Ó hÉigeartaigh: Thanks so much. It’s a pleasure to be speaking to you.
Philip Bell: The starting point for this conversation was your article called “The Rhetoric and Reality of the AI Race,” which I found fascinating. Could you explain the core argument?
Sean Ó hÉigeartaigh: The argument was basically this: I’ve been going to China for about a decade now, watching things evolve there as I’ve been watching them evolve here. It’s been really notable to me how much divergence there has been between the narrative presented around AI competition—the AI race—when I visited China, versus how it’s being spoken about in the West, particularly in the US.
I’d written about this a couple of years ago with some colleagues when the AI race narrative was just starting to come to the fore, but it’s only become way more intense now—this framing that the US and China are locked in a competitive geopolitical race for AI dominance. The more I saw this, the more it felt like it didn’t gel with my experience, and the more I wanted to dig into it.
It comes down to two things in particular. First: what do we mean by an AI race? One version is general competition around a technology. You could argue that yes, there’s obviously competition between companies in the West and companies in China, and there’s obviously both economic and geopolitical competition between the US and China. But that’s what you might consider normal competition. They compete over AI and energy and electric vehicles and any number of things. A race in that context isn’t one that can be definitively won because you can be in the lead in a particular aspect at a given time, but then be overtaken or be behind in another area.
But what I found particularly interesting is that there’s another conception of a race—a race to artificial general intelligence, AI that can do everything that human intelligence can. This is perceived by some technologists, including some of the big companies in the West, as representing a step change in technology where whoever gets there first ends up with a decisive advantage, not only over AI, but they can then use that AI to accelerate everything else. The idea is that once you get to a particular stage of development, your own AI helps you accelerate your own development of AI because AI takes over more and more of the research process. So instead of a race with no particular endpoint where you can be overtaken at any point, maybe you reach a particular threshold and have a lead that can’t really be made up. People like Dario Amodei at Anthropic have alluded to this, as has Leopold Aschenbrenner in his “Situational Awareness” paper.
So the paper does two things. It tries to differentiate these two types of races. It asks: is there generally a race going on? I think the answer is, to a certain extent, yes—but the narrative around it has definitely been exaggerated and overblown, and it seems to serve certain interests. It’s used to argue in the US for things that clearly benefit companies, like lighter regulation or energy system rollout.
But then there’s a second specific claim to do with the race to AGI and geopolitical strategic advantage. The paper looks into whether that’s a race meaningfully ongoing where the US and China are competitors. I argue that based on the evidence I’ve been able to find, that’s not really a sensible way of characterizing what’s going on. There’s definitely some degree of race going on between leading companies, most of them in the US, but there doesn’t seem to be much evidence that there’s a top-down national push to achieve AGI before the US happening in China. If anything, the way AI is framed and prioritized there is much more around benefiting the economy in a wide range of different tool-AI type ways.
Philip Bell: That’s really interesting. So there’s no evidence that China is racing—or are you saying there’s no evidence that both the US and China are racing?
Sean Ó hÉigeartaigh: The evidence that China is racing in this particular way is pretty weak. There has been some regional support for institutes like the Beijing Institute for General Artificial Intelligence, but that’s one example, and they’re not really developing frontier models yet—they’re taking a very different approach. There haven’t been very big funding initiatives that seem to be focused on AGI. There was at one point a push to develop LLMs because ChatGPT was such a phenomenon, and certainly China has caught up and is close to the frontier on LLMs and reasoning models. But there doesn’t seem to have been this push for anything like a Chinese Manhattan Project.
This is quite important because the claim has been made by various people in the US context that the US should do this as a national priority, and it’s been argued that the US should do this because China is doing it. Reading those kinds of claims is what motivated me to look into whether that’s actually a sensible way of looking at it.
A second question is whether the US is doing this. I think the US has been allowing the market to do its thing, and there have definitely been companies pushing towards this. The previous administration seemed to take the idea of AGI very seriously. With the current Trump administration, the signals are much more mixed—most recent statements by people like Sriram Krishnan and David Sacks cast a lot of doubt around the idea that they think near-term AGI is plausible. They seem to be thinking about AI much more in terms of what Arvind Narayanan and Sayash Kapoor would call a “normal technology”—something where you win the competition by diffusing it out and having applications across a wide range of industries. A little more like the way the Chinese are thinking about it, less focused on artificial general intelligence and superintelligence.
Philip Bell: I read an article by Graceshaw on Exponential View called “China’s Different AI Path,” which talks about China approaching AI from the perspective of faster open-source models rather than heavier, larger closed-source models. Another data point is China’s recent AI Plus strategy. Is that one of the things you’re looking at to argue that China doesn’t appear to be racing from the top down towards AGI?
Sean Ó hÉigeartaigh: The recent AI Plus release certainly supports that argument. It’s not so much a new thing—it’s an update of an ongoing process—but it is an evolving policy that I’ve looked into with a colleague, Christy Luck. We released an article on it in The Wire China a couple of months ago. She’d gone through Chinese AI policy going back years, and there’s been this evolving strategy very much focused on developing AI to benefit the economy in a lot of different ways—much less this idea of strong AI and superintelligence, and in fact a lot of skepticism from some of the people who advise government on whether this is a near-term thing.
You can also see it reflected in speeches President Xi has given, where the language hasn’t changed much over the last couple of years and seems much more tied to ideas of machine learning applied in innovative ways, less focused on a unified goal of artificial general intelligence. Various analysts have noted it’s interesting that it hasn’t changed over time, where the narrative in the US has shifted—and maybe it’s shifting back a little bit at the moment.
Philip Bell: One reason this narrative may have developed is the use of historical analogies. In the AI Plus plan, the Chinese government explicitly talks about electricity and the internet as comparators. Meanwhile, in your article you mention Alex Karp, CEO of Palantir, talking about the “Oppenheimer moment,” and other US organizations talk about a potential Manhattan Project—implicitly comparing it to the atom bomb. Do you think historical analogies played a harmful part in this narrative developing? Would thinking through which analogies are most useful be a way of addressing it?
Sean Ó hÉigeartaigh: This is a really interesting question because it’s hard to know the directionality. You could argue that if you think of AI as being like electricity or the internet, then you think: now you have electricity or the internet, let’s make it as easy as possible for people to innovate and figure out lots of smart, unexpected ways to use it, set up cool new companies, and so on.
If you think about AI as a Manhattan Project, you think of a big scary thing that probably needs to be under government control, where you develop it once and that changes your position on the landscape. That frames your thinking very differently.
Another way of approaching it: maybe you have fundamental scientific beliefs around the technology. You look at AI and see advancing machine learning with lots of different applications that need to be developed for specific contexts, requiring domain knowledge. Thus the analogy that comes to mind is the internet or electricity. Maybe you think of artificial general intelligence and think: well, if this can really do what Demis Hassabis says—if it’s able to automate the kind of scientific discovery that Einstein did when he discovered general relativity—then that’s very clearly not the internet. It’s something that fundamentally changes the scientific and maybe even civilizational landscape.
I have some sympathy with the latter framing, but I think it makes sense to maintain some degree of skepticism and not allow these analogies to constrain our thinking too much.
Philip Bell: Historical analogies are used a lot when talking about AI. Sundar Pichai talked about the invention of fire when launching Gemini, and Demis Hassabis talks about the industrial revolution. It’s not clear at this stage what the right historical category is—it’s unfolding all the time, and maybe there isn’t a single category that’s most useful.
Going back to what you said about the narrative from major labs and governments not chiming with your experience of China—do you think part of the problem is a more general limited understanding of China?
Sean Ó hÉigeartaigh: I do think it’s part of the problem, and it’s a problem that has become worse in recent years, largely because we’ve had more decoupling. If you went back to 2019, you had Google and IBM and others having technology headquarters in places like Beijing as well as in the West. But over the last six years there’s been a big retreat. So there’s just much less interaction and exchange of ideas, which makes it easier to maintain misconceptions.
I think there’s also a tendency to see China as a monolith—the idea that because of what kind of government it is, it just makes a decision at the top and then the whole thing moves like the Borg. Whereas my experience with China is lots of people, the usual jockeying for position, one department wants ownership of this because it gets prestige, but some other group would like to own it too—interdepartmental bickering over who’s responsible for this bit of AI. Exactly the kind of thing you get in the West.
I will caveat this: I go to China quite a number of times, but I’ll only ever see one part of it. I meet with academics who are internationally minded and pro-cooperation, and we have extremely constructive conversations. But I’m sure there are huge parts of the Chinese ecosystem I just don’t see. I’ve had no interaction with their defense sector, for example. The more hawkish elements would have no reason to interact with me. So I have to acknowledge I may have some biases.
Philip Bell: Are there clear ways shared understanding could be improved? Would that be useful for diffusing these race dynamics?
Sean Ó hÉigeartaigh: I think it would be really useful. The more tensions ratchet up between the US and China—and even between the West more broadly—the more important it becomes that there be some dialogue and development of common understanding.
Another historical analogy: if we go back to the Cold War, even in the tensest moments you had academic exchanges, scientists from across these divides sharing perspectives and working together. That played a really important role in maintaining shared understanding and communication. I’m not saying that’s the situation we have now, but there are certainly elements of tension ratcheting up, and it’s surprising and a little worrying to me that there’s so little of this happening.
There are quite a few US think tanks who write about China, but comparatively few who engage with China. It’s becoming more difficult to do so—it’s harder to travel, and more entities that would be valuable to engage with are being put on the US entities list, which makes it very complicated for US participants to engage in, say, track-two dialogues. When I take part in track-two dialogues, you see the same faces quite a lot. It’s a relatively small community. Most of us aren’t doing this as our full-time job—we’re governance researchers or AI researchers who happen to do some of this because we think it’s important. We could really use a whole lot more of it.
Philip Bell: Could you say more about the track-two dialogues and which organizations are working on this?
Sean Ó hÉigeartaigh: There are several good ones. Carnegie Endowment has been doing very good work. The Oxford China Policy Lab has been doing good work. Concordia definitely have been very helpful—they’ve been doing this for a decade. They’ve produced really good reports for Western audiences on what’s happening in AI safety in China, and work on what’s happening in the West for Chinese audiences. They’ve worked with leading labs like the Shanghai AI Lab on their frontier AI safety frameworks. They’ve organized workshops or sessions at major conferences like the World AI Conference, bringing together both Western and Chinese experts.
Caleb Lundquist and Scott Singer and their team at Carnegie Endowment have been involved in some recent events—I really like the pieces they’ve been putting out explaining the role that the Chinese AI Safety Network is playing and other developments in AI safety for a Western audience. The Safe AI Forum has been running the IDAIS dialogues for a couple of years now, bringing together leading scientists from China like Andrew Yao, from the West like Yoshua Bengio and Stuart Russell, as well as governance thinkers like Jin Zhilan of the Schwarzman College. They’ve done a really good job developing consensus on things like red lines on risks in AI—my understanding is that’s been quite influential both internationally and in the Chinese context, and has shown there’s a lot of room for common ground.
For my own part, the Royal Society in the UK has done a really good job organizing joint workshops, not just on AI but on other scientific topics. I’ve taken part in joint workshops with the Chinese Academy of Sciences. I’ve been involved with Yi Zeng, who’s a professor at the Chinese Academy of Sciences and now head of the Beijing AI Safety Institute. We’ve co-chaired an international governance program at the World Internet Conference, and we’ve just set up an international cooperation research network under the auspices of the ANSO network.
There are definitely things happening, and I’ve been very grateful for the support we’ve gotten from the international community. Everyone’s so busy these days—it’s not a small thing to ask people to fly to Beijing and deal with the jet lag. But I do think it’s really important.
Philip Bell: Something I thought was true because I’d read it quite a few times—but I now wonder whether it is true—is whether attitudes towards AI appear to be different in the US and China among the general population. Stanford’s AI Index appears to show that attitudes towards AI in China are a lot more positive than in the US, but I’ve heard this critiqued. Jordan Schneider of the ChinaTalk podcast was critiquing the methodology, suggesting the respondents might be people in large cities who don’t reflect the full population. Does overall attitude of Chinese and US populations have an impact on the race dynamic?
Sean Ó hÉigeartaigh: It’s an interesting question. Jordan Schneider is brilliant, by the way—I highly rate his ChinaTalk website. I haven’t looked deeply into the Stanford AI Index methodology, but intuitively it feels like I wouldn’t be surprised if it were true.
Part of it is that the scientific technological miracle is such a big part of the Chinese cultural narrative—it’s really invested in science and technology, and that’s allowed it to boost quality of life and its place in the world. That feels like a very powerful part of the cultural narrative, much more so than in Europe, where there’s much more skepticism around whether all these technological developments are actually in our interest, different attitudes towards individual privacy, and a more precautionary approach.
In the US, it certainly seems like it’s not as overwhelmingly positive as in China—there are concerns around environmental impact, water use, what this is going to do to jobs and livelihoods, and more skepticism around whether the tech sector is definitely acting in the interests of the average person.
Whether that plays into a race narrative, I couldn’t say for certain, but it’s not entirely implausible. If a uniformly positive narrative around the benefits of AI was achieving less purchase with policymakers and a broader audience, then you would look for other messages. A competitive narrative that there’s a threat to US preeminence if the US allows itself to fall behind China could be a very compelling message.
I’ve got a paper under review at the moment where I’ve looked at the role the tech sector has played in promoting the race narrative—it’s played a really prominent role. Tech investors and people in leading positions at tech companies have been pushing this right from 2017 and the release of the Chinese plan, with messaging that China is poised to overtake the US and then it’ll be really bad in all sorts of ways. And it’s never too far away from recommendations that would boost US AI progress but also benefit the companies and investors—whether it’s increased federal investment into fundamental AI research, lighter regulation, laxer environmental laws, or access to copyright data.
Philip Bell: I have a more abstract question. You’ve worked on a number of different areas in existential risk. I’ve recently been reading a book by Jonathan White, a professor at LSE, called “In the Long Run.” He argues that we struggle to face the future because we have both pressures on our short-term attention from social media and the speeding up of the news cycle, but also a feeling that we’re at some decisive point in history where we need to take action on climate change or AI. He describes this creating a sense of emergency—he talks specifically about politics and the recent US election being described by both sides as the last chance to decisively change the country. Do you think there’s a more general phenomenon where a sense of emergency is being generated or emerging?
Sean Ó hÉigeartaigh: There’s a concept in the governance literature very related to this called securitization. If you can turn an issue from a normal policy issue into an existential one—a giant threat to your nation’s sovereignty or survival—then it can be very powerful because it allows you to justify a great many things you might not under normal conditions.
I worry there are some elements of this happening with the race. A lot of the ways AI has been framed, including by the tech sector, has taken this from a normal policy issue to one where the future of the United States is at stake. Therefore all the other considerations that would normally come into play—whether it’s thinking about how you scale up energy while being environmentally friendly, or how you do democratic oversight around development—need to be set aside because there’s this fundamental threat.
I think that’s both relevant and quite a powerful concept. There’s clear evidence this is being employed. Some lobbying bodies like American Edge very clearly lean into this—they talk a bit about the positives of AI, but a lot of the messaging is around this looming terrifying threat of China getting ahead. And if you’re a member of the public or a policymaker hearing this from many different sources, it can feel like an organic independent thing. But American Edge doesn’t reveal who their funders are, though somebody traced back that they got a very large amount of funding from Meta in 2021. So you’re getting Newsweek articles and Fox News interviews and evidence submissions—lots of different sources telling you there’s a giant looming threat, and that you need to put aside the normal niceties like people’s copyright protection or environmental laws because in an emergency you just do what needs to be done.
Philip Bell: What do you think is the most likely scenario if we continue on our current path, and what would be a best-case scenario for you?
Sean Ó hÉigeartaigh: That’s a very hard question. One way to answer it is to ask what happens in the science. The way I see it, the future branches in two broad directions.
One is that it becomes more clear that AI is what we might consider a normal technology—very transformative, but more along the lines of the internet or electricity. In that case, exporting technology and getting countries to adopt it and competing in individual domains rather than competing over the best overall model makes more sense. In some ways that feels less worrying because it just looks like the world we have now, except with better technology, and the kinds of worries we have now, just a bit accelerated or amplified.
Then there’s another world where the idea of AGI really is looming and imminent, where it becomes clear we’re going to develop something that will upend civilization in coming years, and whoever develops it first really does have a huge geopolitical impact. That’s a much more worrying situation because I think we need to be very careful in how we develop such a technology. The worst possible situation is one where an actual race emerges between the US and China, or the US and any other country, or even two companies in the US. If companies are racing towards this instead of going slowly and taking care about fundamental alignment issues, that could be very concerning.
It seems realistic that it’s going to become clearer over the next couple of years which of these worlds we’re in. It’s going to be harder and harder to justify putting absolutely huge investments into AI research and data center buildout with development models that are never quite in profit, unless it’s very clear that something very big is coming down the line. So it feels like this has to bifurcate at some point.
On the politics and governance side, I think more about the world in which AGI is plausible because if we get it wrong there, it’s most worrying. Within that world, there’s a future in which we achieve the level of dialogue between the US and China where we recognize that competition is fine in a whole load of areas, but on some key issues we do need cooperation. For example, red lines on safety, being really careful about initiating recursive self-improvement—systems that improve upon themselves in an accelerated fashion. My hope is we get to the point where we have those red lines in place while allowing competition in other domains, and some sort of mechanism where the rest of the world gets to say as well. I’d ideally like a future in which AI wasn’t just done to us by the leading two nations, but where there was some sort of global democratic conversation around it.
The other world in the AGI scenario is one where development is very much within a geopolitical race, where leading nations both feel under huge threat from the other if the other gets there first. That feels like a roll of the dice—hopefully it goes well both technologically and on the governance side, but it’s a very high-risk scenario. I would love for us to avoid that if at all possible.
Philip Bell: Is there anything people interested in this area could do to try to make it go well?
Sean Ó hÉigeartaigh: It depends on what area people are in. First: don’t always take the more extreme race narratives at face value. There have been claims that the US has to initiate a Manhattan Project because China’s racing to AGI, made by senior government advisors—and I think they turn out to not really be true. So do question them. Recognize that there are vested interests involved. I’m not claiming everybody’s lying, but people can end up convincing themselves of an idea or talking to the same echo chamber and getting a little disconnected from what’s actually happening. Take these things with a grain of salt.
If you’re an academic working in AI or AI governance or more broadly, if you have opportunities to do collaborative engagement across these divides, it can be of a lot of value. The more links and opportunities we have to talk to the human beings on the other side and recognize we probably care about the same things, the better. There will be competitive issues and points of tension and conflict, but that doesn’t mean we can’t make things better by finding points of common ground.
There’s a lot of interesting work happening around verification—not just forming agreements, but being able to monitor and verify that different governments or labs are sticking to agreements. There was a very long, detailed, and excellent report from the Oxford Martin AI Governance Initiative last summer—I believe Ben Harrick and Robert Trager were authors. That would be the go-to I’d recommend.
There’s also a proposal called FlexHex—hardware-level governance that David Dalrymple and Noa Mann and others have been working on, hosted on Yoshua Bengio’s website. It looks at how you embed agreements in hardware so chips only work if you’re doing what you said you would. That could mean making sure chips are being used for inference but not training, or putting in geolocation so you know where chips are going. There’s a lot you could do so that when we come up with agreements, we have ways of assuring everyone that everybody is standing by them.
One thing we’re a bit lucky about with AI—though perhaps this is changing—is that for very leading models, they’ve been quite big endeavors requiring a lot of money, hardware, and energy. So it’s been at least possible to track them rather than have a secret project. Some recent models have had good performance with more limited requirements, but it’s at least not the kind of endeavor where three people in a basement can develop superintelligence. At the time being, there are some advantages.
Philip Bell: Could you say more about the work on verifying that labs are sticking to what they’ve said?
Sean Ó hÉigeartaigh: The Oxford Martin AI Governance Initiative report from last summer would be the go-to—it’s about 80 pages and really comprehensive. The FlexHex proposal is really looking at hardware-level governance, examining how you embed agreements in the hardware so the chips only really work if you’re doing what you said you were going to do. You could ensure chips are being used for inference but not training, put in geolocations so you know where chips are going. My hope is that we can develop a toolkit that means when we come up with agreements, we have ways of assuring everyone that everybody is standing by them.
Philip Bell: Thank you so much, Sean. This was absolutely brilliant. I really enjoyed it.
