I spoke to Carl Benedikt Frey, Professor at the Oxford Internet Institute and one of the most cited scholars on the political economy of technology. His new book How Progress Ends argues that technological progress is far from inevitable. In this conversation, we discuss the delicate choreography between exploration and implementation, why different institutions suit different phases of the technology lifecycle, why Europe caught up in mass production but has failed in digital, and what this means for AI.
Philip Bell: To start off, could you outline the key arguments in your book, How Progress Ends?
Carl Benedikt Frey: The purpose of the book is to push back against the narrative that technological progress is inevitable. If progress was inevitable, it wouldn’t have taken humanity 200,000 years to have an industrial revolution. If progress was inevitable, most places around the world would be rich and prosperous today, and we wouldn’t see places that once prospered stagnating or declining or collapsing.
I think the reason that progress isn’t inevitable is that as technology moves on, institutions need to adjust as well. Different institutional settings are more conducive to different stages of the technology lifecycle. Early on, when you’re exploring, you don’t know if something is going to catch on or not, so you’re better off having a system where people take different bets and then you see what works out.
The Soviet Union was the most centralised economy the world had ever seen. If you were an aircraft engineer in the Soviet Union, you could develop a new engine and go to the Red Army to ask for funding. If they declined, maybe you had two or three other options. If they all declined, your idea would die with you. That’s quite different from the US system, where Bessemer Ventures famously declined to invest in Google back in 1999. They probably regret it today, but it also underlines that Google wasn’t a safe bet at the time—AltaVista and Yahoo were dominating search. To know if something will catch on, someone needs to take the risk and invest.
On the other hand, when technology is more mature, it’s more conducive to planning. Airplane technology was quite well established when Europe set up Airbus as a competitor to Boeing. The jet engine—the last significant innovation—had already been invented. You were catching up to a static target, not trying to catch a moving one. When that’s the case, it’s much easier to plan, coordinate, and scale.
The implication is that you need to move between these two phases. You explore, you get a prototype, you scale—and sometimes that runs into diminishing returns, so you need to move on to something new. That requires more decentralised structures. Some things are conducive to both exploration and scaling, though: having a large connected marketplace where people can move around and spread ideas, low transportation costs, good distribution networks. That helps both innovation and scaling.
Philip Bell: I really liked that argument—the idea that you need the push of exploitation and the pull of exploration. You illustrated it through the tinkerers and inventors of 18th and 19th century Britain, and then the centralised bureaucracy of Bismarck’s Germany and Meiji Japan. One term you used which I found evocative was the idea of adapting your “ecological niche.” Is it easy for contemporaries to understand what ecological niche they’re in? Is it obvious at the time, or is there also luck involved?
Carl Benedikt Frey: I think the big question is “make or buy.” Businesses do this all the time: do we invent something internally from scratch, or do we take existing technology, tweak it, and scale it?
When the Industrial Revolution took off in Britain, I think it was a fairly straightforward decision for many firms and states on the continent to buy rather than make—although there was a bit of both. States pursued various tactics to attract talent from Britain who could make things in Germany and France. It was more a question of catching up to the technological frontier than pushing it forward. Germany did come to push the frontiers, particularly during the Second Industrial Revolution. But the key question is: are you lagging behind or are you at the frontier? That determines which path you choose.
Philip Bell: That’s interesting for the current question about geopolitical sovereignty in the age of AI. Canada has championed Cohere, France has championed Mistral—even though those models don’t benchmark as well as Google, OpenAI, or Anthropic. Do you think the make-or-buy question is relevant today for AI specifically?
Carl Benedikt Frey: Most certainly. The question is how easy it is to access the technology you need from abroad. During the post-war period, American technology and know-how was readily available through Marshall Aid, largely because of concerns about Soviet influence on Europe. America wanted Europe as a buffer, and that paved the way for much of the technology transfer we saw. It’s less clear today that Europe will have access to some of the technology being developed in the United States.
One of the puzzles in my mind is that for the past couple of decades—this predates the Trump administration—Europe has been less dynamic, which is maybe not surprising. But it has also not caught up in digital. Europe managed to catch up in mass production in the post-war period but failed to do the same in digital. Why?
I think a big part of the answer is that the single market is much more harmonised for goods than for services. The IMF estimates that if you take all barriers to trade inside the European Union and add them up, they amount to something like a 110% tariff. Trump Liberation Day tariffs, self-imposed inside the European Union. That obviously caps the return to investment in digital in Europe. What both China and the United States have in common is large domestic markets that firms can scale into. For Europe, a key priority needs to be harmonising the single market for services.
Philip Bell: I’d never thought about it that way—that Europe hasn’t caught up in digital. There are European tech companies like Spotify and Klarna, but they’re few and far between. And it’s not a reflection of talent, because major tech companies open large offices in London to hire talented individuals at cheaper prices than in the US.
In the book, it seems like the nature of the specific technology has some bearing on what institutions are most fruitful. You argue that centralised Soviet bureaucracy was useful for building heavy industry, but not for digital. Is there a specific institutional arrangement that will particularly help in the age of AI?
Carl Benedikt Frey: I don’t know is the honest answer. With regard to the Soviet Union, they did well in heavy industry because they prioritised it—it aided the military-industrial complex, and defence was the key priority. As long as technology was static, Soviet elites could benchmark factory performance and hold managers accountable. The problem was that when mass production petered out, something new was needed for growth. That new thing was the computer revolution, to which Soviet contributions were essentially none.
Part of the reason is that when you introduce a new technology in production, you can’t really benchmark. It becomes much harder to monitor performance. And unlike China, which is much more regionally decentralised, there was very little space to experiment in the Soviet Union without pulling the rug on the entire system.
I don’t think AI is that different in that regard. AI development is quite concentrated in a few places. When technologies are implemented, they proliferate more rapidly than manufacturing industry did. And you see work beginning to migrate abroad as well—law firms reducing headcount in London, hiring more in Poland and India to save labour costs. AI aids that process because it reduces the productivity differential between a worker in London and a worker in India.
There are probably things that will be different with AI, but it’s too early to say what institutional arrangements will be needed. What we can say is that certain things matter regardless of technology: barriers to entry are important for innovation, and having a large homogeneous market helps scaling.
Philip Bell: Do you think AI will change the cost of exploration differently from how it changes the cost of exploitation? I’m building a startup, and AI has accelerated our implementation to an extreme degree. But the rate of exploration has increased much less—the people using our app still use it at a similar rate. We can build things quicker, but do you think AI will change the relationship between exploration and exploitation?
Carl Benedikt Frey: I think yes, but probably in similar ways to the internet and the personal computer. When I have an idea, I can check fairly quickly using my preferred AI tool whether somebody else had the same idea. But the internet did that too. AI will reduce the cost of exploration further.
The surprising thing is that despite the cost of setting up a company and exploring going down so much because of technology—the cloud has been enormously helpful to smaller firms—we’re seeing business dynamism in decline. Something is pushing in the other direction and has more than offset the advantages created by ICT and arguably AI.
What is that? I’m at the university, and we see new rules and regulations and forms almost every week. That might be extreme, but I do think regulation has something to do with increasing costs. If you take an extreme example: most people believe AI will have a material impact on medical discovery. Even if that’s true, you still have to go through clinical trials, which is tremendously expensive. You probably need to partner with a large pharmaceutical company. I’m not suggesting we get rid of clinical trials—they fulfil a purpose. But those safety measures do come at a cost, primarily prohibitive for solo inventors and smaller firms.
Another relevant aspect is what we incentivise. In academia, it’s publish or perish. When a new productivity tool arrives, we can do one of two things: use it to drill more holes, or use it to dig deeper and focus on one thing. It seems we’re incentivising people to do many things at any given point in time, which means attention is spread more thinly across multiple projects. We have research showing that the more projects you pursue at any given point in time, the less likely you are to make a real breakthrough. Those incentive structures might be another reason we haven’t seen a real upsurge in breakthrough innovation despite having tools that clearly aid exploration.
Philip Bell: That reminds me of Marconi, who developed the radio on his estate near Bologna. He probably didn’t have loads of distractions on his phone and could concentrate. It feels like today it would be harder to do that. Do you think AI could facilitate more of that tinkerer culture coming back?
Carl Benedikt Frey: Marconi had some help from the family butler too, so most people don’t have that. It’s entirely plausible that AI will democratise innovation. If everybody has access to their army of personal tutors and research assistants, that should tremendously aid people with good ideas. We could see an era of more decentralised exploration, unlike what we saw in the 19th century.
What makes me think that’s probably not going to happen is the experience of the past two decades. ICT has already provided many tools that should make exploration cheaper and easier, and we haven’t seen the re-emergence of the solo inventor to the same degree as in the 19th century.
It is true that the computer revolution in the United States was driven by new firms—Apple, Microsoft, Google, Amazon. But since the 2000s, we have a trend where new firms exit by acquisition rather than IPO. WhatsApp, Instagram, YouTube didn’t become firms in their own right. That’s not necessarily bad, but the trend towards killer acquisitions—where incumbents buy smaller firms just to shut them down and avoid competition—is concerning. The dynamism we saw during the early days of the computer revolution is no longer quite there, despite computers being much better today than in 1980.
Philip Bell: How useful is it to use historical analogies to understand AI? Sundar Pichai compared AI to fire, Demis Hassabis compared it to the Industrial Revolution, Alex Karp compared building AI to the Manhattan Project. China’s AI Plus plan compares it to the internet or electricity. Is it useful to think about AI in comparison to analogous technologies, or should we take it on its own terms?
Carl Benedikt Frey: It depends on the point you want to make. Going back to my previous book, The Technology Trap, I think the framework of replacing technologies that automate existing work versus enabling technologies that create new types of tasks—developed by Daron Acemoglu and Pascual Restrepo—sheds light on AI as well.
If AI is just used for automation, we’re more likely to see something similar to the First Industrial Revolution, where wages were stagnant and probably even falling at the lower end of the income distribution. But if AI is used to create new products and industries, we’ll see something more similar to the Second Industrial Revolution, where the automobile industry emerged as the largest industrial undertaking the world had ever seen, along with supplier industries, electrical industries, and essentially every household appliance you have in your kitchen.
But to say AI is inevitably going to reproduce the 1990s or 20th century pattern—I think that’s probably not right.
Philip Bell: In The Technology Trap, you describe how technologies have sometimes caused short-term disruption but enabled long-term progress, partly determined by political impact. Do you think society has begun to perceive technology as more deterministic? When Dario Amodei says 20% of white-collar workers will be automated, it feels quite techno-determinist. Does that change how institutions, governments, and populations react to technology?
Carl Benedikt Frey: I’m not sure if people have become more techno-determinist. Generally speaking, the further back in time you go, the smaller the cross-section of society you get on record. If you go back to the 18th and 19th century, Marx, Malthus, Ricardo—they all believed in some form of iron law of wages. They didn’t think technology could improve standards of living over the long run.
In the mid-20th century, there was enormous optimism about electricity and personalised travel. Today there’s a lot of anxiety over AI, and a lot of people saying, “It’s the way it is, let’s hope for the best and maybe plan for the worst.” I think there have always been some of these impulses, but I may be wrong.
Philip Bell: I’ve seen a debate between Arvind Narayanan, who argues AI is a normal technology, and AI 2027’s Daniel Kokotajlo, who argues AI is completely different because it will be self-replicating. Henry Farrell recently argued we should think of AI as a cultural technology, similar to the market or bureaucracy or democracy, because it allows classification on a large scale, which has never been possible before. Do you think it’s too early to make that classification?
Carl Benedikt Frey: I haven’t read those articles, but I don’t think AI is there yet. AI is not going to replace an organisation or a firm anytime soon. Anthropic did an experiment recently where they had a vending machine being run by Claude—stocking it and selling products, a relatively straightforward operation. But Anthropic employees are a bit more experimental, and they managed to trick it into selling everything at 25% discounts and stocking the machine with metal cubes.
This is a relatively straightforward operation, and I wouldn’t want AI running anything too complex right now. When you apply AI for fairly well-defined tasks with a lot of precedent, it does pretty well. But the messier something gets, the more moving parts you have that affect each other, the more unpredictable it becomes. AI generally doesn’t do very well at generalising to situations it hasn’t seen before in training.
Advances will likely emerge in the next few years. Time will tell. Right now, I think it’s best to think about the AI we have as a tool, similar to the internet or the personal computer. In the future, maybe—but I’m not sure.
Philip Bell: Do you think historical research will change in the age of AI? I saw an economist recently write a paper about how to use AI agents for economics. Are there going to be huge teams of AI agents going through archives?
Carl Benedikt Frey: I’m not a historian, I’m an economist, so I’m not sure I’m best placed to answer. But speaking to historian colleagues, the challenge is that for AI to be of much use, things need to be digitised. AI might help that process, but that’s the first step. I suspect historians who spend much of their time in the archives will be some of the last researchers to be replaced—but I may be wrong.
Philip Bell: That’s a good point. A lot of things aren’t digitised, and that information will become increasingly precious because AI companies want very specific data involving lots of reasoning. Do you personally use AI much in your work?
Carl Benedikt Frey: I use it a bit. Partly because I write about it, I feel like I have to. I find it quite useful for some tasks, less useful for others. But the only way of knowing is trying it. So yes, I use it on a daily basis now.
Philip Bell: One last question: do you have any book or article recommendations? It could be about the political economy of technology or anything else.
Carl Benedikt Frey: The book that made me want to become an academic was Joel Mokyr’s The Lever of Riches, so I have to plug that one. I really like Doug Irwin’s Clashing Over Commerce—it’s a wonderful economic history of the United States in general, and a wonderful history of trade policy in particular. And Adam Tooze’s The Wages of Destruction: The Making and Breaking of the Nazi Economy is a book that’s beautifully executed and a pleasure to read.
Philip Bell: Thank you so much, Carl. It’s been such a pleasure to speak to you.
Carl Benedikt Frey: Thank you. It’s been a pleasure, enjoyed it myself.
