Why I'm skeptical simulation can enable superintelligence

Accurate simulation requires a lot of real-world data.

In a November piece, I argued that singularists were drawing the wrong conclusions from the stunning success of computer chess over the last 25 years. After Deep Blue beat grandmaster Garry Kasparov, chess software kept getting better. Today the best chess engines are much better at chess than any human player. Singularists like Nick Bostrom predict that something similar will happen with other cognitive tasks.

But I pointed out that chess is different in a crucial respect. Because chess is a deterministic game of perfect information, it’s possible to "look ahead" many moves in a chess game. The real world is far more complex and we often have to make decisions with limited information. So the same approach doesn’t work for most problems humans face.

A few readers pointed out that I was oversimplifying the situation with chess and other games—like Go and Starcraft—where AI software has made impressive progress.

Early chess engines worked by “brute force,” searching through a huge number of possible move sequences to find the best one. In contrast, recent software such as DeepMind’s AlphaZero relies on neural networks to recognize promising moves in a more human-like way. These systems still analyze many possible move sequences, but neural networks let them focus on the “most likely” sequences and thereby “see” even further into the future.

Training these neural networks requires training data, and these systems get their training data through self-play: the AI plays against itself and uses the outcomes of these games to train the next iteration of the software.

In a comment on November’s post, reader Isaac King argued that a similar approach could work for many real-world problems. “The real world, just like chess, follows a set of relatively simple deterministic rules called ‘physics,’” King wrote. King believes that the only difference is that the number of possible “moves” in real-world problems is much larger. But King argues that real-world problems can be solved in a similar way: simulate the physical world and use data from the simulation to train a neural network.

King is right up to a point. Some physical tasks are simple enough that it’s possible to get useful training data from a simulation. For example, in its early years OpenAI used simulations to train a robot hand to solve a Rubik’s Cube. The simulation was good enough that software trained in simulation could successfully solve a Rubik’s Cube in the real world (though there were some significant caveats).

A simulation like this doesn’t need to be perfect. It just needs to be good enough that skills learned in the simulation transfer over to the physical world. That’s not too difficult for a simple, mechanical task like a robot hand manipulating a Rubik’s Cube. But it gets exponentially more difficult for complex tasks.

Waymo shows the value—and limits—of simulation

A Waymo vehicle navigates its environment. (Waymo)

One of the most ambitious companies in this area is Waymo. Back in 2021, Waymo said its software had driven 20 million miles on real roads and 20 billion miles in simulation. That latter figure is around 25,000 times what a typical American will drive in a lifetime.

Waymo has put a ton of effort into making its simulator realistic. For example, you can read about Waymo’s efforts to accurately simulate other drivers and vehicle sensors.

The results have been pretty impressive. As I noted last week, Waymo’s vehicles get into crashes much less often than the average human driver. But in other ways, Waymo’s vehicles are still worse than human drivers. They sometimes get confused by complex and unusual traffic patterns. They have trouble following directions from police officers and firefighters. And Waymo’s rider-only service does not yet operate on freeways, presumably because Waymo isn’t yet confident they can do so safely.

Many of the situations where Waymo’s technology still struggles are situations that are difficult to simulate. For example, to accurately simulate a police officer directing traffic, you need to not only simulate the officer’s body, but his brain as well. And accurately simulating even a single human brain—to say nothing of a whole city full of them—is far beyond the capabilities of today’s computers.

So the Waymo simulator necessarily makes a lot of simplifying assumptions. As a result, people and objects in Waymo’s simulated world sometimes behave differently than people in the physical world. And this greatly limits the potential for Waymo's vehicles to improve through “self-play.”

Simulation clearly adds some value or Waymo wouldn’t do so much of it. But even after practicing for thousands of simulated human lifetimes, Waymo’s software does not seem to have achieved superhuman driving abilities. That’s not because Waymo is doing something wrong. There’s only so much that can be learned from self-play in an environment that doesn’t perfectly match the real world.

Many tasks are much harder than self-driving

Suppose Xi Jinping wanted to create an AI to figure out the best strategy for invading Taiwan. Of course China could create a model of the Taiwan Strait and use it to conduct digital war games. But the simulated world will differ from the real one in significant ways.

The challenge here isn’t just that it’s computationally intractable to simulate the actions of millions of soldiers and civilians in China, Taiwan, the United States, and elsewhere—though it absolutely would be. Another challenge is the need for accurate information about initial conditions:

• How many troops does Taiwan have and where are they deployed?

• Does Taiwan or its allies have weapons systems China doesn't know about?

• What’s morale like among soldiers in Taiwan and on the Chinese mainland?

• Will public opinion in the US favor intervention or staying out of the conflict?

An AI system can’t deduce facts like this from first principles.

Waymo’s simulator is as good as it is because the company has collected tens of millions of miles of real-world driving data. This provides the company with “ground truth” it can use to evaluate and improve its simulation software.

China wouldn’t have the option to do a bunch of “practice invasions” to collect data for a simulator. But without such information, the simulated world is going to differ from the real world in significant and hard-to-predict ways.

This isn’t to say simulations are useless. Militaries today conduct war games, and AI might make them much better. But singularists make a much stronger claim: that self-play in a simulated world can push an AI far beyond human performance. I don’t see how this can work if the simulation doesn’t closely resemble the real world.

A lot of problems are like this. If you’re trying to run a company, design a new product, or negotiate a contract, a lot of the challenge comes from predicting the behavior of other people. And that’s far beyond the capabilities of today’s computers—both because human brains are far too complicated and because we don’t have nearly enough data about what’s inside people’s heads.

Unknown unknowns

Now let’s consider a simpler case where accurate simulation does seem possible in principle: imagine an AI trying to design a new rocket by building and launching rockets in a simulated world.

It’s not feasible to simulate a rocket at the atomic level, so some simplifying assumptions will have to be made. But which aspects of a rocket can be abstracted away without undermining the predictive power of the simulation? To answer that question, you need to know a lot about how rockets fail in the real world.

Building and launching a rocket involves thousands of steps, and a single mistake can cause a catastrophic failure. Maybe someone orders a part that isn’t rated for extreme temperatures or pressures. Maybe engineers fail to account for the way fuel “sloshes around” inside a tank, causing the rocket to veer off course. Maybe someone forgot to add an essential step to the pre-launch checklist.

It’s theoretically possible to build a simulator that would help predict any of these mistakes. But to build that simulator, you probably need to understand these potential mistakes well enough that you could just not make them in your prototype rockets. The challenge is predicting failure modes you don’t know about yet. And it’s easy to overlook those same unknown failure modes when designing the simulator.

My point here isn’t that simulation and AI can’t be useful in designing rockets. I would not be at all surprised if AI were helpful for developing better simulations for rockets and all sorts of other complex technical challenges.

But the singularist premise is that self-play in simulated environments will allow AI systems to rapidly zoom far beyond human capabilities. And I don’t understand how this is supposed to work. Because for a lot of real-world problems, simulation isn’t an alternative to collecting data from the real world; gathering data is an essential precondition for creating a realistic simulation.

AI systems would have to collect data using the same slow, labor-intensive methods that the rest of us do: driving physical cars on real streets, building and launching physical rockets, and so forth. And on the flip side, human beings will be able to use narrow AI systems to help them build simulators of their own.

When a computer wins at chess or Go, it is winning the real game even if the board is represented digitally. In contrast, when an AI system wins at driving a simulated car or designing a simulated rocket, it’s solving a different—and probably much easier—version of the problem than the one it will face in the real world. So I don’t understand how an AlphaZero-like strategy can lead to superhuman performance at tasks like this.