Tesla's near-infinite training data isn't a silver bullet

Imitation learning has inherent limits for self-driving cars—and LLMs for that matter.

It’s Autonomy Week! This is the fourth of five articles exploring the state of the self-driving industry.

---

Waymo has completed more than 22 million driverless miles with an impressive safety record. Tesla hasn’t begun driverless operations, but Tesla fans predict its technology will soon surpass Waymo’s. They say one of Tesla’s secret weapons is access to training data from millions of customer-owned Tesla vehicles.

Last month I appeared on my favorite AI podcast, the Cognitive Revolution, to discuss the state of self-driving technology. Host Nathan Labenz and I went deep on the differences between Waymo and Tesla’s self-driving strategies, including their different approaches to collecting training data.

While Tesla undoubtedly has access to more data than Waymo, I pointed out that Waymo’s data is higher quality in a couple of important ways.

First, Waymo uses lidar sensors to capture three-dimensional point clouds, whereas Tesla cars only have cameras on board. This means Tesla must estimate the three-dimensional structure of the world from two-dimensional images, an inherently error-prone process.

Waymo also has safety drivers who take careful notes each time they take control. This enables Waymo’s engineers to focus on times when the software makes a mistake—and ignore times when the safety driver just needs a bathroom break. Tesla customers can leave audio notes when they disengage FSD, but I doubt they do this as consistently or thoroughly as Waymo’s drivers.

In my conversation with Nathan, I pointed to a related downside of Tesla’s approach: nervous customers are likely to disable the Full Self Driving software as they approach a tricky situation like a construction site or car crash, depriving Tesla of valuable data about how well its software performs.

But Nathan responded with a reasonable question: can’t Tesla’s software just observe how the human driver handles the situation and learn from that? You can listen to my answer in the podcast (the discussion starts around 22 minutes) but I didn’t feel like I had a great response.

I kept thinking about this question after the podcast, and I have a much better answer now.

If you’ve ever tried to learn a new, difficult skill like driving a car, you know there’s a profound difference between watching someone else do a task and trying it yourself. No matter how many hours a teenager spends watching his parents drive, he’s not going to become a good driver without his own time behind the wheel. And when he does get behind the wheel, he’ll learn faster with an experienced driver providing guidance from the passenger seat.

The same principle applies to AI systems. Learning by watching a human expert perform a task—known as imitation learning—is slow and tends to produce brittle results. Like human teenagers, AI systems learn more quickly with real-world experience and strong feedback. The brittleness of a purely imitation-based approach means that massive quantities of purely observational data is unlikely to be a big strategic advantage for Tesla.

Learning to play SuperTuxKart

Brian Christian’s 2020 book The Alignment Problem is my all-time favorite book about AI, and it has a fantastic explanation of the limits of imitation learning. Christian tells the story of two computer scientists at Carnegie Mellon University—graduate student Stéphane Ross and his advisor Drew Bagnell (now a co-founder of Aurora, which I covered on Monday)—who were trying to teach an AI model to play an open source Mario Kart clone called SuperTuxKart.

The researchers wanted to train a neural network to play SuperTuxKart by watching Ross play the game and mimicking his behavior. But even after hours of gameplay, his AI model struggled to stay on the track.

The problem, Christian writes, was that “the learner sees an expert execution of the problem, and an expert almost never gets into trouble. No matter how good the learner is, though, they will make mistakes—whether blatant or subtle. But because the learner never saw the expert get into trouble, they have also never seen the expert get out.”

When Ross played the game, he mostly kept the car near the center of the track and pointed in the right direction. So as long as the AI stayed near the center of the track, it mostly made the right decisions.

But once in a while, the AI would make a small mistake—say, swerving a bit too far to the right. Then it would be in a situation that was a bit different from its training data. That would make it more likely to make another error—like continuing even further to the right. This would take the vehicle even further away from the distribution of training examples. So errors tended to snowball until the vehicle veered off the road altogether.

You might think Ross could fix this problem by driving more erratically. This would give the AI model examples of how to recover from mistakes. The problem is that the model will start to emulate the swerving behavior. And it still probably won’t learn how to recover from every possible driving mistake, since there are far more ways to drive badly than to drive well.

Eventually, Ross found a better way to train his SuperTuxKart bot: randomly trade control between a human expert and the neural network. The AI model would drive for a few seconds and then hand control off to Ross, allowing Ross to correct any mistakes the model had made during its turn. Ross’s corrective actions would then be used to retrain the model.

Christian reports that this worked remarkably well:

With static demonstrations alone, the learner was still crashing just as often after a million frames of expert data as it was after a couple thousand—it was just as hopeless after hours of tutelage as it was after only minutes. Yet using this interactive method, the program was driving almost flawlessly by its third lap around the track.

The real world is far more complex than the world of SuperTuxKart. In SuperTuxKart, there are no other vehicles, no intersections, no pedestrians or bicyclists, and so forth. So I think the lessons of Ross’s research apply with even more force in the real world, where the set of mistakes that an AI system might make is exponentially larger.

There are ways to augment observational data to make it useful for training. For example, in a 2019 paper, Waymo researchers took human driving data and modified it to cause a simulated vehicle to run off the road or crash into another vehicle. They could then train a model not to engage in these behaviors. In a 2022 paper, Waymo researchers augmented imitation learning by penalizing driving trajectories that came uncomfortably close to other vehicles, encouraging safer driving behavior.

This research seems worthwhile as far as it goes, but as the 2019 paper notes, there are “highly interactive situations such as merging” where simple tweaks to reinforcement learning are unlikely to cut it. Emergency scenes seem like a particularly good example, since the desired behavior is far more subtle than merely not running into other vehicles. This is why self-driving systems need opportunities to make mistakes in the real world and then learn from those mistakes.

The implications for Waymo and Tesla

Photo by Edie Leong for The Washington Post via Getty Images

It wouldn’t make sense for a self-driving company to literally adopt Ross’s approach and randomly swap control between a self-driving system and a human driver. But Waymo’s approach seems like a close substitute: the safety driver lets the software drive until it starts to make a mistake, then takes over and corrects the mistake. Not only can Waymo train its software based on whatever the safety driver does in the seconds after he takes over, it can also train the software not to do whatever it was doing in the seconds before the safety driver took control.

Tesla relies on customers rather than safety drivers to supervise its vehicles. And these customers are not going to be as fastidious about recording which disengagements were in response to an error and which were for other reasons.

This means that Tesla’s customer-generated data will tend to be a lot noisier than Waymo’s data generated by professional safety drivers. And this problem will get worse as Tesla’s software gets better.

Suppose that the average driver deactivates FSD once every 10 miles for reasons unrelated to safety (bathroom break, accidentally bumped the steering wheel, nervous the car might be about to make a mistake). If Tesla’s software makes a mistake every mile, then 90 percent of disengagements will be safety-related and Tesla can treat a disengagement as a fairly reliable signal that the car made a mistake. But suppose a few years later FSD has gotten a lot better and is only making a mistake every 1,000 miles. Now 99 percent of disengagements will be false positives. At this point, it will take extensive post-processing to make the disengagement data useful for training.

And it’s only going to be useful if Tesla customers let the vehicle try and fail at a difficult driving situation. If the customer proactively disables FSD as she’s approaching a tricky situation (like the site of a car crash, fire, or construction project), then Tesla will only get low-value observational data.

An analogy to LLMs

Some Tesla fans point to rapid progress in large language models in recent years as evidence of the value of large data sets. But I think this analogy actually points in the opposite direction: while LLMs are impressive in many ways, they are nowhere close to the level of reliability society will expect from autonomous vehicles. And a big reason is that LLMs are mainly trained using imitation learning. The lesson of SuperTuxKart applies to them too.

Ross was trying to train an AI model to predict the next move a human SuperTuxKart player would make in a particular situation. In a similar way, models like Claude, Gemini, and GPT-4 try to predict the next token that would be produced by a typical human writer given a sequence of input tokens. In both cases these are sequential tasks: each “move” (joystick movement or token choice) alters the situation the AI faces at the next time step. And this means that both types of AI model have the potential to “wander off track.”

Each token an LLM produces becomes part of the input that’s used to select subsequent tokens. As long as an LLM makes good token choices, its output so far will look similar to its training data and it’s likely to continue making good choices. But if a model ever makes a bad token choice, then it will be trying to complete a sentence that doesn’t look very much like its training data. This makes it more likely that it will make another mistake, and another, until its input looks dramatically unlike anything in the training data.

I suspect this is how Microsoft’s early Bing chatbot wound up on the front page of the New York Times: reporter Kevin Roose spent two hours talking to the chatbot. As the conversation went on, early mistakes snowballed and the model got farther and farther from the examples in its training data, leading to an increasingly crazy conversation.

There are two basic strategies for addressing this problem with LLMs, and they mirror the strategies taken by self-driving companies.

One is to have human beings give direct feedback. Just as the safety driver will grab the wheel if a Waymo car makes a mistake, so frontier AI labs hire human beings to rate LLM outputs in a process known as reinforcement learning from human feedback.

The other approach is to generate synthetic training data to cover more situations than are represented in the organic training data. Every major self-driving project augments its real-world training data with billions of simulated miles. In a similar vein, the Llama 3.1 paper from Meta describes how the company used earlier versions of LLMs to generate millions of training examples in areas like math and computer programming where the correct answers can be verified programmatically.

But it’s an open question how well this will work. Some machine learning experts believe that autoregressive LLMs—those that simply “predict the next word” without any higher-level planning or reasoning ability—are inherently limited because it will never be possible to generate enough training data to reflect the complexity of the real world.¹

1. Yann LeCun, Meta’s top AI scientist, has famously written that “auto-regressive LLMs are doomed.” The last section of this piece is inspired by a comment by user Razied at lesswrong.com that purports to explain LeCun’s argument. However, I think Razied’s argument is different and significantly more persuasive than LeCun’s.