What I learned watching 78 videos from Tesla's Austin robotaxis

Tesla exceeded my expectations, but it still has a long road ahead.

A lot of people expected Tesla’s new robotaxi service in Austin to be a flop. Let’s be honest: some people seemed to be rooting for it to fail. After its June 22 launch, video clips started circulating on social media that seemed to justify their skepticism.

I tweeted one of those clips myself, but I also knew that short clips can easily give a misleading impression. So before forming an opinion, I wanted to view as much raw footage as I could. Over the last two weeks, I’ve done exactly that: I’ve watched 78 videos posted by pro-Tesla influencers who got early access to the service. Those videos documented more than 16 hours of driving time across nearly 100 rides.

These videos exceeded my expectations. Tesla’s robotaxi rollout wasn’t perfect, but it went as well as anyone could have expected. A handful of minor glitches got outsized attention online, but a large majority of trips were completed without incident.

For the most part, Tesla’s robotaxis are smooth, confident drivers. They can drive in the rain and the dark. They are unfazed by construction sites and pull over for emergency vehicles. They can navigate chaotic parking lots.

So Tesla fans can be justifiably proud of the robotaxi launch. At the same time, I think many people are underestimating how much work Tesla still has ahead of it.

I watched many videos featuring two pro-Tesla influencers riding in a robotaxi together. A common topic of conversation was how long it would take Tesla to expand its service. Some predicted that Tesla’s superior manufacturing capacity would allow it to quickly eclipse Waymo, the current industry leader.

But I don’t think Tesla’s accomplishments over the last two weeks—impressive as they are—mean that Tesla is ready to quickly scale up its service.

For one thing, Tesla hasn’t completed nearly enough miles to determine whether its robotaxis are as safe as a human driver. Human drivers go hundreds of thousands of miles between serious crashes. With about a dozen vehicles in its robotaxi fleet, Tesla has probably logged fewer than 10,000 driverless miles so far—not nearly enough to make any judgments about safety.

Second, we don’t know how quickly Tesla can reduce its reliance on human oversight. Right now, every Tesla robotaxi has a “safety rider” in the front passenger seat. There is reason to believe Tesla also has a teleoperation team that can control robotaxis remotely.

I don’t know how this works or how often it happens. But if Tesla’s robotaxis are heavily dependent on remote assistance—which they might be—that would make it impossible to profitably expand the service.

Still, Tesla’s Austin launch is a sign that Tesla is serious about its robotaxi ambitions. Which means that Waymo needs to keep its foot on the accelerator if it wants to maintain its lead. And the company seems to be doing just that:

• A startup called Moove is reportedly on the verge of raising $1.2 billion to purchase and manage a fleet of Waymo robotaxis, which should accelerate Waymo’s commercial expansion.

• In recent weeks, Waymo has begun preliminary steps to expand its service to Boston, Philadelphia, New York, Washington DC, Houston, Dallas, San Antonio, Orlando, and other cities.

It’ll take a couple of years for Waymo to launch commercial services in most of these cities. Still, I think Waymo is likely to get to most of these cities before Tesla does.

I didn’t see any serious robotaxi mistakes

Tesla’s robotaxis drove flawlessly during the vast majority of the 16 hours of driving footage I watched. They stayed in their lane, followed traffic laws, and interacted smoothly with other vehicles. Here are a couple of moments that impressed me:¹

• A Tesla robotaxi encountered a parked car with both its hood and the driver’s door open. The Tesla waited patiently for a break in the oncoming traffic and then nudged into the opposing lane to give the parked car a wide berth.

• A Tesla robotaxi approached an intersection behind a vehicle that was trying to turn left but then changed its mind. As the vehicle swerved back into the Tesla’s lane, the robotaxi smoothly slowed down to give it space.

Neither of these maneuvers were all that amazing; an experienced human driver would have handled them without any trouble. But situations like these might have tripped up earlier versions of Tesla’s Full Self-Driving technology.

Although Tesla’s driving was good, it wasn’t perfect. Let me now run down the most significant mistakes I noticed. Almost all of them have been discussed elsewhere online.

Tesla’s most widely discussed error occurred around seven minutes into this video. The robotaxi approached an intersection and got into the left turn lane. But the robotaxi couldn’t make up its mind whether it wanted to turn left or go straight. The car’s steering wheel jerked back and forth several times. On the car’s display, the blue ribbon showing the car’s intended path jumped back and forth erratically between turning left or continuing straight. Finally, the Tesla decided to proceed straight but ended up driving the wrong way in the opposite left turn lane.

It’s hard to say how much weight we should give to this incident. Clearly it’s not optimal driving behavior, but there were no other cars nearby and so no serious risk of a collision. Later in the same video, the car hesitated as it seemed to briefly consider making a left turn. Then it changed its mind, headed straight, and made the next left instead. Again, this wasn’t ideal but also didn’t pose a safety hazard.

Tesla’s vehicles also had some trouble with sudden braking. Here were the three most serious examples:

• In this video, the vehicle briefly slows down for no obvious reason.

• In this video, a robotaxi seems to slow down as it approaches a police car with its lights on—even though the police car is in an adjacent parking lot, not on the road the robotaxi is driving on.

• In this video, a robotaxi suddenly slams on its brakes. The vehicle was driving toward the sun and may have been temporarily blinded by the sunlight.

Interestingly, one video from Austin showed a Waymo vehicle slamming on its brakes and jerking the wheel as it went under an overpass. I viewed many fewer hours of Waymo footage than Tesla footage for this story. So the fact that I saw even one incident like this suggests that Waymo may have as much of a phantom braking problem as Tesla does.

Another area where Tesla struggled was with pulling over. When a Tesla robotaxi is close to its destination, it offers the passenger a “drop me off now” button. If this is pushed, the robotaxi is supposed to find a safe place to pull over. But on at least two occasions this worked poorly:

• In this video, a robotaxi let passengers off in the middle of an intersection for no obvious reason. After the passengers exited, the vehicle remained frozen in place for another 30 seconds, blocking cross traffic.

• In this clip, the robotaxi tried to drop a passenger off in a left turn lane. The passenger ended up calling support to resume the ride.

In another video, a Tesla robotaxi was behind a UPS vehicle that began to back up to get into a parking space. The safety monitor quickly disabled the self-driving software, bringing the robotaxi to a stop. It isn’t clear if there would have been a collision without that intervention, though it seems unlikely to me.

In this clip, a Tesla tried to squeeze by another car in a parking lot when its tire touched the adjacent vehicle. The safety passenger eventually slid over to take the wheel and drive the car away. This is the only time I saw a Tesla employee manually driving a robotaxi.

In total, that’s eight fairly minor mistakes over 16 hours of driving. I don’t think we can draw any firm conclusions from this—the footage I watched was not a random sample and I don’t know how many incidents like this I’d see if I watched 16 hours of human or Waymo driving footage. But personally, I don’t think any of these incidents suggest that Tesla launched its driverless testing program too soon.

Tesla is following in Waymo’s footsteps

Waymo’s vehicles are only available in a handful of metropolitan areas. Waymo’s commercial vehicles don’t operate on freeways. And Waymo has remote operators who can intervene if the vehicles get stuck. For years, Tesla fans argued that these precautions showed that Waymo’s technology was brittle and unable to generalize to new areas. They claimed that Tesla, in contrast, was building a general-purpose technology that could work in all metropolitan areas and road conditions.

But in a piece last year, I argued that they were misunderstanding the situation.

“Tesla hasn’t started driverless testing because its software isn’t ready,” I wrote. “For now, geographic restrictions and remote assistance aren’t needed because there’s always a human being behind the wheel. But I predict that when Tesla begins its driverless transition, it will realize that safety requires a Waymo-style incremental rollout.”

That’s exactly what’s happened:

• Just as Waymo launched its fully driverless service in 50 square miles near Phoenix in 2020, so Tesla launched its robotaxi service in about 30 square miles of Austin last month.

• Across 16 hours of driving, I never saw Tesla’s robotaxi drive on a freeway or go faster than 43 miles per hour. Waymo’s maximum speed is currently 50 miles per hour.²

• Tesla has built a teleoperation capability for its robotaxis. One job posting last year advertised for an engineer to develop this capability. It stated that “our remote operators are transported into the device’s world using a state-of-the-art VR rig that allows them to remotely perform complex and intricate tasks.”

The launch of Tesla’s robotaxi service in Austin is a major step toward full autonomy. But the Austin launch also makes it clear that Tesla hasn’t discovered an alternative path for testing and deploying driverless vehicles. Instead, Tesla is following the same basic deployment strategy Waymo pioneered five to seven years ago.

Of course, this does not necessarily mean that Tesla will scale up its service as slowly as Waymo has. It took almost five years for Waymo to expand from its first commercial service (Phoenix in 2018) to its second (San Francisco in 2023). The best informed Tesla bulls acknowledge that Waymo is currently in the lead but believe Tesla is positioned to expand much faster than Waymo did.

The case for optimism about Tesla

Being an automaker gives Tesla a couple of significant advantages over Waymo.

First, Tesla has designed its self-driving software to work with the conventional cameras that are already standard on Tesla vehicles. This means that as soon as Tesla gets its self-driving software working well enough, it can start manufacturing tens of thousands of robotaxis per month.

In contrast, Waymo’s technology relies on lidar sensors and other specialized hardware. Waymo only has about 1,500 vehicles in its commercial fleet today, and the company has struggled to acquire more:

• Waymo announced a deal with the Chinese automaker Zeekr in 2021, but Waymo still hasn’t begun using those vehicles in its commercial fleet. The deal could be torn apart by tariffs and other geopolitical pressures, which would make it hard for Waymo to get enough vehicles over the next two years.

• A 2024 deal with Hyundai should ultimately provide Waymo with the vehicles it needs, but it may take a couple more years for deliveries to start.

Tesla’s second significant advantage is its capacity to harvest data from customer-owned vehicles. This gives Tesla access to a truly massive amount of training data.

Last month, Waymo published a study demonstrating that self-driving software benefits from the same kind of “scaling laws” that have driven progress in large language models.

“Model performance improves as a power-law function of the total compute budget,” the Waymo researchers wrote. “As the training compute budget grows, optimal scaling requires increasing the model size 1.5x as fast as the dataset size.”

When Waymo published this study, Tesla fans immediately seized on it as a vindication of Tesla’s strategy. Waymo trained its experimental models using 500,000 miles of driving data harvested from Waymo safety drivers driving Waymo vehicles. That’s a lot of data by most standards, but it’s far less than the data Tesla could potentially harvest from its fleet of customer-owned vehicles.

So if bigger models perform better, and bigger models require more data to train, then the company with the most data should be able to train the best self-driving model right?

“We are not driving a data center on wheels”

I posed this question to Dragomir Anguelov, the head of Waymo’s AI foundations team and a co-author of Waymo’s new scaling paper. He argued that the paper’s implications are more complicated than Tesla fans think.

“We are not driving a data center on wheels and you don’t have all the time in the world to think,” Anguelov told me in a Monday interview. “Under these fairly important constraints, how much you can scale and what are the optimal ways of scaling is limited.”

Anguelov also pointed to an issue that will be familiar to anyone who read last month’s explainer on reinforcement learning.

Waymo’s scaling paper—like OpenAI’s famous 2020 scaling law paper—focused on models trained with imitation learning. Just as frontier labs train LLMs to predict the next token in a sequence of text, so Waymo trained a model to predict the next move human drivers make in real-world driving scenarios. In this paradigm, larger models with more data performed better.

But as regular readers know, leading AI labs have been de-emphasizing imitation learning over the last year in favor of reinforcement learning. A similar transition has occurred in the self-driving world. Anguelov was a co-author of a 2022 Waymo paper finding that self-driving models trained with a combination of imitation and reinforcement learning tend to perform better than models trained only with imitation learning.

Imitation learning is “not the most sophisticated thing you can do,” Anguelov told me. “Imitation learning has a lot of limitations.”

This is significant because demonstration data from human drivers—the kind of data Tesla has in abundance—isn’t very helpful for reinforcement learning. Reinforcement learning works by having a model try to solve a task and then judging whether it succeeded. For self-driving, this can mean having a model “drive” in simulation and then judging whether it caused a collision or other problems. Or it can mean running the software on real cars and having a safety driver intervene if the model makes a mistake. In either case, it’s not obvious that having vast amounts of human driving data is especially helpful.

One finding from that 2022 paper is particularly relevant for thinking about the performance of Tesla’s robotaxis. The Waymo researchers noted that models trained only with imitation learning tend to drive well in common situations but make mistakes in “more unusual or dangerous situations that occur only rarely in the data.”

In other words, if you rely too much on imitation learning, you can end up with a model that drives like an expert human most of the time but occasionally makes catastrophic mistakes. So the fact that Tesla’s robotaxis drive so smoothly and confidently most of the time doesn't necessarily mean they are likely to react as well as human drivers in extreme situations.

We don’t know how Tesla handles teleoperation

As I watched Tesla’s robotaxis smoothly handle a wide range of tricky situations, I kept wondering if Tesla’s software was actually doing the driving. Here’s a photograph posted by a Tesla employee the day of the Austin launch.

Let’s zoom into the left-hand side of the image:

That appears to be a desk with a steering wheel. And this makes me wonder how—and how often—Tesla employees provide remote guidance to the company’s robotaxi fleet.

Since its 2018 launch, Waymo has acknowledged that it has remote operators who sometimes provide real-time assistance to its vehicles. But Waymo has also said that these remote operators never drive the vehicles in real time. Instead, they provide high-level feedback, while the vehicle always remains in control of second-by-second decisions.

In contrast, Tesla’s job posting stated that teleoperators can be “transported into the device’s world” so that they can “remotely perform complex and intricate tasks.” Could those “complex and intricate tasks” include driving the car for seconds or even minutes at a time?

In the videos I watched, a number of Tesla’s early customers commented on how human-like Tesla’s driving was. That might just be a tribute to the quality of Tesla’s AI model. But it’s also possible that sometimes a human driver is literally driving the vehicle from a remote location.

A few Tesla influencers mentioned this possibility before dismissing it as impossible. But it’s not impossible. A number of startups have developed real-time teleoperation technology over the years. For example, a company called Halo uses teleoperation to deliver rental cars to customers. Here’s how TechCrunch describes the technology:

The vehicles are remotely piloted over T-Mobile’s 5G network, with AT&T and Verizon used for backup. Halo developed an algorithm that allows the data streams to use whichever network connection is strongest at any given time in order to ensure reliable, high-quality streaming and low latency.

Halo’s remote operators can drive cars at speeds up to 25 miles per hour. That’s fast enough to cover a lot of the driving I saw Tesla’s robotaxis do in Austin. And in particular, it’s fast enough to cover a large fraction of the most difficult driving environments, such as parking lots and unprotected left turns.

I’m not going to make an accusation here—I’m not even sure if it would count as a scandal if remote drivers were regularly dropping in to help Tesla’s vehicles out of tough spots. But I do think it’s worth keeping this possibility in mind if you’re trying to assess the progress of Tesla’s technology.

The caliber of driving I saw in those 78 videos would represent a significant breakthrough for FSD. If Tesla continued improving at that pace, it could be ready for large-scale deployment fairly soon. But of course it’s only impressive if Tesla’s software was actually driving the vehicle.

On the other hand, if human drivers were in control much of the time, then the performance of Tesla’s robotaxis in Austin is much less impressive. In that case, Tesla could be quite far from achieving the performance required to scale up its robotaxi service.

1. This list originally included a video with the title “Raw 1x: Happy Tesla Robotaxi Launch Day,” which I wrongly assumed was an Austin robotaxi video. In fact it appears to have been a Tesla vehicle running FSD in New York. I regret the error.

2. Waymo has been doing driverless testing on freeways for 18 months at speeds higher than 50 miles per hour. But the company has not yet rolled out this capability to its commercial service.