It’s day three of Robot Week! You can click here to get 25% off an annual subscription.
Most people first heard about large language models after OpenAI introduced ChatGPT in 2022. But in the minds of many AI researchers, the key breakthrough came two years earlier with the release of GPT-3.
With 175 billion parameters, the OpenAI model was more than 100 times larger than its predecessor, GPT-2. It was trained on a massive 300 billion tokens. And as a result, it generalized far better than previous models. For the first time, a single model could perform a wide variety of tasks — from translating between languages to answering trivia questions — without task-specific training.
It took OpenAI two more years to develop the techniques that transformed this raw “base model” into a user-friendly chatbot like ChatGPT. And then it took a couple more years to develop the techniques — like long-context reasoning, tool use, and context management — that transformed those early chatbots into the powerful agents we have today.
In short, there was a long road from GPT-3 in 2020 to Claude Code in 2025. But for those who knew where to look, the potential of LLMs was already clear in 2020.
The robotics world is now traveling a similar path. Its “GPT-3 moment” came in July 2023, when Google announced a model called RT-2. To create it, Google trained a multimodal LLM to directly generate robot actions. RT-2 wasn’t Google’s first transformer-based robotics model — the company released a predecessor called RT-1 a few months earlier, for example — but RT-2 was massively larger than earlier models. RT-1 had 35 million parameters. The RT-2 models had billions of parameters.
And as with GPT-3, size mattered. The RT-2 team reported its model showed “significant improvements to generalization over objects, scenes, and instructions.” They added that the new model exhibited “a breadth of emergent capabilities inherited from web-scale vision-language pretraining.”
For example, researchers placed a can of Coca-Cola on a counter alongside framed photos of Snoop Dogg, Tom Cruise, and Taylor Swift. They then prompted the robot to “move coke can to Taylor Swift.” The robot grabbed the can and moved it toward Swift’s photo.
At the time, Karol Hausman was a member of the RT-2 team. In a March interview, he described this as a moment of “huge, huge excitement” because “the robot models had never had any of Taylor Swift in their data. It had to understand the concept of Taylor Swift, connect it to the image of Taylor Swift, and then connect it to the right motion that would move the Coke can to the picture of Taylor Swift, all from Internet data.”
“That was the moment where it clicked for us that it could actually work — where you could bring in a lot of prior knowledge from LLMs, from the Internet, and connect it to robot motions,” Hausman said.
Google dubbed RT-2 a vision-language-action (VLA) model. Both Google’s approach and the term VLA quickly became industry standards. But as impressive as RT-2 was, it also had significant shortcomings — shortcomings the industry has been working to remedy over the last three years.
The RT-2 breakthrough kicked off a robotics boom that’s been underway ever since. Big companies in both the US and China have poured resources into robotics. Numerous robot startups have been created, and several have raised hundreds of millions of dollars in venture capital. And the models powering most of these robots are based on the basic architecture Google pioneered back in 2023.
The origins of RT-2, the first VLA model
Google invented the transformer in 2017 and had been experimenting with large language models ever since. The company had also been working on robotics for many years. So combining LLMs and robots was an obvious research direction.
In March 2023, Google announced PaLM-E, a 12-billion-parameter model that was optimized for robotics (the “E” stood for “embodied”). PaLM-E was a vision-language model (VLM) — meaning an LLM trained to understand images as well as text. It had been trained to generate natural-language robot commands like “move the blue block to the left.”
But PaLM-E couldn’t control a robot directly. Google’s robots didn’t have enough onboard computing power to run a VLM as large as PaLM-E. So PaLM-E ran in the cloud, and it was designed to work with a second, smaller model that would run on the robot. This second model would translate PaLM-E’s English instructions into low-level robot commands.
The RT-2 team’s plan was simple: delete the smaller model and instead train PaLM-E to directly control the robot.1 RT-2 — like PaLM-E — was too big to run directly on a robot. So the team ran the model in a Google data center and had it send commands to the robot over the network.
Like any LLM, RT-2 worked by prompting. Google would send RT-2 a prompt like “What action should the robot take to move coke can to Taylor Swift?” along with an image from the robot’s camera.
RT-2 would respond with a sequence of numbers like “1 128 91 241 5 101 127 217.” The robot would interpret this as a command to move the robot’s gripper to certain x-y-z coordinates (like x=128, y=91, and z=241), rotate the gripper to a certain angle (roll=5, yaw=101, pitch=127), and open (or close) the gripper to a certain position (217).
Then RT-2 would get the same prompt again, but with a fresh image. The model would generate another sequence of numbers representing a new target position for the robot arm. The robot would move its arm another few inches. Then the whole cycle would repeat again. It might take dozens of iterations to complete a task like “move coke can to Taylor Swift.”
To transform PaLM-E into RT-2, Google had to teach the model how to generate low-level robot instructions. That required a different kind of training data.
To collect that data, Google built three test kitchens and purchased 13 robots. Over the course of 17 months, human workers teleoperated the robots as they performed tasks — picking up objects, opening drawers, placing objects in the drawers, and so forth — more than 130,000 times.
Training PaLM-E on this data gave RT-2 surprisingly broad capabilities. Robots could manipulate objects they hadn’t seen before. They could operate in new kitchens. And they could complete tasks on counters that were cluttered with “distractor objects” that weren’t needed for the assigned task.
Five roboticists left Google to co-found Physical Intelligence
Karol Hausman was excited by the RT-2 breakthrough, but he also concluded that Google wasn’t the right place to develop the technology.
“It became clear that the way to accomplish this is to create an organization whose sole purpose is to solve physical intelligence,” Hausman said in March. “It can’t be solved as priority number 20 in another organization.”
So Hausman became the CEO of a startup called Physical Intelligence. He was joined by four other members of Google’s RT-2 team and two others from outside Google.
According to Hausman, the team sought out “investors that are fully aligned with this starting as a research company and not being oriented around short-term revenue.”
“If we do this right, this is going to completely change the world and it’s going to be the most valuable business of all time,” Hausman said. “But you need to have the patience to let us do it the right way.”
There was a lot to do. RT-2 was a big improvement over previous robotic models, but it was still far less capable than the average human. Over the last two years, the Physical Intelligence (PI) team has been working hard to close that gap. The company has been remarkably transparent, publishing at least 10 papers describing their work. For this story, I read all the PI papers I could find — along with 20 more from other companies and academic labs.
I’ll use PI’s research as a lens to explain the evolution of VLA models over the last three years. During that time period, VLA-controlled robots achieved much better fine motor control. They gained the ability to perform complex tasks that take several minutes. And companies are exploring new ways to have models reason using images as well as text — which could unlock the ability to learn from videos of humans performing tasks.
At the end, I’ll discuss the view that VLA models are on the verge of being eclipsed by a new architecture called world models. PI co-founder Sergey Levine has a perspective on this that I find pretty persuasive.
Robot Week special: Click here to get 25% off an annual subscription.
Improving robots’ fine motor skills
Hausman was impressed that RT-2 was able to move a Coke can to Taylor Swift. But later in the same interview, he described it as “totally unimpressive” and a “pretty pathetic demonstration of what robots could do.” That sounds like a contradiction, but you can see what he meant if you watch the video:
You can view all the RT-2 videos on this page. You’ll see that this was far from the only time the robot was hilariously clumsy. Another trial asked a robot to place a strawberry in a bowl. The gripper grabbed the strawberry and then lurched back and forth indecisively for several seconds before finally — just barely — heaving it into the correct container.
A key challenge was action frequency. Ideally, a robot should receive fresh instructions about 50 times per second, but RT-2 couldn’t generate actions that quickly. The smallest version of RT-2, with 5 billion parameters, generated about five actions per second. Larger versions were even slower. This made the robot’s motions jerky and erratic.
To speed things up, PI models engage in “action chunking”: they generate as many as 50 sequential actions using a technique called flow matching. Flow matching is a cousin of the diffusion techniques used in many generative image models. Whereas LLMs generate one token at a time, flow-matching models can generate all 50 actions in parallel.2 This means the robot is never waiting around for the model to send it the next instruction.
Flow matching has another advantage too: it isn’t constrained to output discrete values. Earlier, I said that RT-2 would output actions like “1 128 91 241 5 101 127 217,” where 128 (for example) was the x coordinate the robot’s gripper should move to next. These numbers are all integers because LLMs output discrete tokens.3
In contrast, flow matching enables PI models to natively generate decimal values like 127.5 or 127.351; they aren’t forced to choose between integers like 127 or 128. This allows PI’s models to control robots more smoothly and precisely.
This approach has rapidly become an industry standard. Over the last two years, Google, Nvidia, Hugging Face, Xiaomi, ByteDance, and Agibot have all announced VLA models that generate action chunks using diffusion or flow matching.
Improving speed and accuracy with reinforcement learning
PI has also worked to give its robots hands-on experience.
The first step in training a VLA model is imitation learning — also known as behavior cloning. That’s when a model is trained to mimic human beings teleoperating robots.
But as I explained in an in-depth piece last year, imitation learning has some limitations. A big one is that models trained with imitation learning suffer from compounding errors. If the model makes an early mistake, that puts it into an unfamiliar situation that’s poorly represented in its training data. That makes it more likely to make a second mistake, and then a third.
The alternative is hands-on experience. Nobody learns to drive a car merely by watching other people drive. You need to get behind the wheel and make your own mistakes. The same principle applies to AI models; reinforcement learning is a family of techniques that allow models to learn by trial and error.
In a November paper, PI described its effort to build a VLA model that learns from experience. During training, PI had its robots work on difficult tasks like making coffee or folding laundry. Sometimes PI let its robots attempt a task without human supervision. Other times, humans observed the robots and intervened if they made a mistake. PI’s reinforcement learning algorithm used data from both types of trials (fully autonomous and human-assisted) to improve the robot’s performance.
PI presented data showing that its latest model could fold laundry and make espresso better than earlier models. But you can quickly grasp the pace of PI’s progress by watching videos of the robot in action. Here’s a robot making espresso in PI’s office:
Compare this robot’s smooth and confident motions with the clumsy Coke can video I shared earlier. The first video was from July 2023. The second one was from November 2025. That’s a lot of progress in a little more than two years.
And these models are not executing pre-set scripts. They can notice and recover from minor errors along the way. In another trial, the company had a robot folding boxes at a nearby chocolate factory. The robot accidentally grabbed two boxes at the same time. It noticed the problem, gently pulled the boxes apart, and put one back in the unfolded pile:
Videos like these reflect a dramatically higher level of manual dexterity than Google’s robots showed three years ago.
Executing more complex tasks
RT-2 performed simple, short-term tasks like picking up an object and placing it in a drawer. These have little economic value on their own. Human workers perform sequences of many simple tasks to accomplish some larger goal, like cooking a meal or cleaning a room.
In recent months, PI has been trying to enable its robots to perform long, complex tasks like these.
Researchers have known for years that language models perform better on difficult tasks if they are trained to “think step by step.” In 2024, OpenAI released o1, a model that was trained to think about a problem for hundreds or even thousands of tokens before generating an answer.
More recently, language models have grown more “agentic” — capable of executing a sequence of tasks in pursuit of a long-term goal. In the last year or two, frontier labs have built sophisticated harnesses like Claude Code to help models manage their context windows during long-running projects.
Physical Intelligence has been moving in a similar direction. In an April 2025 paper, the company introduced a model called π0.5, which was trained to break a high-level goal (like “clean up the kitchen”) into lower-level actions (like “pick up the blue cup” and “place the cup into the sink”). PI explicitly compares this procedure to chain-of-thought reasoning.
Alert readers might notice a similarity to Google’s 2023 research in which a high-level VLM (PaLM-E) broke a task down into subtasks that were executed by a smaller robot action model. The difference is that PI now uses the same model for making high-level plans and generating low-level actions. This means the model generating the actual robot commands is a full VLM that can make sophisticated, context-rich judgments.
How PI models remember the past
Early PI models like π0.5 did not keep information from previous actions in their context window. Rather, each time the model needed to choose a new action, it would start out with a fresh context window and make decisions based on the state of the world as shown by the robot’s cameras.
But in recent months, PI has been experimenting with giving its models the kind of longer-term memory that text-based agents have. The challenge is that robotics models receive images from a robot’s cameras multiple times per second. Depending on how these images are encoded, that can translate to hundreds of tokens every second. So storing more than a few seconds of data would clutter up the context window and degrade the model’s performance.
PI deals with this in two ways. First, the company has developed a compact way to represent a sequence of images.
Like any VLM, PI’s models break up an image into a grid of rectangular patches and then compute a vector representation for each patch. But in PI’s most recent models, the vector for each patch incorporates information from that same region in previous images. The total size of the input doesn’t change (which helps with model performance), but the information inside the patch changes.
For example, a robot might move its arm in a way that temporarily obscures the object it is trying to pick up. Incorporating information from previous images helps the robot to have object permanence — to remember that the object is still there even if it’s not currently visible.
For longer-term memory, recent PI models use words rather than images. They maintain a written summary of what they have accomplished so far — something like “I placed a plate in the cabinet, moved to the counter, and picked up a bowl.”
This can be useful in situations where the next step isn’t obvious from camera images alone. Imagine, for example, that a robot is baking something and needs to mix together flour, sugar, salt, and baking soda. It’s important to add each ingredient exactly once. But an image of the bowl won’t reveal which ingredients have been added so far. A summary like “I put two cups of flour and a pinch of salt into the bowl” can help the robot figure out that it still needs to add sugar and baking soda.
Sometimes a robot will “get stuck” and keep trying the same fruitless action over and over again. Keeping a running summary of previous actions can help the robot realize that it needs to try something different.
Still, this kind of long-term memory is pretty limited. For example, it’s unlikely to enable a robot to fix a car’s engine. That requires gathering information about the engine, developing a theory about what’s broken, attempting a repair (which itself may involve many steps), and then testing to see if it worked. A simple list of past actions probably won’t provide enough information to do this.
The power of visual reasoning
Earlier, I noted that recent PI models break a complex goal down into subgoals. In April, PI supplemented this with the capacity to generate “subgoal images.” In addition to generating descriptions like “pick up the blue bowl,” PI’s latest model, π0.7, generates images of what each of its three cameras (the main camera plus one on each arm) should see after the next step is complete — for example, an image of one of the robot’s grippers grasping the blue bowl.
In other words, PI’s latest model has a human-like ability to “visualize” what the world will look like after it takes a particular action. Including these synthetic images in the model’s context seems to cause it to output better robot actions than it would have otherwise. It can particularly help the robot master tricky tasks that are hard to describe in words.
This is part of a broader effort — at PI and across the industry — to train robotics models to become more visual learners.
As we’ve seen, VLA models are built on an LLM “backbone”: they are trained first with text and static images before robotic actions are added to the mix. But there’s another potentially vast source of data that could be used to pre-train robotics models — videos.
In an April talk, Nvidia’s Jim Fan argued that the future of robotics belonged to models that are trained primarily using videos rather than text. Fan was a co-author of a February research paper that described a step in that direction.
Nvidia trained a model called DreamZero on several hundred hours of videos showing teleoperated robots performing various tasks. Alongside each video, Nvidia collected data on the commands used to control the robot. The DreamZero model was then trained to predict the next video frame and the robot’s actions — all based on previous video frames and robot actions.
With LLMs, we train a model to predict the next word in training documents and get a model that can generate new text. By the same token, if we train a model to predict the next robot action in a video, we can use that model to control a robot based on video input.
Nvidia also showed that the DreamZero model could be used to control multiple robots. It was hardly the first paper to do this.
A few months after Google announced its RT-2 results in 2023, Google introduced another hugely influential robotics model called RT-X. Google organized a coalition of robotics researchers who contributed training data from a variety of different robots with different numbers of arms, different numbers of joints per arm, and so forth.
You might expect that a different model would be needed for each type of robot. But the RT-X coalition showed that cross-embodiment training improved the performance of robotics models in general. In other words, a robot’s performance on a particular task will improve if the training data contains examples of that task being performed by other robots.
Combining these two ideas — cross-embodiment training and videos as training data — points to a tantalizing possibility: could a training algorithm treat humans as another kind of robot? After all, YouTube and TikTok have millions of videos of human beings performing various tasks. If we could turn those videos into training data, it would dwarf the specialized teleoperation data sets that have been the industry’s bread and butter so far.
Fan is certainly optimistic. In his April talk, he showed this meme suggesting that VLAs were dead and “World Action Models” like DreamZero were the future of robotics.
So are traditional VLA models doomed?
In April, journalist Craig Smith asked Sergey Levine, a UC-Berkeley robotics professor and PI co-founder, about the trend toward World Action Models. Here’s what Levine said:
If I want to figure out how to get from where I am now in San Francisco to New York City, maybe I imagine how I get my car keys and I get in my car. I might imagine that I’m taking an airplane. But the further out I think about this, the more abstract that imagination becomes. I’m not imagining exactly what my seat in the airplane is going to look like.
So abstractions are really key to actually effective world modeling. And I think at some level, what language models do, what vision-language models do, and what video prediction models do, and other kinds of world models, is not actually that different, they’re just operating with different abstractions.
And I think a real capable embodied intelligent system like a robot will need many different abstractions. And I suspect that once we figure out how to use abstractions in general, that general part of the question is actually the important one.
The design of π0.7 reflects this vision. As I mentioned earlier, it can “imagine” future states of the world by generating subgoal images. The portion of π0.7 that generates these images is based on BAGEL, a generative image model that you could plausibly describe as a world model.4
I think it’s too early to say whether PI’s VLA-centric architecture will ultimately work best or whether the industry will shift over time to WAMs like Nvidia’s DreamZero — or to some other architecture altogether. But I suspect that Levine is right that the winning approach will be some kind of hybrid. Maybe it’ll be a VLA with a world model inside, like π0.7. Maybe it’ll be a WAM like Nvidia’s DreamZero with an LLM to help with symbolic reasoning.
But I’ll mention one reason Fan’s talk left me skeptical. Around the 10-minute mark, Fan says, “Anyone driving Tesla or Waymo here? You know when you’re driving, you’re actually contributing to the biggest physical data flywheel.”
Fan argues that “we need an FSD equivalent” for robotics, and he believes that egocentric video — that is, video collected by a human being wearing a head-mounted camera — is the solution.
The problem with this argument is that Waymo doesn’t have a data flywheel that involves harvesting data from customer-owned vehicles. And yet Waymo is far ahead of Tesla in the robotaxi race.
As I wrote two years ago, Tesla’s data advantage isn’t a silver bullet. Vast quantities of human demonstration data are helpful for reaching a certain level of performance through imitation learning. But beyond that point, it’s necessary for the vehicle to learn from trial and error. That’s what Waymo has been doing over the last decade as its vehicles have logged tens of millions of miles on real roads with safety drivers.
By the same token, egocentric human videos could certainly be useful for training robotics models. PI is working to incorporate them into its training process, and I expect that lots of other companies are as well. But other types of data also matter — and may ultimately be more important.
In his April interview, Levine said “I suspect that the dichotomy between language models and what people call world models is not as large as some people make it seem.” That sounds right to me.
There were actually two versions of RT-2. I’m going to focus on the version that was based on PaLM-E. But Google also tried fine-tuning another VLM called PaLI-X with the same robot data. PaLI-X had 55 billion parameters. It hadn’t been designed for robotics. Yet the PaLI-X version of RT-2 delivered performance roughly on par with the PaLM-E version.
PI has published three papers explaining its approach to action chunking. During training, PI actually trains recent models to produce robot actions in two different ways: via discrete tokens using a compressed representation called FAST and using continuous values with flow matching. During training, gradients flow backwards from the discrete FAST tokens but not the flow-matching portion of the network. PI does this because it found that propagating gradients backwards from the flow-matching “action expert” tended to degrade the performance of the VLA. Also, training to produce FAST tokens is significantly more compute-efficient, so the company saves compute costs by training only on FAST tokens in the early phases of training, then adds the action expert later on. At inference time, the model discards the discrete FAST tokens and uses the continuous tokens from the action expert to control the robot. PI has also developed a sophisticated scheme to smooth the transition between action chunks so that the robot doesn’t lurch in a different direction at the start of a new action chunk.
Yes, you could train an LLM to output decimal values, but that would require more tokens, exacerbating the problem of too-slow inference.
As far as I can tell, “world model” doesn’t have a widely accepted definition. It has become a catch-all term to describe alternatives to conventional VLM and VLA models.
Thanks for the great explainer. It brings me back to reading the SayCan paper when it came out in 2022. I remember trying to tell my coworkers that maybe transformer models would start doing more stuff in the physical world. Wild that it was only one undergrad degree ago! https://say-can.github.io/
It's great to understand how this all works. And particularly in the context of Kai's article yesterday about the wide range of difficulties robotics faces.