It’s day three of Robot Week! You can click here to get 25% off an annual subscription.
Most people first heard about large language models after OpenAI introduced ChatGPT in 2022. But in the minds of many AI researchers, the key breakthrough came two years earlier with the release of GPT-3.
With 175 billion parameters, the OpenAI model was more than 100 times larger than its predecessor, GPT-2. It was trained on a massive 300 billion tokens. And as a result, it generalized far better than previous models. For the first time, a single model could perform a wide variety of tasks — from translating between languages to answering trivia questions — without task-specific training.
It took OpenAI two more years to develop the techniques that transformed this raw “base model” into a user-friendly chatbot like ChatGPT. And then it took a couple more years to develop the techniques — like long-context reasoning, tool use, and context management — that transformed those early chatbots into the powerful agents we have today.
In short, there was a long road from GPT-3 in 2020 to Claude Code in 2025. But for those who knew where to look, the potential of LLMs was already clear in 2020.
The robotics world is now traveling a similar path. Its “GPT-3 moment” came in July 2023, when Google announced a model called RT-2. To create it, Google trained a multimodal LLM to directly generate robot actions. RT-2 wasn’t Google’s first transformer-based robotics model — the company released a predecessor called RT-1 a few months earlier, for example — but RT-2 was massively larger than earlier models. RT-1 had 35 million parameters. The RT-2 models had billions of parameters.
And as with GPT-3, size mattered. The RT-2 team reported its model showed “significant improvements to generalization over objects, scenes, and instructions.” They added that the new model exhibited “a breadth of emergent capabilities inherited from web-scale vision-language pretraining.”
For example, researchers placed a can of Coca-Cola on a counter alongside framed photos of Snoop Dogg, Tom Cruise, and Taylor Swift. They then prompted the robot to “move coke can to Taylor Swift.” The robot grabbed the can and moved it toward Swift’s photo.
At the time, Karol Hausman was a member of the RT-2 team. In a March interview, he described this as a moment of “huge, huge excitement” because “the robot models had never had any of Taylor Swift in their data. It had to understand the concept of Taylor Swift, connect it to the image of Taylor Swift, and then connect it to the right motion that would move the Coke can to the picture of Taylor Swift, all from Internet data.”
“That was the moment where it clicked for us that it could actually work — where you could bring in a lot of prior knowledge from LLMs, from the Internet, and connect it to robot motions,” Hausman said.
Google dubbed RT-2 a vision-language-action (VLA) model. Both Google’s approach and the term VLA quickly became industry standards. But as impressive as RT-2 was, it also had significant shortcomings — shortcomings the industry has been working to remedy over the last three years.
The RT-2 breakthrough kicked off a robotics boom that’s been underway ever since. Big companies in both the US and China have poured resources into robotics. Numerous robot startups have been created, and several have raised hundreds of millions of dollars in venture capital. And the models powering most of these robots are based on the basic architecture Google pioneered back in 2023.
The origins of RT-2, the first VLA model
Google invented the transformer in 2017 and had been experimenting with large language models ever since. The company had also been working on robotics for many years. So combining LLMs and robots was an obvious research direction.
In March 2023, Google announced PaLM-E, a 12-billion-parameter model that was optimized for robotics (the “E” stood for “embodied”). PaLM-E was a vision-language model (VLM) — meaning an LLM trained to understand images as well as text. It had been trained to generate natural-language robot commands like “move the blue block to the left.”
But PaLM-E couldn’t control a robot directly. Google’s robots didn’t have enough onboard computing power to run a VLM as large as PaLM-E. So PaLM-E ran in the cloud, and it was designed to work with a second, smaller model that would run on the robot. This second model would translate PaLM-E’s English instructions into low-level robot commands.
The RT-2 team’s plan was simple: delete the smaller model and instead train PaLM-E to directly control the robot.1 RT-2 — like PaLM-E — was too big to run directly on a robot. So the team ran the model in a Google data center and had it send commands to the robot over the network.
Like any LLM, RT-2 worked by prompting. Google would send RT-2 a prompt like “What action should the robot take to move coke can to Taylor Swift?” along with an image from the robot’s camera.
RT-2 would respond with a sequence of numbers like “1 128 91 241 5 101 127 217.” The robot would interpret this as a command to move the robot’s gripper to certain x-y-z coordinates (like x=128, y=91, and z=241), rotate the gripper to a certain angle (roll=5, yaw=101, pitch=127), and open (or close) the gripper to a certain position (217).
Then RT-2 would get the same prompt again, but with a fresh image. The model would generate another sequence of numbers representing a new target position for the robot arm. The robot would move its arm another few inches. Then the whole cycle would repeat again. It might take dozens of iterations to complete a task like “move coke can to Taylor Swift.”
To transform PaLM-E into RT-2, Google had to teach the model how to generate low-level robot instructions. That required a different kind of training data.
To collect that data, Google built three test kitchens and purchased 13 robots. Over the course of 17 months, human workers teleoperated the robots as they performed tasks — picking up objects, opening drawers, placing objects in the drawers, and so forth — more than 130,000 times.
Training PaLM-E on this data gave RT-2 surprisingly broad capabilities. Robots could manipulate objects they hadn’t seen before. They could operate in new kitchens. And they could complete tasks on counters that were cluttered with “distractor objects” that weren’t needed for the assigned task.
Five roboticists left Google to co-found Physical Intelligence
Karol Hausman was excited by the RT-2 breakthrough, but he also concluded that Google wasn’t the right place to develop the technology.
“It became clear that the way to accomplish this is to create an organization whose sole purpose is to solve physical intelligence,” Hausman said in March. “It can’t be solved as priority number 20 in another organization.”
So Hausman became the CEO of a startup called Physical Intelligence. He was joined by four other members of Google’s RT-2 team and two others from outside Google.
According to Hausman, the team sought out “investors that are fully aligned with this starting as a research company and not being oriented around short-term revenue.”
“If we do this right, this is going to completely change the world and it’s going to be the most valuable business of all time,” Hausman said. “But you need to have the patience to let us do it the right way.”
There was a lot to do. RT-2 was a big improvement over previous robotic models, but it was still far less capable than the average human. Over the last two years, the Physical Intelligence (PI) team has been working hard to close that gap. The company has been remarkably transparent, publishing at least 10 papers describing their work. For this story, I read all the PI papers I could find — along with 20 more from other companies and academic labs.
I’ll use PI’s research as a lens to explain the evolution of VLA models over the last three years. During that time period, VLA-controlled robots achieved much better fine motor control. They gained the ability to perform complex tasks that take several minutes. And companies are exploring new ways to have models reason using images as well as text — which could unlock the ability to learn from videos of humans performing tasks.
At the end, I’ll discuss the view that VLA models are on the verge of being eclipsed by a new architecture called world models. PI co-founder Sergey Levine has a perspective on this that I find pretty persuasive.
Robot Week special: Click here to get 25% off an annual subscription.
Improving robots’ fine motor skills
Hausman was impressed that RT-2 was able to move a Coke can to Taylor Swift. But later in the same interview, he described it as “totally unimpressive” and a “pretty pathetic demonstration of what robots could do.” That sounds like a contradiction, but you can see what he meant if you watch the video:
Facts Only
* OpenAI released GPT-3 in 2020 with 175 billion parameters.
* GPT-3 was trained on 300 billion tokens.
* Google announced the RT-2 model in July 2023.
* RT-1, a predecessor to RT-2, had 35 million parameters.
* RT-2 models contain billions of parameters.
* Google announced PaLM-E, a 12-billion-parameter vision-language model, in March 2023.
* Google utilized three test kitchens and 13 robots to collect 130,000 teleoperated task examples over 17 months.
* Karol Hausman, a former member of the RT-2 team, co-founded a startup called Physical Intelligence.
* Physical Intelligence has published at least 10 research papers.
* RT-2 operates by receiving image prompts and outputting sequences of numbers representing x-y-z coordinates, rotation angles, and gripper positions.
Executive Summary
The robotics field is currently experiencing a scaling trajectory similar to that of large language models (LLMs). The introduction of Google's RT-2 in 2023 marked a shift toward Vision-Language-Action (VLA) models, where massive parameter increases allowed robots to generalize tasks using web-scale pretraining. Unlike previous iterations that required a secondary model to translate high-level commands into physical movement, VLA models can directly generate the numerical coordinates and rotations required for robot action.
Despite these breakthroughs, early VLA models exhibit significant limitations in fine motor control and efficiency, often requiring dozens of iterative cycles to complete simple tasks. This gap has led to a surge in venture capital and the emergence of specialized startups, such as Physical Intelligence, which aim to prioritize "physical intelligence" as a primary research goal rather than a secondary corporate objective. The industry is now moving toward improving motor precision, extending task duration, and exploring world models that can learn from human video data.
Full Take
The strongest version of this narrative is that robotics has found its "scaling law" moment. By shifting from task-specific programming to VLA architectures, the field is attempting to import the emergent reasoning of LLMs into the physical world, moving from rigid automation to flexible, generalized agency.
The narrative relies heavily on the "GPT-3 analogy," a load-bearing pattern that frames the current state of robotics as an inevitable progression toward a predetermined destination of "intelligence." This creates a teleological glide path where the reader is encouraged to believe that because LLMs scaled, VLA models must also scale to reach human-level competence. There is a tension between the "huge excitement" of the RT-2 breakthrough and the subsequent admission that the demonstrations were "pathetic," suggesting a gap between symbolic victory (the robot knows who Taylor Swift is) and functional utility (the robot cannot move with precision).
This paradigm assumes that "intelligence" is primarily a matter of data scale and parameter count, potentially overlooking the unique constraints of physics—such as latency, friction, and real-time sensory feedback—that do not exist in text-based tokens. If this trajectory holds, the benefit accrues to those owning the massive compute and proprietary datasets required for VLA training, potentially centralizing physical agency in the same hands that control digital information.
Patterns detected: none
What physical constraints might prevent robotics from following the exact scaling curve of LLMs? If "world models" eclipse VLA architectures, what does that imply about the nature of physical learning versus linguistic prediction?
Counterstrike Scan: A coordinated campaign would use the "inevitability" frame to inflate venture capital valuations by overpromising the timeline to general-purpose humanoid labor. This content provides technical context and acknowledges failures, remaining clean of that pattern.