A robot folding a shirt is a surprisingly ambitious demonstration. It might be trained on thousands of hours of robot data, learning to connect camera observations with coordinated joint movements. It has to recognize the shirt’s shape, decide where to grasp it, and adjust as the fabric moves. A sleeve can slip, a fold can land unevenly, and at some point it becomes easier to start over rather than keep folding. What looks like a simple household chore brings together perception, planning, and precise physical control.
In the past month, language models have improved on the previous human-established bound on prime gaps, produced a computer-checked formalization of Fermat’s Last Theorem, and generated an apparent solution to the Navier-Stokes Millennium Prize problem. With the raw intelligence of frontier language models growing faster than ever, the natural question is when these capabilities will move into the physical world. It’s easy to dismiss AI as an abstract concept when it’s lines of code on a computer — much harder to dismiss when it’s washing dishes in your house and delivering your packages.
But how much of this intelligence transfers to a machine that has to see, move, and adapt to a changing environment? LLMs have the benefit of synthesizing structured inputs/outputs, via text in a chat box, while physical models need to handle a wide variety of inputs and unknown states. With the release of many highly capable zero- and one-shot robotics models1 in recent weeks, I’ve spent time thinking about physical AI: how far robot models have come, where we stand today, and what still separates a demonstration from real-world utility.
Over the past ~5 years, robotics has progressed by leaps and bounds. Progress came from collecting more data and training larger models (just like language), but also from finding ways to be more efficient with existing data. A change in how we encode actions, a better way to collect demonstrations, or a different way to prompt physical tasks can change the return on everything that follows.
I’ve found it helpful to map recent developments across five axes:
Learning signal: how it learns
Architecture: model structure and built-in knowledge
Data: what it learns from
Instruction: how we communicate a task
Execution: how actions unfold
This is not meant to be a complete history of robotics, but rather a formal exposition of how I’m making sense of the core advancements in physical AI.
How we got here
The simplest starting point is to explicitly program every movement you want a robot to perform: move here, close your hands, lift up, release here. Early industrial robots like the Unimate worked precisely this way. Engineers later built systems that could adapt to their surroundings: Boston Dynamics’ Atlas combined planned movements with live perception and a physics-based controller, resulting in impressive backflips and parkour, but engineers still had to program the movement and control sequences behind them.
Hardcoding actions and coordinates is pretty cumbersome. Instead, we can teach robots to mimic human actions. With behavioral cloning, we can train a policy to reproduce demonstrated behavior: given what the cameras and robot sensors observed, repeat what the demonstrator did. In ACT/ALOHA, researchers teleoperate a pair of arms through tasks like inserting a battery, and the policy learns from the recorded camera views and joint movements. This feels natural, similar to demonstrating a new task to a child, and remains one of the primary methods used to train robot models today. However, this introduces two problems: 1) it’s difficult to demonstrate a counterexample (eg. how to not crash a car); 2) if the robot makes a small mistake, it struggles to recover from a situation that never appeared in its demonstrations.
Over the past several years, several developments have advanced in parallel, bringing general physical AI closer to real-world use:
Figure 1: Timeline of papers discussed below
2022. General policy: Google’s RT-1 trained a transformer policy on a large collection of demonstrations across tasks, given input images and language instructions. This model uses a convolutional image encoder conditioned on the language instruction, compresses visual features into tokens, and passes these tokens through a transformer to predict discrete commands for the arm, gripper, and base. The important idea was that large, diverse datasets could accumulate into a shared policy; opening drawers, picking up objects, and moving things around don’t require three isolated models. RT-1 was an early step towards a real-robot, multi-task transformer at scale, showing data diversity and model capacity were useful levers for generalist policies.
2023. A low-cost manipulation recipe: ACT/ALOHA is a seminal paper in modern robotics, showing how much can be achieved (even in low-cost systems) with the right training recipe. ALOHA used an affordable leader-follower interface: a person moved smaller arms while the working arms mirrored them, collecting demonstration data. ACT’s transformer combines camera images with the robot’s joint positions to predict a chunk of future joint targets (actions), learning the demonstrated behavior. Predicting several subsequent actions together (action chunking) reduces compounding errors, while blending overlapping predictions (temporal ensembling) smooths execution. The important idea here is that the right mix of practical data collection, hardware, and a well-chosen imitation policy can produce impressive precision.
2023. Better representations of movement: Diffusion Policy, released shortly before ACT, addressed how to represent the actions themselves. There are often several valid ways to grasp an object — approach from the left, the right, or above. Imagine demonstrations going to either side of an obstacle, but the policy averages these paths and attempts to approach directly from the middle. Diffusion Policy learns a distribution of possible action sequences, with separate peaks corresponding to different valid ways to move. It repeatedly refines random noise into a sequence drawn from this distribution, allowing it to follow a coherent path around either side of the obstacle rather than averaging the two. The important idea is that a policy produces better behavior when it’s able to represent multiple valid paths.
2023. Bring web-scale knowledge into robotics: RT-2 connected robot actions to pretrained vision-language models (VLMs), leveraging models trained on internet-scale data. Its PaLI-X/PaLM-E variants process images and instructions to generate action tokens through the same output mechanism used to generate text2. By jointly training on robot and vision-language data, RT-2 brought broader semantic knowledge from VLMs into robot control, improving its understanding of instructions and object relationships beyond what appeared explicitly in demonstrations. Later that year, Open X-Embodiment (OpenX) attempted to tackle the generalist robot policy problem by pooling data across different tasks, robots, and environments using the RT-1 and RT-2 architectures. Together, these projects expanded the knowledge available to a robot: RT-2 brought in what models had learned from the internet, while OpenX proved experience from different robots can contribute to a shared policy.
2024. Make data collection and adaptation more accessible: Universal Manipulation Interface (UMI) used handheld grippers to collect demonstrations, no longer requiring a full teleoperated robot at the collection site. UMI combines a pretrained vision transformer with a diffusion-based action generator to turn camera observations into relative gripper poses and opening commands. This expanded the environments in which we could collect useful training data. That same year, OpenVLA was trained on the OpenX dataset with a vision backbone and a Llama2 language model, processing visual features and instructions to generate action tokens. OpenVLA helped formalize “start from a general model, fine-tune for my robot and task” as a reproducible workflow.
2024-25. Combine pretrained knowledge with continuous action generation: Physical Intelligence’s π0 connected two ideas: pretrained VLM knowledge and the continuous action sequences produced by methods like Diffusion Policy. Its VLM processes images and instructions, while a specialized action expert generates movement chunks through flow matching: starting with random action values, it repeatedly adjusts them into a coordinated movement sequence, guided by what the robot sees and is asked to do. Its contribution was combining web-scale knowledge with a continuous action generator suitable for dexterous tasks like folding laundry. π0.5 extended this with broader training data and subtask prediction, predicting low-level instructions (“pick up the plate”) that fulfill a higher-level direction (“load the dishwasher”). NVIDIA’s GR00T N1 similarly paired a VLM with a flow-matching action transformer, while SmolVLA used a compact VLM and action expert trained on open-source community data. Together, these models showed how a robot’s general knowledge and movement can improve together: richer pretraining helps with semantic understanding, while the action expert learns how to execute specific actions.
2025. RL from experience: Demonstrations show a robot what successful behavior looks like, but it should learn and improve from its own attempts. π*0.6 and RECAP combined demonstrations, autonomous experience, and human corrections to improve through reinforcement learning. This system pairs a vision-language backbone and flow-matching action expert with a value function that estimates future rewards, favoring actions that improve expected outcomes. A successful episode can contain actions that set the robot back, while a failed episode can contain useful actions that bring it close to success. Labeling the entire episode good or bad fails to capture this distinction; RECAP uses the value function to estimate which actions made better-than-expected progress at each step, accounting for both time to completion and a penalty for failure. This still required human corrections and supervision, but RECAP established a practical route to learn from real-world attempts.
2026. Scale with human experience: EgoScale pretrained on a large collection of action-labeled egocentric (recorded from the person’s perspective) human video, then adapted through small amounts of robot data to transfer human experiences to robot control. Its VLM feeds a flow-matching action transformer that predicts wrist-and-hand action chunks. The action labels make this a richer version of supervised learning than any arbitrary collection of online videos. The paper found increasing human data improved downstream robot performance, giving concrete support to the human-data argument: people perform an enormous amount of useful physical activity, and robot models may be able to acquire motion knowledge from it.3
2026. Richer data labels and more subtask guidance: π0.7 extended language-based subgoals with generated images of what intermediate steps could look like. A separate lightweight world model, based on BAGEL, uses the current camera views and subtask instruction to generate these images, giving the robot a visual target to work toward. π0.7 also trained on richly annotated data describing qualitative aspects of an attempt (speed, quality, whether a mistake was made). Its VLM, flow-matching action expert, and memory encoder (now fed a window of past frames) combine observations, subgoals, and qualitative context to generate action chunks. Subgoals help specify the next steps; richer labels help the model learn from varied experiences including failures or suboptimal attempts. The takeaway is that language and image subgoals make tasks easier to execute, while richer labels make experience more useful to learn from.
2023-26. Learn by predicting what comes next: RL is already used in robotics to train humanoid movement: ASAP learned whole-body movements in sim and refined using real-robot data; Boston Dynamics trained Atlas to lift a fridge across simulated variations in weight, grip, and position. Teleop alone is poorly suited to teaching every split-second adjustment or practicing counterexamples like how to “not fall”. Simulations allow robots to explore these situations repeatedly, with rewards to guide which behaviors improve. Today, much of this happens in meticulously constructed physics simulators; learned world models could give us another way to simulate actions and predict outcomes. DreamerV3 trained policies on imagined experience; Genie 3 generates interactive worlds in real time. As simulators get more physically accurate, they could make simulated training more useful for a wide range of real-world tasks.
Taken together, progress over the last 5 years has been substantial. Robots can learn delicate behaviors from low-cost equipment; reuse knowledge across different tasks, embodiments, and environments; improve predictably with more data; and improve through experience. The hard part is turning these successes into a general policy that performs reliably over long time horizons.
The five axes
Terminology can make the landscape seem more fragmented than it is to a newcomer. “Diffusion”, “reinforcement learning”, “egocentric video”, and “in-context learning” all describe different parts of a system. A single model could involve all four. The axes below were a way for me to reason from first principles about the design choices that go into the model recipes.
1. Learning Signal
This is the feedback that shapes what the model learns. For a robot putting a mug on a shelf, we might train it to match a demonstrated movement, maximize a reward for placing the mug upright, or predict what it will see after moving. Each supplies a different kind of guidance:
Imitation teaches the model to reproduce actions from demonstrations. This teaches you how to act but may not show how to recover when the mug slips.
Rewards/Preferences teach whether a behavior was desirable (eg. succeeding quickly), but not necessarily which actions contributed to that result.
Self-supervised prediction and representation learning teach patterns in the experience itself: what follows an action, which observations belong together, how video relates to language. This can supply useful information without labeling every movement, but does not by itself specify desired behavior.
This axis matters because this signal determines how our model learns and improves.
2. Architecture
Architecture describes how the model is organized, what knowledge it starts with, and how it represents observations and actions.
Inherited knowledge: a policy can learn from robot experience, reuse a pretrained visual encoder, build on an existing language or vision-language model, or adapt an image/video generator. RT-2 inherits semantic knowledge from a VLM; DreamZero relies on video prediction; these starting points affect what gaps must be filled with robot data.
Action representation/generation: actions can be output directly as continuous values (joint angles or gripper poses) or as discrete tokens encoding value ranges (bins) or compressed sequences4. Either representation can describe a single action or a chunk via direct prediction, token decoding, diffusion, or flow matching. Action representation and generation affect precision, the ability to represent valid movements, and computational cost.
Division of labor: a policy can map observations to actions directly, or work with a separate action planner, world model, or value function. In HiRobot, a high-level VLM supplies language subgoals to a lower-level VLA (a precursor to π0.5’s hierarchical approach).
Memory: a model can use the current observation, a short history, or persistent memory of earlier events. MEM combines recent visual observations with longer-term language memory, helping a robot remember what it’s already tried and what is no longer in view.
These choices determine what information the model can use and how it structurally approaches a task. Intuitively, the core architecture of a model is one of the largest levers we have5.
3. Data
Data describes the specific forms a model learns from, what information it contains, and what situations it covers. This is distinct from #1, which describes structurally how a policy learns but not the form of its inputs.
Teleoperation pairs observations with commands demonstrated on the robot itself.
Autonomous robot experience records the model’s own attempts, including failures and human interventions, as in RECAP.
Handheld grippers make video data collection more portable while capturing movements closer to what a robot gripper can execute.
Human recordings can pair egocentric video with motion, sensor, or text labels as in EgoScale. Ordinary internet video offers variety but lacks action commands or force measurements.
Simulated or generated experience supplies virtual practice, either through a physics simulator or world model. Usefulness depends on how accurately they reproduce reality.
There’s usually some tradeoff between how much experience we can collect and how directly it teaches the robot what to do. A video might show how to open a fridge without revealing the force needed to break its seal, or where to grip on the handle of a double-door variant. A thousand identical successful grasps may still leave a robot unprepared for its first missed grip. Scaling data is most useful when it adds something the model has not learned to handle.
4. Instruction
Instruction describes how we specify behavior at inference. Different formats resolve different ambiguities:
A fixed task or task identifier selects a dedicated policy or learned skill; no detailed prompt is needed
Language communicates an objective or constraint: “put the mug away” or “keep driving down this road”
A goal image or target state shows where something should go, or specifies a target robot pose
Spatial instructions supply points, waypoints, or a rough path, as in RT-Trajectory
Demonstrations in context show the desired behavior through video, images, or recorded observations and joint actions
Remember these formats aren’t mutually exclusive. They can be combined, or come from another model: π0.7 can receive language subgoals from a high-level policy and visual targets from a lightweight world model. A model might have useful skills but lack the context to assemble them into the task we want without appropriate instruction.6
The distinction between instruction and training is important. Fine-tuning changes a model’s weights; in-context learning supplies examples as input without changing them. Instruction determines how we direct existing capability to a novel task.
5. Execution
Execution describes how predictions translate into movement and how the robot responds as the world changes.
Prediction horizon: does it generate one action, a chunk of actions, or a longer sequence?
Commitment and feedback: how much of a predicted sequence does the robot execute before sampling fresh evidence? Every action, a short chunk, or the entire sequence?7
Scheduling: what happens while the model computes the next update? Does the robot wait for each prediction, or generate new actions while moving?8
Planning: does it use the policy’s output directly, or does it evaluate multiple paths before choosing an action?
If the mug slides halfway through a grip, the robot needs a fresh observation, an updated movement, and a smooth transition quickly enough to recover; execution describes how inference and movement are coordinated.
Mapping models to the axes
Figure 2. Interactive five-axis map of papers and company model releases, mapped onto the axes above.
One practical benefit of this map is that apparently contradictory claims become easier to interpret. A policy might be described as learning from human data without robot experience, while its underlying control layer learns in sim; these statements can both be true because they refer to different components. The complete recipe matters more than the descriptions of individual components.
What general pretraining should get us
For all the variety above, I keep coming back to one question: for a given general model, how much additional learning is required before it can reliably perform the task in front of us?
One way to think about this is an adaptation curve: how a model’s success rate changes as we give it more task-specific experience. The y-intercept tells us what it can already do; the slope shows how quickly it improves; the plateau shows how far data scaling gets us.
Ideally, generalist pretraining should improve the whole curve10. A model that has learned useful physical representations across a range of tasks should be more capable at applying those representations to new ones. It should require fewer demonstrations to adapt and be better at synthesizing information from fine-tuning.
There’s already evidence for this interaction. Physical Intelligence found that increasing the scale and diversity of pretraining (using robot data alone) improved human-to-robot transfer during fine-tuning. This suggests the model starts learning shared structure in what is happening (eg. grouping by task) rather than distinguishing who does it (eg. grouping by human vs robot). EgoScale found a scaling relationship between human pretraining data and validation loss. These results support the idea that better pretraining makes later learning more productive.
Figure 3. Human (yellow) and robot (green) features are largely separate with small-scale/no robot pretraining. With broad pretraining across tasks and embodiments — pretrained on robot data alone! — feature representations overlap much more closely when fine-tuned with human data. Xemb = cross-embodiment. Source: Physical Intelligence.
The dotted curve below represents this possibility: pretraining gives us both more capability out of the box and a better foundation to fill in the gaps.
Figure 4. Illustrative adaptation curves. The dotted line illustrates how generalist pretraining could improve both starting ability and the return on task-specific fine-tuning.
The question is where today’s models sit on this curve. Do stronger general models simply start higher, or do they also need fewer task-specific demonstrations to replicate a task? Does this hold when we change the robot, the environment, or aspects of the task itself? I’d love to see future studies vary both the scale of generalist pretraining and the amount of task-specific fine-tuning to measure how much additional experience each model needs to reach the same level of performance.
After all, this is the practical promise of a general model: experience acquired elsewhere reduces the amount of effort required for a new task. This is the kind of compounding benefit we need as robot models scale.
Scaling the right things
A tempting conclusion is that robotics simply needs the equivalent of the internet: enough examples, enough compute, and eventually the capabilities follow. There’s clearly a strong case for this: language models benefited from learning at an enormous scale, and robotics simply has far less usable experience to learn from.
But the history of language modeling also shows why the recipe matters. Statistical n-gram models started by predicting words from short sequences of preceding words. Long contexts created a combinatorial coverage problem: there were too many possible sequences to observe them all. Neural language models established shared representations, allowing related words and contexts to inform one another. Recurrent language models passed a hidden state through a sequence. LSTM helped stabilize memory across long gaps. These were meaningful structural changes before transformers entered the picture.
In machine translation, attention let the model look back at the most relevant parts of the original sentence instead of relying on a single compressed summary. The Transformer then removed recurrence, enabling parallel computation across multiple sequence positions during training. All of these earlier approaches benefited from more data and compute too — they were just less efficient at turning data/compute into better predictions.
The lesson I take from this history is that scaling and structural improvement are complementary. We find a promising recipe, scale it, discover a limitation, and revise that recipe. The marginal FLOP will probably ~always be useful, but it’s the recipe that determines how orders of magnitude more useful it is.
Robotics seems to be in a particularly active phase of this process. Everyone is tweaking and merging components of the recipe, trying to figure out what is worth scaling. My strongest intuition is that longer-horizon planning and fast physical control will benefit from this same division of labor11. Choosing which objects belong in a drawer and adjusting a gripper as it makes contact have different information and latency requirements. There’s something naturally intuitive about letting a broadly capable model reason about the job while a fast controller handles movement.
I’m less certain that the intermediate plan needs to be interpretable to humans. Text is convenient to inspect and explain, but “grasp gently” leaves a great deal unspecified. An image describes physical arrangement but says little about the force and dexterity needed to get there. Some abstract internal representation might connect planning and action more efficiently than either. But I’m also reminded of the Bitter Lesson; describing execution as System 1/2 thinking could be reflective of my biases — a policy is under no obligation to organize its thoughts for my benefit (obviously interpretability etc etc is important).
That’s one reason why I would judge these ideas by their adaptation curves. Does the next model require fewer demonstrations for an unfamiliar task? Does experience on one robot reduce the amount of work required on another? Can a system learn from its own mistakes and apply corrections forward? These are concrete signs a general model is becoming more general.
I’m optimistic about the scaling law in robotics. Much of the interesting work today is in discovering the pieces worth scaling: the signals, representations, data sources, interfaces, and execution methods to produce a robot that arrives at the next task with more of the learning already done.
Appendix: Mapping companies
Sources: 1. Physical Intelligence π0.7, 2. NVIDIA — GR00T N1, 3. NVIDIA DreamZero, 4. Sunday ACT-2, 5. Skild S1, 6. Generalist GEN-1.5, 7. Rhoda DVA, 8. Reward AI OM-1, 9. Figure Helix 2.5, 10. Google DeepMind Gemini Robotics 1.5 / Robotics-ER 1.5
Thanks to Claude and ChatGPT for editing, graphics, and research help.
Figure Helix 2.5, Reward AI OM-1, Skild S1, Generalist GEN1.5, Rhoda’s study on web-video pretraining.
RT-1 and RT-2 both generate actions as tokens, while ACT predicts continuous joint targets and Diffusion Policy predicts continuous actions as either joint targets or gripper poses. Concretely, a joint-angle command might be represented by token 42 (corresponding to an angle bin) or directly as 0.3 radians.
Another paper from PI similarly found fine-tuning with human data works better with larger pretrained generalist models trained on robot data; large models tend to group features more by task and less by embodiment, making human demonstrations more useful during fine-tuning.
In language, the Transformer replaced RNNs with attention, allowing parallel computation during training while improving translation quality. In vision, Vision Transformers showed processing images as sequences of patches could outperform strong convolutional networks when pretrained at scale. Architecture changes how effectively we can scale data and compute.
π0.7’s big breakthrough is its emergent compositional generalization: the ability to piece motions or action components together in a novel way to complete an unseen task.
With temporal ensembling, ACT predicts a full chunk at each step then weighted-averages predictions from overlapping chunks for the same timestep; predicting 50 actions does not require executing all 50 before incorporating a fresh observation.
Real-time action chunking (RTC) generates the next chunk while the current one executes, conditioning on actions committed to execute during the computation delay, so the new chunk joins the ongoing movement consistently.
I’m a big fan of graphs — ChatGPT helped map papers and recent company announcements to my five axes in a cool visual that connects models based on similar design choices.
See footnote 3.
This feels analogous to System 1/2 thinking, and the way many are looking to apply decision models like Jev to robotics.