Why Now
Since ChatGPT’s release in 2022, language models have hit an inflection point. A model, pretrained on the internet, can plan and execute a complex multi-step task with a simple language prompt. The recipe behind it is now familiar: pre-train a large model on the internet then adapt it to a task with some amount of fine-tuning. The explosion of adoption and subsequent advancements have shown the vast potential of these models. Language models have had their “GPT moment” and it feels abundantly clear that the natural progression after language is the physical world. General physical AI feels achievable within our lifetimes.
In many ways, robotics has followed the same recipe: start with a pretrained vision-language model, add an action controller, train on robot demonstrations and internet data. Adapt these policies to specific robots and tasks with fine-tuning. Other approaches learn from video or use demonstrations directly at inference time. Over the past few weeks, several releases have made this range of approaches especially interesting:
Skild’s S1 and Generalist’s GEN-1.5 demonstrate learning from a single example supplied at inference time, without updating model weights. Both suggest a shift towards in-context learning by specifying example tasks rather than simply more fine-tuning.
Rhoda’s September 10 research release shows scaling pretraining on general web video (eg. from humans) can improve downstream robot performance, where the majority of robotics models have pretrained on robotics data.
Within days of GPT-6 Astra’s release, demonstrations of the model controlling robots suggested surprising physical capabilities: physical in-context learning with cups, a pick-and-place task, Blender-based arm movement, mobile manipulation and painting the Golden Gate Bridge.
While the field feels close to an inflection point, the route to useful general robotics remains unclear:
Data. There isn’t an internet of robot actions; most training data is collected by hand for a specific physical form.
Cross-embodiment. Skills learned from one body do not necessarily transfer to another — from human hands to robot grippers, or between different robots.
Latency. Slow inference limits how quickly a robot can respond to new observations, even when motor commands execute smoothly.
Generalization. Reliability can fall when objects, layouts, lighting, or scenes differ from training data.
Long-horizon tasks. Long horizon, sequenced, multi-step tasks remain difficult.
Physics. Contact, friction, deformable objects, occlusion, pressure all make execution complicated.
Recovery. Robots need to detect failed actions and change their approach; adjusting to failures on the fly is a challenge.
Evaluation. Limited testing in the real world, imperfect simulation, and varied testing environments make benchmarking difficult.
To explore these challenges firsthand, I bought a Hugging Face SO-101 and tested three robot policies on the same pick-and-place task. I then tried a fourth approach: a general language model controlling the arm, with no demonstrations or fine-tuning on my task.
ACT, a task-specific imitation model
SmolVLA, a small vision-language-action (VLA) model trained on open-source data for a closely related arm
π0.5, a large general VLA model from 2025
GPT-6 Astra, no task-specific fine-tuning
The three policies
ACT (Zhao et al., 2023) is a transformer that learns to reproduce human demonstrations through behavioral cloning. ACT predicts a sequence of future actions —a technique called action chunking — rather than the next action one at a time. Action chunking reduced compounding policy errors and made imitation learning precise enough to work on a real arm. ACT trains from scratch per task with no language instruction; I trained on 100 teleop demonstrations for my task.
SmolVLA (Hugging Face, 2025) is a VLA model: a vision-language model reads the cameras + a natural language instruction and an action expert turns its output into a chunk of motor commands. SmolVLA was pretrained on Hugging Face community datasets recorded on the SO-100 (the predecessor of my arm) then fine-tuned on 100 teleop demonstrations.
π0.5 (Physical Intelligence, 2025) is a larger VLA that combines a PaliGemma vision-language backbone with an action expert. Its training mixes robot demonstrations, web data, and high-level task information. In the published system, this model predicts a semantic subtask before generating actions to execute it. I fine-tuned the whole model on the same 100 teleop demonstrations.
All three policies predict action chunks: sequences of future motor commands. In my synchronous setup, the arm executes a chunk, then pauses while the policy processes another observation. Real-Time Chunking (RTC) (Physical Intelligence, 2025) instead generates the next chunk while the current one is executing. It conditions the new prediction on actions already committed to execution, aiming to preserve continuity while incorporating new observations. I also test whether RTC improves performance in my setup.
Setup and Method
Hardware. Hugging Face SO-101 follower and leader arm. Two cameras, one overhead and one on the wrist, both 640×480 at 30 fps. A mat with a bowl flanking each side, one red block and one blue block. “Left” and “right” refer to the operator’s viewpoint throughout, including the bowl labels in the figures.
Figure 1. The setup — follower arm center, leader arm right, wrist and overhead cameras, 20 numbered positions, red and blue blocks placed into bowls flanking left and right of marked area.
Task and data. Two instructions: “put the red block in the left bowl” and “put the blue block in the right bowl.” I recorded 50 demonstrations of each via teleoperation, with the block placed at random positions on the mat. In a minority of cases (~10%), I deliberately missed the grasp to demonstrate retry and recovery. The 100 demonstrations (50 red to left, 50 blue to right) span 29 minutes of arm motion and took about two hours to record.
Video 1. 50 sampled teleop demonstrations at 10x speed. Overhead camera left, wrist camera right.
Training. For the baseline comparison, ACT, SmolVLA, and π0.5 used the same 100 demonstrations with different training settings:
ACT: 100k steps, batch size 8
SmolVLA: 20k steps, batch size 64, with vision-language backbone frozen
π0.5: 30k steps, batch size 32, full model fine-tuned
Each gradient step processes a batch of training examples, pairing observations with target action sequences. 100 demonstrations contain approximately 54,000 frames, dividing steps × batch size by 54,000 corresponds to roughly 15, 24, 18 passes through our teleop dataset, respectively. These are approximate measures of data reuse.
Training ran on rented Modal GPUs: about $5 for ACT, $6 for SmolVLA, $100 for π0.5, excluding data collection and evaluation.
Evaluation. I marked 20 starting positions on the mat and used the same positions, block colors, orientations, and trial order across the compared conditions. The main comparisons alternated red-left and blue-right tasks, with a budget of 30 seconds of arm motion per trial, excluding inference pauses.
A trial counted as successful if the block ended in the instructed bowl within the time budget. A dropped block could still count as a success if the robot recovered and completed the placement. Unsuccessful trials were categorized by the furthest stage achieved: no contact with the block, contact without grip, or a grip without successful placement.
I report 95% Wilson confidence intervals for success rates. Most comparisons contain 20 trials; the unseen task pair eval contains 10 seen/10 unseen trials. Large confidence intervals reflect the limited evaluation count, but do not capture variation across independent training runs.
Comparison. I explore 6 questions:
Model family: ACT, SmolVLA, π0.5, Astra. Does the underlying model impact task success rate?
Amount of fine-tuning data: SmolVLA on 10, 25, 50 and 100 demonstrations. Does the amount of fine-tuning data impact success rate?
Untrained instructions: the models trained on one task pair (red block to left bowl, blue to right). Evaluate on the swapped pair (red to right, blue to left), can models generalize language instructions to unseen tasks?
Camera ablation: SmolVLA with both cameras, the overhead only, and the wrist only.
Execution method: SmolVLA with and without RTC.
LLM extension: a general language model, Astra, driving the arm through camera frames and target joint coordinates, given no demonstrations and no fine-tuning on my arm or task.
Results
Figure 2. Pick-and-place success rate across 20 trials on trained task pair, with 95% intervals. Robotics models (blue) were trained on 100 teleop demonstrations. Language model (orange) was not fine-tuned.
All systems received 30 seconds of arm-motion time, excluding inference pauses. Astra used a custom interface that converted target gripper positions into joint movements via an inverse kinematics library, and also allowed direct joint commands in its second run. The results above compare these systems as configured; they do not isolate model architecture or penalize for inference latency.
1. Model family
Video 2. π0.5's 20 trained-task-pair evaluations at 10x speed labeled with outcomes.
SmolVLA 9 / 20, π0.5 7 / 20, ACT 4 / 20, Astra 17 / 20 (Figure 2). The trained robotics models struggle in the same way: the arm reaches the block, the jaws land a few millimeters off or close a moment early, and the fine motor control needed to pick up the block is where many of these attempts fail. π0.5 touched the block on 19 of 20 trials and converted only 7 touches into grips. SmolVLA converted 10 of 17 touches into grips.
Takeaway. π0.5 performs similarly to SmolVLA (7 against 9 of 20, intervals overlapping). Both score above ACT. Small sample size leaves absolute performance uncertain. Plausible explanations for why SmolVLA performed better include embodiment (its vision-language backbone was pretrained on a closely related arm and camera setup) or camera resolution (π0.5 sees each camera at 224×224 pixels, SmolVLA at 512×512).
Follow-ups. To test sensitivity to visual detail, I’d compare SmolVLA’s images with the same images downsampled then resized back to the expected input size. To test embodiment, I’d compare both models on a different arm family.
2. Amount of fine-tuning data
Figure 3. SmolVLA fine-tuned on 10, 25, 50, and 100 total teleop demonstrations. Evaluated on the trained task pair (n=20) plotted with my predictions before evaluation.
Figure 4. Trial outcomes by starting position for SmolVLA trained on 10, 25, 50, and 100 demonstrations. Each panel contains one trial at each of 20 positions. Color between positions is smoothed for visualization and does not represent a measured local success probability.
Successes rose from 0/20 to 1/20, 8/20, and 9/20 as the dataset increased from 10 to 25, 50, and 100 demonstrations. My predictions before evaluation were 0, 1, 4, and 8 successes respectively (Figure 3). The largest observed step change was between 25 and 50 trained demonstrations.
Takeaway. Larger demonstration sets performed better under this fixed-step training recipe. Broader coverage of positions and approaches is a plausible explanation.
Follow-ups. Repeat the sweep with additional training runs and more granular dataset subsets to see whether the jump between 25 and 50 demos persists. Test sensitivity to data coverage and training steps.
3. Trained vs untrained instructions
For each model, I ran a separate test of 20 trials: 10 on the trained pairings and 10 on the swapped pairings. SmolVLA scored 7/10 on the trained pairings and 3/10 on the swapped ones; π0.5 scored 2/10 and 3/10. The swapped pairings were absent from my fine-tuning dataset.
Takeaway. Both models completed 3/10 trials on block-bowl pairings absent from my fine-tuning data. SmolVLA scored higher on the trained pairings; π0.5 did not show a trained-instruction advantage in this small sample.
Follow-ups. Fine-tuning on 25 demonstrations of each task while fixing total demos at 100 tests whether broader pretraining coverage improves performance.
4. Camera ablation
Figure 5. Trial outcomes by starting position for the three SmolVLA camera configurations. Each panel contains one trial at each of 20 positions. Color between positions is smoothed for visualization and does not represent a measured local success probability.
Wrist-only control scored 12/20, both cameras 9/20, and overhead-only control 4/20 (Figure 5). With the overhead view alone, I observed the arm approaching the block but struggling with final grasp alignment. The wrist-only policy sometimes continued moving around the mat after a successful placement rather than returning to rest. That could reflect difficulty recognizing completion or some kind of task-state confusion.
Takeaway. In this setup, wrist-only control achieved the highest observed success rate. The overhead-only model struggled with grasp alignment; adding the overhead view did not improve the point estimate over wrist-only control.
Follow-ups. Repeat the ablation with higher complexity, state-dependent tasks (eg block stacking where sequence matters) where an overhead view may provide more useful information about the overall state.
5. Real-Time Chunking (RTC)
Real-Time Chunking generated continuous motion but a lower success rate. Twice the arm gripped the block and released it. My guess is that small grasping errors made the actual outcome diverge from the policy’s prediction. While RTC conditions each new chunk on actions already committed to execution, those actions may continue on an approach that now needs correcting. Near the block, this could create a cycle of overshooting and undershooting, producing the back-and-forth motions observed. This suggests some trade-off between motion continuity and reacting to unexpected outcomes.
Takeaway. RTC reduced inference pauses but scored 3/20, compared with 9/20 for synchronous execution. I observed dropped grasps and back-and-forth corrections. I haven’t isolated whether these arose from timing, RTC settings, implementation details, or policy errors.
Follow-ups. Retry RTC with higher fidelity models or different implementation settings.
6. Where models fail
Figure 6. Categorical model performance scored by how far each model got: no contact with block (0), contact without grip (1), grip without successful placement (2), success (3).
Per-run heatmaps for every model are in the github repo.
Pooling five SmolVLA evaluations — the 25-, 50- and 100-demonstration models and the two single-camera variations — the ten central positions succeed 50% of the time, compared to 18% along the outer edges. Uneven demonstration coverage and differences in reach geometry are possible explanations.
7. A general language model with no fine-tuning
Figure 7. Trial outcomes by starting position for SmolVLA, π0.5, and the two Astra configurations. Each panel contains one trial at each of 20 positions. Color between positions is smoothed for visualization and does not represent a measured local success probability.
Video 3. Sampled trials with Astra’s justification for each move at 10x speed, including inference pauses. All recordings are saved in the repo.
At each decision point, Astra received an overhead image, a wrist image, and information about the arm’s pose. It could request a target gripper pose, which the inverse kinematics library converted into joint angles. The prompt explained the coordinate system, reach limits, orientation conventions, and gripper commands1. After each move, Astra received fresh images from both cameras and feedback on where the gripper actually ended up, how far its jaws were open, and whether the requested movement was blocked or unreachable. It then uses this information to adjust its next attempt.
In the first run my interface let it set position and finger pitch but not wrist roll, so the jaws always opened along the line from the base — this scored 12 out of 20. Several times, it moved the arm aside to get an unobstructed overhead view before re-planning, which seemed indicative of advanced reasoning behavior not observed in VLA trials.
For the second run I gave Astra access to wrist roll and a second tool that sets the six joints directly. I amended the prompt to specifically hint “match the [wrist] roll to how the block is turned in the image”. This configuration scored 17/20. As in the first run, Astra sometimes moved out of the way to reassess before taking another observation. In trials 3, 5, 11, and 13, it also rotated the wrist by 90 degrees after a failed approach to try a different grasp orientation. This strategy change was not observed in VLAs.
The comparison has several important differences:
Astra received no demonstrations and no fine-tuning on my task.
Its visual input consisted of a pair of snapshots at each decision point, rather than a continuous video stream. It could inspect fresh images after each tool action, but it could not revise that action from new images while the movement was underway.
It requested target poses or joint configurations; a separate controller generated the intervening motor commands. The robot policies instead predicted sequences of motor commands.
Astra spent roughly 100 seconds deliberating per trial. That time was excluded from the 30-second arm-motion budget, as were the robot policies’ inference pauses.
The systems differed in pretraining, control interfaces, and inference compute, so the results do not isolate the effect of any one of these factors.
I was mind blown to see a general language model control the gripper this precisely using repeated image snapshots and a tool interface. It feels like some of this ability likely comes from robotics-related pretraining, broader visual and spatial knowledge, or the detail/structure provided by my manipulation interface; many on X seem to think Astra was pretrained on robotics data.
Takeaway. Without demonstrations or fine-tuning on my task, Astra’s second configuration achieved the highest observed success rate: 17/20. Its behavior included retries, changes in grasp orientation, and movements that improved camera visibility. Those observations suggest useful capabilities for recovery, but the experiment does not isolate their contribution from perception, control-interface differences, or the additional inference time.
Follow-ups. I would compare success under equal wall-clock budgets, then validate whether a slower reasoning model can guide a faster controller. When does deliberate replanning improve outcomes enough to justify its latency, what decisions must remain in a fast feedback loop.
Other thoughts
Reducing latency is delicate. Pausing the arm during inference reduces changes caused by the robot’s own motion, but it does not guarantee that the scene stays unchanged. During asynchronous execution, the controller must account for actions taken since the observation was captured. My RTC result shows that reducing inference pauses alone did not guarantee better task performance.
Recovery matters beyond reaching the object. SmolVLA and π0.5 touched the block in 17 and 19 of 20 trials, compared with Astra’s 20, yet Astra completed substantially more placements. Our trials suggest that responding effectively to an unsuccessful grasp deserves as much attention as the initial approach.
Recovery demonstrations may help. SmolVLA and π0.5 each recovered from a missed grasp twice. About 10% of my demonstrations deliberately included a missed grasp followed by a retry, which likely helped.
Evaluation is expensive. I ran more than 200 trials by hand, with 20 trials for most comparisons and 10 per condition in the instruction test. Simulation benchmarks exist, but we will not always have a simulator validated against our exact arm, camera setup, and task. Establishing whether a policy worked reliably still required hours at the physical robot.
Binary success rates hide useful information. The 25-demonstration SmolVLA policy completed only 1/20 trials, but it made contact with the block in half of them. Scoring intermediate outcomes helps distinguish an approach failure from a grasp or placement failure, even when the final success rate is low.
Embodiment remains an open question. SmolVLA’s pretraining on the closely related SO-100 makes prior experience with similar hardware a plausible contributor to its performance. A useful extension is how much adaptation data each policy needs when moved to a less familiar robot.
Next steps
These experiments sample several approaches to robot learning: task-specific imitation, pretrained vision-language-action policies, and a general language model operating through tools. Recent in-context-learning demonstrations offer another route, and reinforcement learning uses rewards from experience to improve behavior—including dynamic skills that can be difficult to teach through teleoperation (eg. responding to external stimuli, how to not trip, etc).
My expectation is that deliberate reasoning and fast control will become more tightly integrated. The published π0.5 system already combines semantic subtask prediction with action generation, while Astra’s behavior illustrates high level reasoning and planning as integral to accurate, high-success execution. The question is how to allocate responsibility between planning vs action, preserve responsiveness, and allow recovery when the initial plan fails.
I’d like to spend time thinking about: cross-embodiment; mental frameworks for the progression of robotics models; sim-prod diffs; robotics reinforcement learning; and the various applications of physical AI.
Code, the evaluation protocol, every trial's outcome, and trial videos are in a GitHub repo: github.com/timzwu/robotics.
Thanks to Claude and ChatGPT for editing, graphics, and research help.
How the language model drove the arm. Each turn the model (Astra, reasoning effort high) received the overhead frame, the wrist frame, one line of text with the fingertip position and orientation, and had to answer with a tool call: move_to(x, y, z, pitch_deg, roll_deg, gripper), an absolute fingertip pose; move_joints(...), the six joint angles directly; or done. Each pose went through LeRobot’s inverse-kinematics solver (the placo library underneath) on the SO-101 model; a pose the solver could not reach within 1.5 cm, or that would put the fingers below the mat, was refused without moving. The arm walked to the solved joint angles at 30 Hz, at most 3° per tick, with the commanded position never more than 15° ahead of the real arm; a move ended when the arm arrived, stopped moving for half a second, or after 8 s. The model then got the achieved pose, whether the move was blocked, whether the jaws had closed on something, the motion time left, and fresh frames. A trial ended at 30 s of arm motion, same as every other pass, or when the model called done. Thinking time did not count and there was no limit on calls.
The full system prompt used in Run 2:
You are controlling a small 5-axis robot arm (SO-101) with a parallel-jaw gripper, on a table. Task: put the [blue/red] block in the [left/right] bowl
Coordinates: meters in the arm's base frame. +x points forward across the table away from the base, +y points to the arm's own left, +z is up. The pose you command is the point between the fingertips. z = 0 is the table surface: with the fingers pointing straight down (pitch 90), fingertips on the mat means z is about 0.00 to 0.01. The arm cannot go below z = -0.01. Reachable: x from 0.12 to 0.35, y from -0.25 to 0.25, z up to 0.25; with the fingers straight down the arm reaches up to about z 0.13. The arm starts at its rest pose, x 0.18, y 0.00, z 0.01, fingers pointing forward and down.
Orientation: `pitch_deg` is the finger direction below horizontal (0 = forward, 90 = straight down). `roll_deg` turns the jaws about the finger axis: at 0 the jaws open along the line from the base to the point, so they close on the near and far faces of a block; at 90 or -90 they open sideways and close on its left and right faces. Match the roll to how the block is turned in the image. `gripper` is the jaw opening: 0 = closed, 37 = fully open. There is no force sensor. After a close command the reported opening tells you what happened: the jaws stop near 6 when they meet nothing and near 12 when the block is between them (the result also reports `holding`).
Left and right in the task are from the OPERATOR's seat, facing the arm from the far side of the table: the operator's left is the arm's right (negative y), and the operator's right is the arm's left (positive y).
Two cameras each turn. The overhead camera sees the whole mat with the arm's base at the bottom edge of the image, so +x is toward the top of the image and +y (the arm's left) is toward the left of the image; the operator's "left" therefore appears on the right side of the image. The wrist camera looks out from between the fingers.
Each turn, call one tool: `move_to` with an absolute target pose (position, pitch, roll, gripper), or `move_joints` with the six joint angles directly (every joint of the arm is yours: base rotation, shoulder, elbow, wrist bend, wrist roll, gripper; the current angles are reported each turn), or `done` when the block is in the bowl or the task cannot be completed. The arm gets 30 seconds of motion in total; thinking time is not counted. The result of each move reports where the gripper actually ended up; if it differs from the target, the pose was out of reach or the arm was blocked, so adjust.