When I first started thinking about robotics and training policies a week ago, I collected training data by teleoperating a leader arm while a second arm copied my motions. This produces examples for behavioral cloning: a model learns to reproduce the demonstrated actions from camera images and the robot’s joint positions. It is the simplest form of robotics imitation learning, built on a straightforward idea: show the robot how to perform a task, then train it to copy those actions in similar situations. Recording 100 demonstrations of a pick-and-place task took over two hours. A two-camera SmolVLA baseline completed the block-to-bowl task in 9 of 20 trials.
That experience made the data problem tangible. Each demonstration required someone to perform the task, reset the scene, and check the recording. Internet video offers a much wider database of physical activity, but usually lacks the corresponding robot commands and joint positions that a policy requires to learn to imitate. Compared with language models, which can learn directly from vast collections of internet text, robotics models have a much smaller amount of high-quality, paired observation-action data. Collecting useful examples across different robots, tasks, and environments is a substantial part of the work in building a general robotics model.
Simulation offers a way to generate those examples. A virtual robot can attempt a task, reset automatically, and record both camera images and motor commands simultaneously. Instead of a human performing tasks, multiple simulations can be run in parallel on a GPU. Object positions, lighting, and camera views can be varied deliberately. Research such as NVIDIA’s GR00T N1, which combines real robot trajectories, human video, and synthetic data, hints at automating part of this data collection process.
How much real data do you need: could sim-only fine-tuning work, or is combining real-plus-sim demos necessary?
How useful is sim data in training, and how much additional task-specific setup and validation is required?
Carrying over the pick-and-place task from my previous attempt, I recreated my tabletop in simulation and generated 100 scripted demonstrations. I then compared SmolVLA policies fine-tuned on:
Simulated data alone.
Simulated + real data, co-trained in the same fine-tuning run.
Real data alone (control).
After revising our simulation environment, fine-tuning on sim-only data completed 0/20 trials; co-trained completed 7/20, compared with the real-only baseline of 9/20. Inside the simulator, the policies again scored sim-only 2/20, co-trained 4/20, and real-only 0/20. The sim-trained policies were already struggling in simulation, so sim-to-real transfer alone can’t explain the low success rates.
Setup and method
Real-world setup
SO-101 arm, overhead and wrist cameras, and two pick-and-place tasks: red block into the left bowl, blue block into the right. Each evaluation used the same 20 marked starting positions and 30 seconds of arm motion budget, excluding inference pauses. Each model predicts a sequence of movements, the arm executes it, then the model receives a new observation.
The real-only success rate on the physical arm is pulled from the first experiment. The sim-only and co-trained policies are newly evaluated. I report 95% Wilson intervals for success rates; these reflect the limited evaluation sample size, not variation across training runs or evaluations.
Simulation setup
I used the SO-101 assets from NVIDIA’s sim-to-real workshop and built my scene in the Isaac Sim simulator. The virtual scene contained the arm, mat, bowls, blocks, and two cameras. A physics engine calculated motion and contact; a renderer generated the camera images.
Figure 1. Simulated and real overhead views, after calibration. Primarily to compare scene layout; blocks are at different positions.
Accurately recreating the environment required measurements of the bowls, four numbered mat positions, camera height, and block size. Replaying real demonstration commands on the simulated arm allows us to verify alignment: after calibration, the gripper came within roughly 0.4–1.6 cm of the block across three replays. This feels like a reasonable starting point for generating demonstrations, since additional troubleshooting below 1 cm feels marginal (though admittedly important for fine motor tasks like the one described here).
Video 1. Demonstration replayed in sim alongside the original overhead footage. Setup and motion look reasonably similar.
The wrist view also looked quite good, though aspects of the gripper’s appearance, texture, lighting, and framing didn’t match exactly.
Figure 2. Wrist-camera views at rest. Layout calibration leaves differences in appearance.
The simulation closely resembled my setup, but visual alignment alone could not validate a fine motor task like picking up a one-inch block. Camera appearance and contact physics still needed to be tested through the learned policy’s behavior.
Generating simulation demos
Scripted simulations were generated using the block’s exact position and rotation to plan each attempt.
The script followed a fixed sequence: hover over the block, descend, close, lift, move over the bowl, release, and return to rest. Inverse kinematics tools converted each desired gripper position into joint angles. Block color and destination bowl alternated. Nothing in this simulation script had to explicitly recognize a block in an image since block and bowl positions were known and the path was deterministic.
The recorder saved camera images, observed joint positions, commanded joint positions, and the task instruction in the same format as my real demonstrations. SmolVLA learned to predict the recorded joint commands from camera images, joint state, and the task instruction.
Video 2. First 10 simulated demonstrations at 10x speed, through overhead and wrist cameras. These are scripted demonstrations, not the learned SmolVLA policy.
Each training demo started with the block at a random position and rotation in the training zone. I varied camera position, focal length, lighting, and robot color. This is known as domain randomization: purposely varying visual (shadows, lighting, camera) or physical (mass, joint friction) attributes to generate varied simulation data for fine-tuning in hopes the policy will generalize better during deployment. I did not randomize the mat texture or block appearance beyond red and blue.
The simulation script kept an episode only if the block ended inside the bowl. It produced 100 accepted demonstrations from 107 attempts in 59 minutes, for about $2 in compute on an A10 GPU. Building, checking, and generating data in the simulator cost roughly $6 in total; fine-tuning the two new policies cost another $15. Evaluating all three policies in simulation added about $5. Clearly simulation, if successful, offers a cost-effective solution to the data problem.
Thematically, the simulated data had a few differences from my real-world teleop demos:
Accepted episodes lasted about 12 seconds on average (vs 18s), with direct, consistent movements.
Sim movements used narrower wrist-roll and gripper-opening ranges, reflecting the scripted approach.
Sim episodes contained no deliberate recovery examples while teleop demos included missed grasps followed by retries (about 10% of demonstrations).
Fine-tuning and evaluation in sim
I compared three policies fine-tuned on 1) 100 real demonstrations, 2) 100 simulated demonstrations, and 3) the combined 200 demonstrations. The real demos were the same 100 used in my previous article.
All three started from the same pretrained SmolVLA model and used 20,000 training steps with batches of 64. Holding the fine-tuning recipe fixed meant different amounts of data reuse: roughly 24 passes through the real dataset, 35 through the shorter simulated dataset, and 14 through the combined dataset. The co-trained policy therefore saw fewer repetitions of the real examples.
During simulated evaluation, all three policies controlled the simulated arm to grip at the same 20 sticker positions, with the block square to the arm in the nominal scene without domain randomization. Each used the same 50-action chunks and 30-second motion budget as the physical evaluation. Success was scored from the block’s position inside the bowl. Categorical success stages used proxies—block movement or fingertip proximity for contact, and a lift above 2.5 cm for a grip—so these are not identical to the human-scored stages on the physical arm.
Results
Task success
Figure 3. Success rates for fine-tuned SmolVLA policies in sim and in the real world, at 20 trials per eval. Error bars show 95% Wilson intervals.
The sim-only model completed no placements in either environment. The co-trained model scored below the real-only baseline on the physical arm, and only scored 1/20 even when evaluated in simulation.
Evaluation in the real world
Sim only: grasping at an offset to the block. The arm repeatedly reached, closed its gripper, lifted, and moved toward a bowl while the block remained on the mat. In 18 of 20 trials it never touched the block. In the other two, it touched but failed to grip.
Video 3. All 20 sim-only policy evals at 10x speed. Arm repeatedly attempts grasping motion at offset from real block.
In the real world, the model produced recognizable pick-and-place movements but poorly aligned them with the object. Even with rendered images directly from its training environment, the policy struggled to complete the task in the simulator.
Co-trained: reaches the block but struggles to grip. The co-trained model touched the block in 17/20 trials, matching the real-only baseline. Of 17 contacts, we saw 6 grips and 4 placements vs 10 grips and 9 placements for the real-only policy.
Video 4. All 20 co-trained policy evals at 10x speed. Small grasp offsets and unsuccessful retries account for most failures.
The co-trained policy repeatedly attempts to grip the block at a slight offset or incorrect wrist orientation, followed by another attempt at a similar offset. In two trials the gripper approached from directly above, resembling the scripted demonstrations.
Figure 4. Categorical success stages in real-world evals across fine-tuned SmolVLA policies.
The sim-only policy mostly failed before contact, while the co-trained policy more often failed when turning contact into a secure grip.
Evaluation in the simulator
The sim-only policy reached the contact/proximity threshold in 13/20 trials but never lifted the block above the grip threshold. It often hovered beside or above it, then moved toward the bowl without completing a grasp. The co-trained policy reached that threshold in 15/20 trials and completed one placement. The first problem was already present before transfer: fine-tuning on simulated demonstrations had not produced reliable task execution, even in simulation.
Video 5. All three learned policies evaluated inside the sim at 10x speed: real-only (top), sim-only (middle), co-trained (bottom). All 20 trials start together across panels; an early finish holds its final frame until the next trial. Outcomes are 0/20 (real-only), 0/20 (sim-only), 1/20 (co-trained).
The real-only policy also scored 0/20 in simulation, usually stopping short of the block. This seems to indicate some gap between policies fine-tuned on real world data and policies fine-tuned on rendered inputs. Across both simulation and real-world evals, it’s clear simulated demonstrations did not produce reliable task execution for SmolVLA even when evaluated back in the simulator.
Refining the simulation data
It feels pretty unsatisfying that simulation data didn’t produce any useful policies. Intuitively, there are a couple of first order changes we could attempt.
75/25 real-sim split: Watching the first set of evals, my impression was that simulated demonstrations gave the arm a more confident trajectory while real demonstrations made it more adaptable. I tested that idea using 100 real demonstrations and 33 simulated demonstrations from the original recipe—roughly 75/25 by episode count.
Revised simulation recipe: Our initial sim recipe was likely too straightforward and uniform. I wanted to think harder about what knobs we could turn to 1) introduce variance and 2) more closely mimic the real world.
Mat appearance: textured from screenshots of overhead camera snapshots, including tape and stickers. Added more variation in block and bowl colors, mat tint, camera pose (±3 cm and ±5°).
Retries: added deliberate misses followed by successful corrections. The first grasp is scripted to land 2-3 cm off target, then try again. Recovery episodes made up ~15% of total simulated demonstrations.
Approach: varied wrist tilt up to 25° from vertical and initial approach height between 6-14 cm. The script chooses the nearest valid jaw orientation 60% of the time, and the second nearest 40% of the time (instead of always choosing the nearest).
Grasp variation: varied opening from 22 to 35 on the arm’s control scale, compared to a fixed 20 in v1. Added a ±4 mm adjustment to the grasp point and varied the closing command around the calibrated squeeze.
Timing and placement: varied each motion segment’s duration from 0.6-1.4x its nominal duration and seeded blocks closer to the edge of the mat. In 30% of episodes, seeded blocks started within 1.2 cm of a sticker.
The revised sim required 135 attempts and 76 minutes per 100 successful episodes on an A10 GPU compared with 107 attempts and 59 minutes for the first recipe. Increasing sim variation increased the number of attempts needed to generate successful demonstrations.
Second-round evaluation
Figure 5. Success rates in sim and in the real world with 95% Wilson intervals. Sim v2 + real co-trained SmolVLA success rates improved roughly 4x in sim and 2x in real-world over a 20 trial eval.
Co-training with v2 data improved observed scores in sim and in the real world: 7/20 on the physical arm (up from 4/20) and 4/20 in simulation (up from 1/20). Interestingly, the co-trained policy remains below the real-only baseline of 9/20 in the real world. At small sample size, these results are encouraging and seem to indicate improvement with a more carefully designed, higher variation simulation environment.
Real-world arm behavior was just as interesting as the score. Several successful trials included an adjustment or retry: in trial 13, the arm dropped the block, picked it back up, and completed the placement; in other trials, the arm rotated its wrist to adjust its approach to the block.
The arm contacted the block in 15/20 trials, gripped it in 7, and completed all 7 placements. v1 co-training contacted it more often (17/20) but converted only 6 contacts into grips and 4 into placements. v2 improved completion after a grip; reaching and aligning with the block remained the main bottleneck.
Video 6. v2 co-trained policy evaluated in the real world (top, 7/20) and in simulation (bottom, 4/20). All 20 trials play at 10x speed.
Figure 6. Categorical success rates in real-world evals. All seven v2 co-trained trials that secured a grip resulted in placement; remaining failures happened before or during grasping.
Sim-only v2 still did not transfer: it completed 2/20 placements in simulation and 0/20 on the physical arm. The 75/25 split scored 3/20 in real-world evals, compared with 4/20 for the original 50/50 co-training run. Reducing the share of the original simulated data did not result in improvement to gripping or generalization.
Takeaways
Successful demos do not guarantee a successful learned policy. Both scripted simulations generated successful placements using exact object coordinates. SmolVLA had to learn the task from images and joint state. The v1 sim-only policy failed even in its own simulator; v2 produced some successful grasps in the simulator, but failed to grasp anything in the real world. Evaluation inside the simulator showed that the first bottleneck was already present before sim-to-real transfer: the policy could not reliably execute the grasp.
Training examples need to teach corrections as well as the intended motion. v2 added recovery attempts, varied approaches, timings, and grasp commands. The co-trained policy showed successful adjustments and a drop-and-regrasp recovery. v2 varied several aspects of simulation data; a useful ablation study could separate the contributions of recovery, environment textures, approach, block placement, and timing from each other.
Quality of simulated data mattered more than the real-sim split. Reducing the share of the original simulated data did not improve success in this evaluation. A smaller share of v1 sim data scored 3/20 while equal episode counts of real-v2 sim data scored 7/20. Neither beat the 9/20 real-only baseline. Simulation is still cheap to generate at this scale, but its value depends on what the resulting policy learns, not how many successful scripted sim demos we can produce.
The v2 co-trained policy yields promising improvements, recovering parts of the performance lost in the v1 co-trained policy with only modest adjustments to the simulation environment. It seems likely that improvements to our sim environment could continue to scale policy success rates; the next test is whether further improvements to the simulation data can produce repeatable gains in excess of the real-only baseline.
Code, methods, trial outcomes, and videos are collected in the simulation experiment folder in the GitHub repo.
Thanks to Claude and ChatGPT for help with implementation, analysis, and editing.