A world built to be argued with
Most robot demonstrations answer one question: did it work, in this room, on this day. ToyBlocks is built to answer a more useful one: which part failed. Its evaluation can substitute a reference high-level planner or low-level policy, helping researchers investigate whether failures arise primarily from skill selection, execution or interactions between the two. That provides more diagnostic information than a single end-to-end success score, although it does not guarantee that every failure has one identifiable cause. The same separation runs through the data. Because demonstrations are planned and generated rather than filmed, a team can add more flipping or rotation examples, hold the tasks fixed, and measure whether the gap they targeted actually closed.
The generation method deserves a plain reading. The skills inside those 85,336 demonstrations were produced with access to the simulator's own ground truth, including exact block positions that no camera-equipped robot gets. That is a reasonable way to manufacture training data at scale, and Ai2 states it openly. It also means the dataset's size is not evidence about how hard the underlying tasks are for a system that has to perceive the scene itself. The generated demonstrations can use privileged simulator state, including exact object poses. The trained Molmo 2 and ManiFlow components are then evaluated using sensor observations rather than being given that privileged state directly. This separates the process used to generate supervision from the information available to the deployed models.
Two scores, and they do not agree
Ai2 reports two different kinds of success. The low-level ManiFlow policy completed 80.0% of tested atomic skill executions in simulation and 81.1% on a physical Franka FR3 arm. The hierarchical Molmo 2–ManiFlow system, which must select and sequence multiple skills to spell a three-letter word, completed 55.5% of simulated episodes and 50.0% of physical episodes. The atomic-skill and full-episode rates measure different tasks and should not be compared as though they share a denominator.
The full system tells a different story. With Molmo 2 selecting skills and ManiFlow executing them, success on three-block spelling tasks was 55.5% in simulation (111 of 200 episodes) and 50.0% on the physical arm (15 of 30). These tests used the easiest scene configuration: the block characters were visible and upright at the start. Physical scenes contained four to ten blocks drawn from those seen during training, and an episode counted as a failure if an operator stopped the robot for an unsafe motion. Ai2 cautions that further trials are needed to establish statistical confidence. The result measures one specific system on a constrained task, not general robot manipulation.
With only 30 physical episodes, each outcome changes the reported success rate by about 3.3 percentage points. Three additional successes would raise it from 50% to 60%. Ai2 explicitly cautions that more trials are needed to establish statistical confidence, so the result should not be treated as a stable deployment success rate.
Ai2 also connected Claude Opus 5.5 to ToyBlocks skills through MCP. In search-and-spell tasks, half of which were deliberately infeasible, the agent solved the task or correctly declared it impossible in 63% of cases. Ai2 identifies look-alike letter confusion (13%) and incorrect letter orientation (15%) as prominent error categories. In separate, simpler stacking evaluations, its Claude-based system completed 30 of 30 tasks using text descriptions and another 30 of 30 with closed-loop camera observations. These results suggest that visual interpretation and orientation handling are important bottlenecks in the harder search-and-spell setting. They do not establish that planning is generally solved or that perception accounts for every failure.
What is actually in the box
The release includes simulation assets, procedural task and demonstration generators, 85,336 long-horizon demonstrations containing 481,207 skill steps, two trained models, training and evaluation code, and validation sets. The first dataset contains approximately 1,600 hours of simulated robot execution and 144 million frames. Ai2 has not yet included fully integrated reinforcement-learning training in the released package, although the generators and other training infrastructure are available. It is targeting a larger second dataset and expanded capabilities in a future release; that timeline is a stated plan, not a delivered feature.
That narrowness is the point, and also the boundary of what the results can mean. ToyBlocks can tell the field, repeatably, why a policy fails at spelling with blocks. Whether the same failure modes govern a robot folding laundry is a question this release does not ask, and its numbers should not be quoted as if it had.
Sources
- [1] Ai2, "ToyBlocks: A controlled world for robot learning and evaluation", 6 November 2026Read source
- [2] ToyBlocks project site (dataset and sim-to-real figures)Read source
- [3] Unite.AI, "Ai2 Releases ToyBlocks to Test How Robots Learn and Fail", 6 November 2026Read source