One outcome, different failures
A robot that reaches for the right block and drops it has a different problem from one that grasps the wrong block perfectly. A robot that keeps searching for a missing letter has a third problem. All three can nevertheless leave the same word unfinished. What interests me about Ai2’s new ToyBlocks environment is the experimental design that lets these failures be separated. Collapsing task selection and movement into one success number hides which capability improved. Better movement can also make a robot more reliable at pursuing the wrong target. ToyBlocks makes that distinction testable by separating a high-level planner from a motion controller.[1]
At the high level, Molmo2 selects a block, a point and a skill; ManiFlow converts that selection into movement. The environment generates longer tasks by combining pushing, flipping, grasping and placing. The data-generation system can access privileged information such as positions and orientations, whereas the visual models learn from camera observations. That difference matters: what the demonstration generator knows is not automatically what a robot can infer from an image. A clean demonstration does not establish that visual perception represents the same state correctly. The image left by one movement also becomes the input for the next choice, so errors need not originate in the final step.[1]
Sometimes the right answer is to stop
Replacing one component with a reference component while testing the other offers competing explanations for the same failure. If a better controller rescues the task, execution may be the bottleneck. But that alone does not prove that the planner was correct in every case: the stronger controller might compensate for a poorly selected target. Conversely, improvement with a reference planner does not make every movement from the visual model defective. I see the value of this arrangement in asking what changes when the other component is held fixed, rather than assigning success or failure wholesale to one model. Component performance and complete-task performance need to be read together.[1]
Missing-letter scenes change the measurement again. Some word tasks omit a required block, making recognition of impossibility the correct outcome. Searching longer in such a scene is not necessarily progress. Turning over a hidden face can provide information, but cannot produce an absent letter. A measure counting completed words alone would label a correct refusal as failure. Conversely, accepting impossibility judgments must not create an easy escape: premature refusal on solvable scenes should remain visible separately. This is a different capability from making more movements. It requires separating useful observation, a decision under available information and the feasibility of the requested goal.[1]
What does the physical table show?
The physical-robot test narrows the scale again. Ai2 reports thirty scene configurations with a Franka FR3 arm and a Robotiq gripper, compared with two hundred trials in the corresponding simulation evaluation. The physical setup uses the easiest task, with familiar blocks and letters already visible and upright. A collision or operator intervention counts as failure. These details distinguish trying a simulated skill on a physical table from demonstrating general transfer. Apparently similar outcomes could also reflect the narrow choice of scenes. Thirty configurations do not automatically carry the same conclusion to hidden letters, different surfaces or harder search tasks. The test’s scope remains part of its result.[1]
My conclusion from ToyBlocks is that failure can be decomposed more usefully, rather than that robots have solved block manipulation. In the next comparison I would look beyond a higher overall score: which errors disappear when the controller changes with the planner held fixed, how impossibility judgments behave on solvable scenes, and which conditions enter the physical test. Those are different capabilities, not three names for the same improvement. The value of the toy table lies in drawing an experimental boundary between an inability to move and a mistaken decision. Preserving that boundary makes it easier to understand which intervention solved which problem, even when the final demonstration looks equally fluent.[1]