Eigen RadarAI
Archive
Analysis

XStack asks coding agents to move entire projects between chip platforms

XStack tests whether coding agents can carry machine learning projects from graphics processors to a different hardware and software stack. Its twenty repositories require working setup, training and weight export, followed by a clean-machine rerun. Numerical results alone do not secure a pass: judges also check whether agents preserved the requested experiment rather than completing an easier substitute.

Artificial Intelligence··Night
Low-angle wide view of two open server racks using different computing accelerators in a university machine-learning laboratory.

Migration includes training and delivered weights

XStack, Princeton University’s benchmark for migrating machine learning repositories, requires more than rewritten code. Agents move projects built with PyTorch, a machine learning framework for graphics processors, to tensor processing units using the JAX software ecosystem. Twenty repositories cover fourteen domains. The assignment includes installing dependencies, preparing data, training, evaluating and exporting weights that can be loaded by the original project.[1]

A fresh machine checks the submitted scripts

The deliverable separates environment setup, training and weight export into three scripts. Rerunning them on a fresh machine checks whether the submission includes preparation that an agent performed interactively. Agents cannot see the numerical targets taken from the source experiments. Two model-based judges also examine whether the requested experiment was actually performed and whether a shortcut invalidates an apparent success.[1]

A second attempt supplies a clearer recipe

A failed initial attempt is followed by another phase with a clearer recipe for the experiment and narrower permission to edit. The hidden target stays hidden. Numerical passes can still fail the structural checks. Seven agents from five developer families are tested, but each agent-task pairing lacks repeated random trials. The preprint’s results describe these migration assignments under the chosen conditions.[1]

References

  1. News sourcearXiv / Princeton UniversityXStack tests coding agents on complete GPU-to-TPU project migrations↩1↩2↩3
AI safety and regulationChip wars and the supply chain

More analysis from this desk

View all Brief analyses from the Artificial Intelligence desk →