Scientist World builds 1,000 test environments from research papers
Princeton researchers introduced a collection of executable research environments that lets artificial intelligence agents attempt improvements to published methods. The initial collection has 1,000 environments, but the model comparison uses ten lower-compute tasks. A separate experiment assigns idea generation and implementation to different models, keeping the test of scientific ideas distinct from the ability to carry out an entire project.
Artificial Intelligence··Night
A thousand environments, ten comparison tasks
Scientist World, Princeton University’s collection for testing research agents, starts from published methods and turns their research questions into runnable assignments. The collection contains 1,000 environments. Each sets out which changes are allowed and how a proposed improvement is scored. The model comparison covers ten tasks chosen for lower computing requirements. The selection favors empirical work that can be executed within a practical computing budget.[1]
Construction and checking have separate roles
Several builder roles prepare an assignment: one reproduces the original experiment, another sets the task conditions and another packages dependencies and data access. A verifier outside that construction process checks the resulting code, execution logs and scores. These are separate automated roles; individual papers do not receive human evaluation. Development data and held-out evaluation data have distinct uses, with the reproduced method providing the baseline.[1]
Idea generation uses a fixed implementer
Another experiment trains a smaller model to propose research ideas while a stronger model implements every proposal. Keeping the implementer fixed isolates the origin of the idea in that comparison. The preprint therefore measures a narrower task than independently managing a complete research project. Its environments retain permitted edits, data interfaces and scoring conditions so that agents attempt improvements under an explicit task specification.[1]
