Vals tests find safety gaps in simulated AI targeting
US evaluation startup Vals found that commercial AI models supplied targeting information in simulated military scenarios containing civilians. Published on November 17, the evaluation examined responses to drone and satellite imagery rather than deployed weapons. The findings prompted Anthropic and OpenAI to reiterate restrictions on violent misuse, while the researchers described limits to what their tests establish.
Artificial Intelligence··Morning
Civilian scenes expose gaps
US model-evaluation company Vals AI published tests on November 17 in which commercial systems supplied targeting information from simulated military scenes. Some outputs selected civilians, ambulances or soldiers who had surrendered or become incapacitated. The evaluation used drone and satellite imagery, combining generated scenes with real images. Its subject was how the models responded when asked to participate in a weapons-related workflow.[1], [2]
What the evaluation measured
Researchers Geby Jaff and Anthony Ozerov examined model behaviour rather than conducting a live weapons trial. They did not ask the systems to make direct strike decisions. The findings concern information supplied ahead of such a decision. In longer sequences, models encountered images depicting the consequences of simulated strikes; seeing those consequences did not stop every system from continuing.[1]
Vals says the work does not establish targeting accuracy, compliance with the law of armed conflict or performance in actual US military processes. It also identifies latency and limited onboard computing as obstacles to deployment. Following a requested response format and actually identifying a target were treated as separate outcomes.[1]
Companies reiterate restrictions
Anthropic and OpenAI told TechCrunch that the conduct simulated in the evaluation breaches their usage rules. Both companies described these uses of their commercial models as prohibited.[1], [2]