Eigen RadarAI
Analysis

Ragas restricts image access during AI evaluation

Ragas version 0.4.4 fixes a vulnerability in image-based evaluation that could expose internal network resources or local images to the model provider. The library now limits image requests to public internet addresses and requires explicit permission for local files. The maintainers’ release document covers both its current and older evaluation interfaces.

Artificial Intelligence··Night
Close view of an open laptop screen showing a blurred grid of image thumbnails, with several flat grey placeholder tiles marking images that were not loaded.

Image evaluation gains access restrictions

Ragas, a library for evaluating AI responses, released version 0.4.4 on November 14 with a security fix for its multimodal metrics. Image references in untrusted evaluation context could prompt the evaluation machine to request internal network resources or read local images and send the content to the model provider. The fix covers both the current collections interface and the older interface. The maintainers’ release document is the single source for this development.[1]

Public addresses are checked at connection time

Image requests now go only to public internet addresses. The check applies to the address actually connected to and is repeated after redirects. Local images are not read by default; permitting access also requires choosing the directory from which images may be loaded. When an image remains unloaded, the model receives its textual reference instead and the software logs a warning. Refused image loads are recorded during evaluation, and local files remain closed unless access is expressly allowed.[1]

Model compatibility changes accompany the fix

The same release makes removed VertexAI imports optional, restoring startup after a dependency change. Anthropic requests omit sampling parameters unless users explicitly set them and use JSON output by default. Recognition of dotted OpenAI model names also changes. Maintainers note that older Anthropic models can consequently produce more variation between evaluation runs because the default sampling settings have changed.[1]

References

  1. News sourceRagasRagas closes evaluation-data access to internal networks and local files↩1↩2↩3