AWS described a nine-stage evaluation framework for the AI agents used by its internal sales organization. The system checks answers for errors in numbers, reporting periods and customer identity alongside safety, speed and cost. Release tests, live probes and feedback feed into the same evaluation loop. The October 16 account describes AWS’s own implementation and its reported experience.
Cloud-computing provider AWS described how it evaluates the AI agents used by its internal sales teams in an October 16 technical account. Before launch, one answer incorrectly labelled a value as millions, and a separate answer invented an annual recurring revenue figure. The company’s framework checks meaning as well as whether the system keeps responding.[1]
The nine stages cover defining success, experiments, merge and deployment checks, live probes, adversarial tests, drift monitoring, observability and feedback. Tools, individual agents and their joint workflows have separate verified test sets. Known factual answers receive deterministic checks; language-model judges assess reasoning and helpfulness. Quality, safety, latency and cost are evaluated together.[1]
A small pre-release set contains 15 high-risk cases, with a fuller evaluation before production promotion. Live checks run in windows of 30 minutes. The on-call team receives an alert after three successive windows exceed a threshold. Low-scoring sessions become permanent regression cases. This single AWS account describes the company’s own implementation, using AgentCore evaluation services, with traces and stored results helping teams locate the cause of a failure.[1]
