Eigen RadarAI
Analysis

Amazon SageMaker now flags slowing AI services by default

Amazon Web Services says it has turned on monitoring by default for eligible SageMaker deployments that serve generative AI models. SageMaker can warn teams when responses take longer or when graphics processors are largely idle. The company says customers can test alternative configurations on separate machines while the current system continues to handle live traffic.

Artificial Intelligence··Morning
Two separate open-frame graphics-processor test rigs with copper-heatsink GPU modules side by side on workbenches in a sunlit cloud-hardware testing workshop.

Weekly checks track how long responses take

Amazon Web Services, Amazon’s cloud computing business, says it has switched on default monitoring for eligible SageMaker AI endpoints, the services that receive requests and deliver model responses. The new checks apply to generative AI deployments that use vLLM model-serving software and graphics processors.[1]

SageMaker compares median times week to week, both for the start of a response and for its completion. It can also detect graphics processors that spend much of their time idle. AWS says the checks use aggregated operating data and do not read individual requests or responses.[1]

One notice groups findings for each service

Findings for a single endpoint appear together in a notification through AWS Health, the cloud provider’s alert service. A team can view the notice in the console for the region where the system runs or receive it by email. Customers can leave an endpoint out of monitoring, which stops the collection of its measurements for this feature.[1]

Tests run separately while the live service stays up

Teams can take a recent traffic profile into SageMaker Studio and test different machines or serving settings. A comparison can include response times, the amount of work the system handles and the cost of generating text. AWS says the live service keeps its settings during separate tests, which are billed at standard compute rates.[1]

References

  1. News sourceAWS Machine Learning BlogAmazon SageMaker starts monitoring slowing AI endpoints by default↩1↩2↩3↩4