Weekly checks track how long responses take
Amazon Web Services, Amazon’s cloud computing business, says it has switched on default monitoring for eligible SageMaker AI endpoints, the services that receive requests and deliver model responses. The new checks apply to generative AI deployments that use vLLM model-serving software and graphics processors.[1]
SageMaker compares median times week to week, both for the start of a response and for its completion. It can also detect graphics processors that spend much of their time idle. AWS says the checks use aggregated operating data and do not read individual requests or responses.[1]
