SageMaker HyperPod Enhanced Observability Is Live: CloudWatch Flags Outlier GPUs on EKS and Slurm Clusters
AWS announced enhanced observability for SageMaker HyperPod to diagnose GPU training problems faster, covering EKS and Slurm clusters and available in all AWS Regions where HyperPod is offered.
News date: About 6 min read
Contents
Observability is the ability to collect metrics and logs so you can see what is going wrong in a system. AWS has now launched an enhanced version for Amazon SageMaker HyperPod, aimed at the situation where a large distributed GPU training job suddenly slows down. AWS notes that teams often spend hours or even days digging through logs and scattered dashboards to work out whether the cause is a throttled GPU, a saturated network link, or a stalled data pipeline on the host. According to AWS, the feature is available today in all AWS Regions where HyperPod is offered. It supports both orchestration options: Amazon EKS (AWS's managed Kubernetes service) and Slurm (a cluster job scheduler).
What's new, and how is it different from before?
AWS notes that large model training is expensive, and a single unhealthy accelerator can hold back an entire distributed job. AWS says that observing HyperPod infrastructure previously required stitching together several separate tools, and the metrics could show that performance had dropped but not why.
The diagnostic summary combines two approaches. The first is GPU outlier detection: according to AWS, it uses CloudWatch's DetectGroupOutliers API to compare each GPU's utilization, memory usage, temperature, power draw and SM clock (the clock speed of the GPU's compute cores) within the same training job. A GPU that deviates markedly from the others is shown as an outlier card. The second is a set of static diagnostic rules that classify problems by known failure patterns, as follows:
- Hardware: GPU ECC memory errors (errors detected by the memory error-correction mechanism), XID errors (error codes reported by NVIDIA GPUs), and accelerator failures.
- Network: reduced EFA (Elastic Fabric Adapter, the network interface used for inter-node communication) throughput, packet drops, and retransmissions.
- Throttling: reduced GPU clock speeds caused by heat, board or reliability limits.
- Overheating: GPU temperature exceeding the configured limit.
AWS says each detected issue is labeled with the affected job, cluster, node, accelerator and rank (the identifier of each process in distributed training), so it can be pinpointed directly.
How do default and advanced metrics, and CloudWatch and Prometheus, differ?
| Item | Contents | Notes |
|---|---|---|
| Default metrics | Cluster health, CPU/disk/OS, GPU utilization and temperature, per-job resource utilization | Enabled automatically; collected every 30 seconds; about 1–2% CPU and 250–300 MB memory |
| Advanced metrics | Full node exporter (host-level metrics), NVIDIA DCGM (GPU monitoring) and AWS Neuron metrics, EFA network metrics | Opt-in; increases telemetry volume and has cost implications |
| Amazon CloudWatch | Viewed in the AWS Management Console through CloudWatch SageMaker Insights | New default; AWS says pricing is lower than Managed Service for Prometheus |
| Amazon Managed Service for Prometheus | Suited to teams standardized on Prometheus and Grafana | Previously the only supported option |
AWS explains that most customers running small jobs on small clusters may not need advanced metrics. They are more valuable for long-running training on large clusters, where problem patterns emerge over time.
Which clusters are supported, and how do you get started?
According to AWS, the feature is delivered as an add-on on EKS clusters and as an extension on Slurm clusters, so IAM (AWS Identity and Access Management) can control who may install or change it. It can be installed through the console, the API or the AWS CLI. The console flow listed in the AWS post is as follows:
- In the SageMaker AI console, choose HyperPod clusters, then Create HyperPod cluster, and select Orchestrated on Amazon EKS or Orchestrated by Slurm.
- For Setup options choose Custom setup, enter a name, set Instance recovery to Automatic, then Submit.
- In the HyperPod observability section, confirm that Enable HyperPod Observability is checked (it is checked by default). Optionally choose advanced metrics, logs, the metrics store (CloudWatch or Prometheus) and a Grafana workspace.
- Once the cluster is ready, open the Observability tab on the cluster details page to confirm metrics are streaming; from there you can open the CloudWatch dashboard.
The AWS post also provides CLI examples such as aws eks create-addon and aws sagemaker create-cluster-extension; see the original post and the developer guide for the full commands and parameters. AWS also explains that the Observability tab shows the number of nodes, accelerators and jobs, along with average utilization of GPUs, vCPUs and memory. The CloudWatch dashboard additionally offers diagnostic summary, hardware, workload and network views.
What does this mean for day-to-day training operations?
AWS says the feature can isolate a single problematic accelerator among hundreds or thousands of GPUs, whether the problem is an XID error, an ECC memory error or thermal throttling, so teams spend their time fixing rather than searching. In the dashboard, the hardware tab provides a device-level view showing Active, Idle and Unhealthy states. The workload tab flags jobs with abnormal network usage. The network tab shows average EFA utilization, the number of degraded nodes and retransmission counts. All of this is AWS's description of the feature; actual results will depend on each team's cluster and workloads.
Frequently asked questions
Is this feature in preview or generally available?
AWS says the feature is available today in all AWS Regions where SageMaker HyperPod is offered. The post does not mention a preview or a waitlist.
Do I have to turn on advanced metrics?
No. AWS says advanced metrics are opt-in and increase the telemetry sent to the metrics store, so they carry cost implications. Most customers running small jobs on small clusters may not need them, but they are more valuable for long-running training on large clusters.
Can I change which metrics are collected after enabling it?
Yes. AWS says any metric category or profiling feature can be enabled or disabled at any time. On EKS clusters, update the add-on configuration with the AWS CLI or choose Actions → Edit on the add-on card in the console. On Slurm clusters, choose Actions → Edit on the Extensions tab of the cluster details page, or update the extension with the AWS CLI.
Browse the latest news in this topic
Found this guide helpful? Share it with someone who needs it
Claude Code 2.1.329: Code Mode Works With Telemetry, /tasks Lists Scheduled Tasks, macOS Computer Use Gets Stricter Safety Rules
Anthropic's changelog lists Claude Code 2.1.329, covering Code Mode behavior changes, new diagnostics in /tasks and /mcp, /code-review changes, and new safety rules for macOS computer use.
Kimi Code CLI v2.7.0 Adds a Native Amazon Bedrock Provider: Use Your AWS Credentials Instead of a Separate API Key
AWS Rounds Up 13 SageMaker Inference Features: Managed NVIDIA Dynamo on HyperPod, MIG GPU Partitioning, Cross-Region Routing
BMW Group Shares How It Governs AI Agents: 12,500+ Developers, Three Autonomy Levels and a Four-Stage CI/CD Gate
Sources
Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.