Amazon SageMaker HyperPod Slurm Adds Cluster Extensions: Managed Observability and User Management, Available in All HyperPod Regions
AWS has introduced cluster extensions for SageMaker HyperPod Slurm clusters, starting with two managed extensions for observability and user management, manageable via API or console. Per AWS, there is no extra charge, but AMI and continuous provisioning requirements apply.
News date: About 6 min read
Contents
AWS announced on its official blog that Amazon SageMaker HyperPod Slurm clusters now support "cluster extensions." Slurm is a system for scheduling and dispatching compute jobs across many GPU machines, commonly used in shared clusters that train large AI models. Previously, monitoring tools and user accounts on nodes were mostly set up with lifecycle scripts, which are installation scripts that run automatically when a node is created. The new feature lets teams declare, update and inspect this software through the API or the SageMaker AI console. AWS says this reduces gradual configuration drift between nodes, and when something goes wrong you can see directly which nodes and which component are affected.
How do cluster extensions differ from lifecycle scripts?
AWS notes that lifecycle scripts run only when a node is provisioned. Every change requires editing the script and then replacing nodes or rerunning it across the cluster. Errors surface only when the script runs, and there is no mechanism to report what was actually installed, so nodes can drift apart over time. AWS describes cluster extensions as AWS-managed software packages that can be installed, configured, updated and removed through the API or console.
- Up-front validation: settings are checked against a schema (rules that define the name and format of each setting) at submission, so misspelled settings are rejected before reaching nodes.
- Consistent nodes: applied to every node, including nodes added by scale-out or replacement.
- Change via API or console: both paths use the same validated configuration.
- Visible status: each extension reports its status and per-node health.
What are the two extensions and what do they do?
Per AWS, two extensions are currently supported. The observability extension collects monitoring metrics covering nodes, GPUs, EFA (Elastic Fabric Adapter, the high-speed network interface between nodes), Slurm jobs and the Slurm cluster. The metrics go to Amazon CloudWatch or Amazon Managed Service for Prometheus, and can be paired with Amazon Managed Grafana to get two prebuilt dashboards, Clusters and Jobs. The user management extension creates users and groups with identical IDs on every node. It also controls, per node type, who can use SSH (remote sign-in) and sudo (running commands with administrator rights). It also registers users with Slurm accounting and can integrate with Active Directory, a corporate directory of user accounts.
| Aspect | Observability | User management |
|---|---|---|
| Extension name | sagemaker-observability | sagemaker-user-management |
| Main function | Collects node, GPU, EFA and Slurm metrics | Creates users and groups with consistent IDs; manages SSH and sudo |
| Output or integration | CloudWatch or Managed Service for Prometheus; can pair with Managed Grafana | Users from local accounts, Active Directory over LDAPS (encrypted directory access), or both |
| Additional requirements | Execution role needs CloudWatch or aps:RemoteWrite permissions | Needs a shared file system (e.g., FSx for Lustre); reading the AD secret needs secretsmanager:GetSecretValue |
What is the adoption process and what are the operational implications?
Before use, the cluster must run a HyperPod AMI dated November 2, 2026 or later and have continuous provisioning enabled. An AMI is a preconfigured system image; continuous provisioning keeps retrying in the background to obtain compute capacity. AWS says extension status is one of Creating, Updating, Active and Degraded. Node-level problems appear in the Health field of DescribeClusterExtension, including a code, a message and the number of affected nodes. HyperPod retries internal failures itself. For other problems, you need to fix the cause and then retry with an UpdateClusterExtension call that changes nothing. When upgrading versions, if the existing configuration does not match the new version's schema, the request is rejected and nothing is changed.
If scripts already provide the same capability, AWS recommends migrating one capability at a time, as follows: first upgrade to the November 2026 AMI or later with UpdateClusterSoftware; then disable that capability in the lifecycle scripts and upload them to S3; next, in a planned maintenance window, remove the script-installed agents from each node; finally call CreateClusterExtension. AWS also stresses that users can still SSH into all nodes, including the controller, and run their own scripts.
What does it cost and where is it available?
AWS says cluster extensions are available from November 2026 in all regions that support HyperPod, with no extra charge for the extensions themselves. Users pay for the AWS resources used alongside them, such as Amazon CloudWatch, Amazon Managed Service for Prometheus and AWS Secrets Manager.
Frequently asked questions
When should I use an extension and when should I still use scripts?
Per AWS's recommendation, use an extension when HyperPod offers a matching managed capability (such as GPU and job metrics or consistent user accounts); use a custom AMI and lifecycle scripts for organization-specific software or initialization.
Can I still log in to nodes directly after using extensions?
Yes. AWS says users keep full access to all nodes, including the controller node, and can still SSH in to configure things and run their own lifecycle scripts.
Does upgrading an extension version require an AMI update?
Per AWS, no. After AWS releases a new extension version, you can list the installable versions and upgrade in place; nodes added later also get the new version. Upgrades only move forward.
Will accounts change when moving from scripts to the user management extension?
AWS says the same user names and user IDs are preserved after switching to the user management extension.
What should I do when an extension shows Degraded?
Per AWS, check the Health field of DescribeClusterExtension for the problem code and number of affected nodes, fix the cause, then retry with an UpdateClusterExtension call that includes no changes; internal failures are retried by HyperPod itself.
Found this guide helpful? Share it with someone who needs it
Claude Code 2.1.330: Enterprise Admin Console Sign-In, easel Chart Module, and Scheduled Tasks Extended to 30 Days
Anthropic's Claude Code 2.1.330 changelog lists Enterprise Admin Console sign-in, a built-in easel chart module, scheduled tasks of up to 30 days, and stricter approval for rm commands.
AWS Quick adds chat-built flows, multi-agent answers and Meta ads/Amplitude connectors, available today in all Quick regions
AWS Distributed KV Caching Agent Skill for Self-Hosted LLMs: AWS Says Cost Drops Up to 40% and Time to First Token Up to 80%
Amazon Textract Insights Now Generally Available: Extract Document Fields in Natural Language, With Confidence Scores and Page Grounding, Up to 3,000 Pages per File
Sources
Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.