AWS SageMaker HyperPod Inference Adds NVIDIA Dynamo: Multi-Node, Disaggregated Prefill/Decode, and 2.05x Benchmark Throughput on Qwen3-235B
AWS announced that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, offering multi-node deployment, prefill/decode disaggregation and KV-aware routing. Here is how it works, what AWS's benchmarks show and where the limits are.
News date: About 7 min read
Contents
In its official blog, AWS announced that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, for deploying large language models that exceed the GPU memory of a single instance. The post does not mention a preview, beta or regional restrictions. In short, prefill is the stage where the model reads and processes the prompt, and decode is the stage that generates the reply token by token. The KV cache is temporary data left over from processing the prompt that can be reused later. KV-aware routing assigns requests by considering reusable prompt-prefix caches and the load on each worker. For teams that run large-model inference themselves, the point is that splitting prefill from decode and deploying across nodes is handed off to a managed add-on.
What problem does this update solve, and how does it differ from before?
AWS points out that the FP8 weights of GLM-5.3 exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance (before counting the KV cache and runtime overhead), and that long prompts and token generation sharing a GPU slow down generation. AWS describes Dynamo as an open-source distributed inference framework that supports multi-node deployment, prefill/decode disaggregation and KV-aware routing, and works with vLLM, SGLang and TensorRT-LLM.
According to AWS, the HyperPod add-on installs Dynamo's custom resource definitions, plus the HyperPod Inference Operator, the Dynamo operator, Grove and the Kubernetes AI (KAI) Scheduler. Grove coordinates related pod groups, and the KAI Scheduler uses gang scheduling so that the pods a multi-node worker needs are scheduled together. The Dynamo runtime is provided by AWS Deep Learning Containers, and adopting a supported Dynamo release requires updating the add-on.
In the disaggregated architecture, AWS says the frontend routes requests to prefill workers, which process the prompt, and decode workers, which generate tokens. The two groups can be scaled and tuned independently, both load the full model weights, and the KV cache is transferred over EFA (Elastic Fabric Adapter, AWS's high-performance networking). Dynamo also supports aggregated serving, where prefill and decode run on the same worker.
Which deployment methods are supported?
AWS offers two paths: bring your own DynamoGraphDeployment (DGD), suited to existing Dynamo users who reuse upstream documentation and manifests; or use an InferenceEndpointConfig with dynamoSpec, suited to users already familiar with the HyperPod Inference configuration format and the OpenAI-compatible interface. The engine can be vLLM, SGLang or TensorRT-LLM, with vLLM as the default. Both paths support a single instance and multiple nodes.
| Item | Bring your own DGD | IEC with dynamoSpec |
|---|---|---|
| What you write | A full DGD manifest | An IEC containing dynamoSpec |
| Best for | Teams that want direct control of the serving graph or to reuse an existing DGD | Teams that want to configure endpoints through managed IEC |
| Model weights | You configure download or mounting | Operator stages them from Amazon S3 or Amazon FSx |
| Endpoint, TLS, registration | You configure and manage | Managed by the HyperPod Inference Operator |
| Autoscaling | You configure and manage | KEDA (Kubernetes Event-driven Autoscaling) via autoScalingSpec, for aggregated and disaggregated |
What do AWS's benchmarks show?
The first test used AIPerf to compare aggregated and disaggregated serving of Qwen3-235B-A22B-FP8 with Dynamo and TensorRT-LLM on two p5en.48xlarge instances (16 H200 GPUs). Requests had 4,000 input tokens and 200 output tokens. Only measurement points that met two targets were kept: a P90 time to first token under one second (90% of requests started replying within a second) and a median generation speed of at least 50 tokens/s per user. AWS says that among those points, disaggregated serving reached 2.05 times the highest aggregated throughput: 207 versus 101 output tokens per second per GPU, at concurrency of 80 and 32 respectively.
The second test used Qwen3.5-397B-A17B-FP8 with Dynamo and vLLM, TP4 × PP2 (the model split across GPUs using tensor and pipeline parallelism), and eight active H200 GPUs. AWS compared eight GPUs on a single p5en.48xlarge with four GPUs each on two instances. On a prefix-reuse-heavy agentic workload at concurrency 32, the two-node setup reached 126.4 output tokens/s per GPU, retaining 99.1% of single-node throughput. AWS explains that the model could fit on a single instance and that this was only a controlled benchmark.
What are the actual deployment steps and best practices?
AWS uses GLM-5.3 with vLLM on ml.p5.48xlarge as its example. The aggregated example uses one model-worker replica spanning two instances (16 GPUs) and two frontend replicas, with frontend.routingMode set to kv-aware to enable KV-aware routing. The disaggregated example has one prefill replica and two decode replicas, each spanning two instances, for six instances in total. The full manifest is applied with kubectl apply, and once the endpoint is ready, OpenAI-compatible requests can be sent through the SageMaker Runtime API; for the specific manifests and setup steps, see the AWS post and the repository it links to.
- First choose a Dynamo recipe that matches your model, inference engine and GPU type, and include weights, KV cache and runtime memory when sizing workers.
- For cross-instance deployments, first confirm that communication uses the expected EFA transport, including KV cache transfer between prefill and decode.
- Replay your own traffic with AIPerf, sweep concurrency, keep the points that meet your SLA (your service-level latency and speed targets), then weigh cost against latency.
- Tune according to measured bottlenecks: change topology and worker allocation first, then engine settings, changing one variable at a time.
Frequently asked questions
When does disaggregated prefill/decode make sense?
AWS says long prompts and token generation sharing a GPU slow down generation, and disaggregation lets the two worker groups scale and be tuned independently. Whether it pays off should be measured on your own traffic; AWS's 2.05x is a result under specific conditions only.
Can teams that already have Dynamo manifests reuse them?
According to AWS, the bring-your-own-DGD path lets you reuse upstream documentation, recipes and manifests. You need to use a supported container image and configure storage, networking and pod placement for the cluster.
What happens when a node fails?
AWS says HyperPod monitors node, GPU and EFA health and takes care of replacing nodes, and the orchestration components recreate the affected workers.
How do I upgrade the Dynamo version?
AWS says the add-on manages the versions of the integrated components, so to adopt a supported Dynamo release you update the add-on.
Found this guide helpful? Share it with someone who needs it
Shortical's All-AI Microdrama 'Inevitable' Uses Actor Aki Avni's Licensed Face; Its Public Face-Licensing Program Has No Signed Contracts Yet
Per TechCrunch, microdrama startup Shortical signed actor Aki Avni to make an all-AI microdrama using his AI avatar. Its public face-licensing program is still early, has no signed contracts and has not disclosed pay.
Chess.com Opens Classic Games Public Beta: 10 New Classic Games on Web, iOS and Android
Claude Code 2.1.330: Enterprise Admin Console Sign-In, easel Chart Module, and Scheduled Tasks Extended to 30 Days
AWS Quick adds chat-built flows, multi-agent answers and Meta ads/Amplitude connectors, available today in all Quick regions
Sources
Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.