AWS SageMaker HyperPod Inference Adds NVIDIA Dynamo: Multi-Node, Disaggregated Prefill/Decode, and 2.05x Benchmark Throughput on Qwen3-235B

AWS announced that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, offering multi-node deployment, prefill/decode disaggregation and KV-aware routing. Here is how it works, what AWS's benchmarks show and where the limits are.

News date: About 7 min read

Contents
  1. 1. What problem does this update solve, and how does it differ from before?
  2. 2. Which deployment methods are supported?
  3. 3. What do AWS's benchmarks show?
  4. 4. What are the actual deployment steps and best practices?

In its official blog, AWS announced that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, for deploying large language models that exceed the GPU memory of a single instance. The post does not mention a preview, beta or regional restrictions. In short, prefill is the stage where the model reads and processes the prompt, and decode is the stage that generates the reply token by token. The KV cache is temporary data left over from processing the prompt that can be reused later. KV-aware routing assigns requests by considering reusable prompt-prefix caches and the load on each worker. For teams that run large-model inference themselves, the point is that splitting prefill from decode and deploying across nodes is handed off to a managed add-on.

What problem does this update solve, and how does it differ from before?

AWS points out that the FP8 weights of GLM-5.3 exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance (before counting the KV cache and runtime overhead), and that long prompts and token generation sharing a GPU slow down generation. AWS describes Dynamo as an open-source distributed inference framework that supports multi-node deployment, prefill/decode disaggregation and KV-aware routing, and works with vLLM, SGLang and TensorRT-LLM.

According to AWS, the HyperPod add-on installs Dynamo's custom resource definitions, plus the HyperPod Inference Operator, the Dynamo operator, Grove and the Kubernetes AI (KAI) Scheduler. Grove coordinates related pod groups, and the KAI Scheduler uses gang scheduling so that the pods a multi-node worker needs are scheduled together. The Dynamo runtime is provided by AWS Deep Learning Containers, and adopting a supported Dynamo release requires updating the add-on.

In the disaggregated architecture, AWS says the frontend routes requests to prefill workers, which process the prompt, and decode workers, which generate tokens. The two groups can be scaled and tuned independently, both load the full model weights, and the KV cache is transferred over EFA (Elastic Fabric Adapter, AWS's high-performance networking). Dynamo also supports aggregated serving, where prefill and decode run on the same worker.

Which deployment methods are supported?

AWS offers two paths: bring your own DynamoGraphDeployment (DGD), suited to existing Dynamo users who reuse upstream documentation and manifests; or use an InferenceEndpointConfig with dynamoSpec, suited to users already familiar with the HyperPod Inference configuration format and the OpenAI-compatible interface. The engine can be vLLM, SGLang or TensorRT-LLM, with vLLM as the default. Both paths support a single instance and multiple nodes.

Division of responsibilities in the two deployment paths described by AWS
ItemBring your own DGDIEC with dynamoSpec
What you writeA full DGD manifestAn IEC containing dynamoSpec
Best forTeams that want direct control of the serving graph or to reuse an existing DGDTeams that want to configure endpoints through managed IEC
Model weightsYou configure download or mountingOperator stages them from Amazon S3 or Amazon FSx
Endpoint, TLS, registrationYou configure and manageManaged by the HyperPod Inference Operator
AutoscalingYou configure and manageKEDA (Kubernetes Event-driven Autoscaling) via autoScalingSpec, for aggregated and disaggregated

What do AWS's benchmarks show?

The first test used AIPerf to compare aggregated and disaggregated serving of Qwen3-235B-A22B-FP8 with Dynamo and TensorRT-LLM on two p5en.48xlarge instances (16 H200 GPUs). Requests had 4,000 input tokens and 200 output tokens. Only measurement points that met two targets were kept: a P90 time to first token under one second (90% of requests started replying within a second) and a median generation speed of at least 50 tokens/s per user. AWS says that among those points, disaggregated serving reached 2.05 times the highest aggregated throughput: 207 versus 101 output tokens per second per GPU, at concurrency of 80 and 32 respectively.

The second test used Qwen3.5-397B-A17B-FP8 with Dynamo and vLLM, TP4 × PP2 (the model split across GPUs using tensor and pipeline parallelism), and eight active H200 GPUs. AWS compared eight GPUs on a single p5en.48xlarge with four GPUs each on two instances. On a prefix-reuse-heavy agentic workload at concurrency 32, the two-node setup reached 126.4 output tokens/s per GPU, retaining 99.1% of single-node throughput. AWS explains that the model could fit on a single instance and that this was only a controlled benchmark.

What are the actual deployment steps and best practices?

AWS uses GLM-5.3 with vLLM on ml.p5.48xlarge as its example. The aggregated example uses one model-worker replica spanning two instances (16 GPUs) and two frontend replicas, with frontend.routingMode set to kv-aware to enable KV-aware routing. The disaggregated example has one prefill replica and two decode replicas, each spanning two instances, for six instances in total. The full manifest is applied with kubectl apply, and once the endpoint is ready, OpenAI-compatible requests can be sent through the SageMaker Runtime API; for the specific manifests and setup steps, see the AWS post and the repository it links to.

  • First choose a Dynamo recipe that matches your model, inference engine and GPU type, and include weights, KV cache and runtime memory when sizing workers.
  • For cross-instance deployments, first confirm that communication uses the expected EFA transport, including KV cache transfer between prefill and decode.
  • Replay your own traffic with AIPerf, sweep concurrency, keep the points that meet your SLA (your service-level latency and speed targets), then weigh cost against latency.
  • Tune according to measured bottlenecks: change topology and worker allocation first, then engine settings, changing one variable at a time.

Frequently asked questions

When does disaggregated prefill/decode make sense?

AWS says long prompts and token generation sharing a GPU slow down generation, and disaggregation lets the two worker groups scale and be tuned independently. Whether it pays off should be measured on your own traffic; AWS's 2.05x is a result under specific conditions only.

Can teams that already have Dynamo manifests reuse them?

According to AWS, the bring-your-own-DGD path lets you reuse upstream documentation, recipes and manifests. You need to use a supported container image and configure storage, networking and pod placement for the cluster.

What happens when a node fails?

AWS says HyperPod monitors node, GPU and EFA health and takes care of replacing nodes, and the orchestration components recreate the affected workers.

How do I upgrade the Dynamo version?

AWS says the add-on manages the versions of the integrated components, so to adopt a supported Dynamo release you update the add-on.

Browse the latest news in this topic

Found this guide helpful? Share it with someone who needs it

Sources

Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.

Lifestyle