AWS Rounds Up 13 SageMaker Inference Features: Managed NVIDIA Dynamo on HyperPod, MIG GPU Partitioning, Cross-Region Routing

On November 20, AWS published part 2 of its 2026 SageMaker Inference review, covering 13 post-launch operations features across recommendations, GPU partitioning, large-model serving, cross-Region routing and monitoring.

News date: About 6 min read

Share on LINE
Contents
  1. 1. What do these updates change?
  2. 2. What are each feature's timing, scope and limits?
  3. 3. How does this affect day-to-day inference workflows?
  4. 4. How does AWS suggest getting started?

On the official AWS blog, AWS published "Amazon SageMaker Inference: 2026 launches in review, part 2", which continues the 13 features from part 1 with new inference features numbered 14 to 26, spanning July to November 2026. Inference means running a trained AI model to answer real requests. AWS does not label each feature as preview or generally available, so check AWS documentation for Region availability and conditions. For teams running generative AI services, the focus is how to tune settings, raise GPU utilization and cut latency after launch. All feature descriptions and performance figures below come from AWS.

What do these updates change?

AWS threads a hypothetical customer-support assistant team through the post, introducing features in five stages: choosing a configuration, letting more models share a GPU, serving models larger than a single instance, steering requests to the right GPU, and continuing to tune after launch.

  • Inference recommendations: AWS says that beyond the April API, support has been added in SageMaker AI Studio (July), Python SDK v3 3.17.0 or later (August), and LoRA adapter support plus SLA concurrency search (September). A LoRA adapter is an add-on module that specializes one base model for different uses.
  • Coding-agent skill: the aws-ai-ml skill is delivered through the Agent Toolkit for AWS; AWS says setup takes about 10 minutes and the output is reviewable SDK v3 code.
  • GPU utilization: the console can manage Inference Components (deploying and scaling multiple models separately on the same endpoint) and use NVIDIA MIG to split one GPU into up to seven partitions.
  • Large models: HyperPod Inference, which runs on dedicated clusters with Kubernetes-native control, offers managed NVIDIA Dynamo, supporting multi-node sharded serving (one model spread across several nodes) and prefill/decode disaggregation (processing the prompt and generating text run on separate GPU pools).

What are each feature's timing, scope and limits?

Selected features from AWS's part 2 review (source: AWS official blog)
FeatureTiming per AWSScope or limits (per AWS)
Studio inference recommendationsJuly 2026Jobs > Inference optimization; can deploy from Studio to a real-time endpoint
Python SDK v3 inference recommendationsAugust 2026Version 3.17.0 or later; can compare LMI and vLLM in parallel
LoRA support and SLA concurrency searchSeptember 2026search_recipe set to max-concurrency-under-sla
Inference Components consoleOctober 2026Create, scale (down to zero copies) and monitor
NVIDIA MIG partitioningOctober 2026Up to seven partitions; still billed for the full instance-hour
Managed NVIDIA DynamoNovember 2026Regions where HyperPod is available; vLLM, SGLang, TensorRT-LLM
KV-cache-aware routingNovember 2026Initial release requires an AWS-verified model container
HyperPod Inference GatewaySeptember 2026P99 TTFT down by up to 97%/98% (vs. round-robin)
Cross-Region inference routingMonth not stated in the postAdds no capacity and copies no models; overflow limits and Region restrictions can be set
Continuous performance monitoringNovember 2026On by default, with opt-out; does not read requests or responses
In-place rolling model updatesNovember 2026Applied automatically; 20 instances, batch of 2 needs 2 new instances

How does this affect day-to-day inference workflows?

AWS says engineers can launch benchmarks and get ranked recommendations from a notebook, a CI pipeline or a coding agent, without depending on the console. Concurrency is the number of requests an endpoint handles at once; SLA concurrency search finds the highest concurrency that still meets the latency and error-rate targets, replacing the previous approach of testing round by round by hand.

On routing, AWS says each HyperPod cluster can serve all models through a single OpenAI-compatible endpoint, with the Gateway choosing targets based on each pod's live state rather than round-robin (assigning in turn). KV-cache-aware routing on standard endpoints sends requests to the instance holding the cache for the longest matching prompt prefix; the KV cache is the reusable intermediate result a model keeps after processing a prompt. Cross-Region routing connects over AWS PrivateLink and can restrict which Regions may handle requests. Continuous monitoring detects latency drift, configuration gaps and idle GPUs and opens guided fixes in Studio, but deploys no change until the user applies it.

How does AWS suggest getting started?

AWS suggests starting with the part that matches your current problem: rerun inference recommendations if traffic has changed; move small models onto partitioned GPUs if placement slots are scarce; and register clusters and endpoints for cross-Region routing if requests queue while your own capacity sits idle. Documentation and getting-started guides for each feature are in the Amazon SageMaker AI Developer Guide. AWS also says teams already on HyperPod Inference can update the add-on and add a short dynamoSpec block to an existing InferenceEndpointConfig.

Frequently asked questions

In which Regions is managed Dynamo available?

AWS says it is available in the Regions where Amazon SageMaker HyperPod is available. The post does not list specific Regions.

Does MIG lower the hourly price?

No. AWS says billing stays at the standard rate for the full instance-hour; the savings come from placing more models on each instance.

Does performance monitoring read my request content?

AWS says no; it looks only at latency, GPU utilization and endpoint serving configuration. It is on by default, you can opt endpoints out, and you can turn it off at any time.

Can cross-Region inference routing fix a capacity shortage?

AWS says the router adds no capacity, copies no models and deploys no endpoints; it redirects traffic among the Regional endpoints you have already deployed.

What is planned next?

AWS plans to extend in-place rolling updates in early 2027 with scale-in-first updates, which remove the need for any additional instances during a model update.

Browse the latest news in this topic

Found this guide helpful? Share it with someone who needs it

Share on LINE

Sources

Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.

Lifestyle