AWS Rounds Up 13 SageMaker Inference Features: Managed NVIDIA Dynamo on HyperPod, MIG GPU Partitioning, Cross-Region Routing
On November 20, AWS published part 2 of its 2026 SageMaker Inference review, covering 13 post-launch operations features across recommendations, GPU partitioning, large-model serving, cross-Region routing and monitoring.
News date: About 6 min read
Contents
On the official AWS blog, AWS published "Amazon SageMaker Inference: 2026 launches in review, part 2", which continues the 13 features from part 1 with new inference features numbered 14 to 26, spanning July to November 2026. Inference means running a trained AI model to answer real requests. AWS does not label each feature as preview or generally available, so check AWS documentation for Region availability and conditions. For teams running generative AI services, the focus is how to tune settings, raise GPU utilization and cut latency after launch. All feature descriptions and performance figures below come from AWS.
What do these updates change?
AWS threads a hypothetical customer-support assistant team through the post, introducing features in five stages: choosing a configuration, letting more models share a GPU, serving models larger than a single instance, steering requests to the right GPU, and continuing to tune after launch.
- Inference recommendations: AWS says that beyond the April API, support has been added in SageMaker AI Studio (July), Python SDK v3 3.17.0 or later (August), and LoRA adapter support plus SLA concurrency search (September). A LoRA adapter is an add-on module that specializes one base model for different uses.
- Coding-agent skill: the aws-ai-ml skill is delivered through the Agent Toolkit for AWS; AWS says setup takes about 10 minutes and the output is reviewable SDK v3 code.
- GPU utilization: the console can manage Inference Components (deploying and scaling multiple models separately on the same endpoint) and use NVIDIA MIG to split one GPU into up to seven partitions.
- Large models: HyperPod Inference, which runs on dedicated clusters with Kubernetes-native control, offers managed NVIDIA Dynamo, supporting multi-node sharded serving (one model spread across several nodes) and prefill/decode disaggregation (processing the prompt and generating text run on separate GPU pools).
What are each feature's timing, scope and limits?
| Feature | Timing per AWS | Scope or limits (per AWS) |
|---|---|---|
| Studio inference recommendations | July 2026 | Jobs > Inference optimization; can deploy from Studio to a real-time endpoint |
| Python SDK v3 inference recommendations | August 2026 | Version 3.17.0 or later; can compare LMI and vLLM in parallel |
| LoRA support and SLA concurrency search | September 2026 | search_recipe set to max-concurrency-under-sla |
| Inference Components console | October 2026 | Create, scale (down to zero copies) and monitor |
| NVIDIA MIG partitioning | October 2026 | Up to seven partitions; still billed for the full instance-hour |
| Managed NVIDIA Dynamo | November 2026 | Regions where HyperPod is available; vLLM, SGLang, TensorRT-LLM |
| KV-cache-aware routing | November 2026 | Initial release requires an AWS-verified model container |
| HyperPod Inference Gateway | September 2026 | P99 TTFT down by up to 97%/98% (vs. round-robin) |
| Cross-Region inference routing | Month not stated in the post | Adds no capacity and copies no models; overflow limits and Region restrictions can be set |
| Continuous performance monitoring | November 2026 | On by default, with opt-out; does not read requests or responses |
| In-place rolling model updates | November 2026 | Applied automatically; 20 instances, batch of 2 needs 2 new instances |
How does this affect day-to-day inference workflows?
AWS says engineers can launch benchmarks and get ranked recommendations from a notebook, a CI pipeline or a coding agent, without depending on the console. Concurrency is the number of requests an endpoint handles at once; SLA concurrency search finds the highest concurrency that still meets the latency and error-rate targets, replacing the previous approach of testing round by round by hand.
On routing, AWS says each HyperPod cluster can serve all models through a single OpenAI-compatible endpoint, with the Gateway choosing targets based on each pod's live state rather than round-robin (assigning in turn). KV-cache-aware routing on standard endpoints sends requests to the instance holding the cache for the longest matching prompt prefix; the KV cache is the reusable intermediate result a model keeps after processing a prompt. Cross-Region routing connects over AWS PrivateLink and can restrict which Regions may handle requests. Continuous monitoring detects latency drift, configuration gaps and idle GPUs and opens guided fixes in Studio, but deploys no change until the user applies it.
How does AWS suggest getting started?
AWS suggests starting with the part that matches your current problem: rerun inference recommendations if traffic has changed; move small models onto partitioned GPUs if placement slots are scarce; and register clusters and endpoints for cross-Region routing if requests queue while your own capacity sits idle. Documentation and getting-started guides for each feature are in the Amazon SageMaker AI Developer Guide. AWS also says teams already on HyperPod Inference can update the add-on and add a short dynamoSpec block to an existing InferenceEndpointConfig.
Frequently asked questions
In which Regions is managed Dynamo available?
AWS says it is available in the Regions where Amazon SageMaker HyperPod is available. The post does not list specific Regions.
Does MIG lower the hourly price?
No. AWS says billing stays at the standard rate for the full instance-hour; the savings come from placing more models on each instance.
Does performance monitoring read my request content?
AWS says no; it looks only at latency, GPU utilization and endpoint serving configuration. It is on by default, you can opt endpoints out, and you can turn it off at any time.
Can cross-Region inference routing fix a capacity shortage?
AWS says the router adds no capacity, copies no models and deploys no endpoints; it redirects traffic among the Regional endpoints you have already deployed.
What is planned next?
AWS plans to extend in-place rolling updates in early 2027 with scale-in-first updates, which remove the need for any additional instances during a model update.
Browse the latest news in this topic
Found this guide helpful? Share it with someone who needs it
Claude Code 2.1.329: Code Mode Works With Telemetry, /tasks Lists Scheduled Tasks, macOS Computer Use Gets Stricter Safety Rules
Anthropic's changelog lists Claude Code 2.1.329, covering Code Mode behavior changes, new diagnostics in /tasks and /mcp, /code-review changes, and new safety rules for macOS computer use.
Kimi Code CLI v2.7.0 Adds a Native Amazon Bedrock Provider: Use Your AWS Credentials Instead of a Separate API Key
BMW Group Shares How It Governs AI Agents: 12,500+ Developers, Three Autonomy Levels and a Four-Stage CI/CD Gate
Amazon SageMaker AI Now Monitors vLLM GPU Inference Endpoints by Default: Latency Drift, Idle GPUs and Missed Optimizations
Sources
Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.