Lifestyle
AWS Launches Multi-Node Ray LLM Container: Prompt Reading and Answer Generation Can Be Split Across Two GPU Nodes
AWS has released a Ray LLM Deep Learning Container for multi-node deployments. On Amazon EKS, it shows two GPU nodes handling prompt reading (prefill) and answer generation (decode) separately. AWS says teams that run large language models on their own infrastructure can use this AWS-maintained, tested container instead of assembling their own software stack.
About 6 min read
What happened
In a blog post, AWS says it has launched the Ray LLM Deep Learning Container (DLC) for multi-node deployments, so the prebuilt approach of DLCs now extends to multi-node inference. Inference means running a trained model to produce answers. According to AWS, a DLC is a prebuilt Docker image, a ready-made software package. It bundles the framework, dependencies and GPU software stack into a tested and patched combination that is ready to download and use. AWS's post shows how to deploy a single large language model across two GPU nodes on Amazon EKS (Amazon Elastic Kubernetes Service, AWS's managed service for running containers). Separate prefill and decode nodes sit behind one OpenAI-compatible endpoint. In other words, users can call the service with the standard chat request format.
AWS says the Ray LLM DLC includes several components: - vLLM as the inference engine. - Ray Serve's HTTP as the serving layer. - NIXL (NVIDIA Inference Transfer Library) for KV cache transfer between nodes. - The EFA (Elastic Fabric Adapter) stack (libfabric and aws-ofi-nccl) for high-bandwidth communication between nodes. AWS also says all DLC components are validated together before each release. As a result, the versions of the CUDA runtime, the framework and the serving layer do not drift out of step. Security patches are applied at build time.
AWS says the entire serving application is just a block of YAML configuration. It needs no handler class, model archiver or custom image. A helper function, build_pd_openai_app, takes the prefill and decode configurations and wires up KV cache transfer behind the same OpenAI-compatible endpoint.
Why separate prefill and decode
According to AWS, large language model inference has two stages. Prefill processes the whole input prompt in parallel and builds the key-value (KV) cache, which is the data the model keeps about the prompt so it does not have to reprocess it. Decode then generates one token (the unit of text a model outputs) at a time, reading and extending that cache. AWS says separating the two stages stops compute-heavy long prompts from slowing token generation for requests already in progress. It also lets the two stages scale independently.
| Item | Prefill | Decode |
|---|---|---|
| Task | Processes the whole input prompt in parallel and builds the KV cache | Generates one token at a time, reading and extending that cache |
| Bottleneck per AWS | Compute-bound | Memory-bandwidth-bound |
| Node in the example | One g6.8xlarge GPU node | Another g6.8xlarge GPU node |
AWS says there is a trade-off. The KV cache built by prefill must be handed to the decode node, and this transfer has to be fast enough not to cancel out the benefit. That is why the connection between the nodes matters. AWS says this architecture suits workloads such as retrieval-augmented generation (RAG), document summarization and agentic applications.
Example architecture and things to watch
AWS's example architecture uses one CPU system node (m7i.xlarge). It hosts system workloads, the KubeRay operator and the Ray head pod. Two GPU worker nodes (g6.8xlarge) do the model work, one for prefill and one for decode. AWS notes that the GPU node group is pinned to a single Availability Zone (AZ, one data-centre location within an AWS Region), because EFA cannot span AZs.
AWS's example uses the Qwen3.5-9B model with max_model_len set to 4096. AWS says that on g6.8xlarge with the caching setup described in the post, model weight load time on restart drops from about 145 seconds to about 2 seconds.
AWS also explains that the worker pods run with privileged: true. This is because NIXL registers CUDA VRAM (GPU memory) with the EFA network card, and the IPC_LOCK capability alone is not enough.
Who this affects
Teams that run large language models on their own infrastructure now have an AWS-maintained and tested container. They no longer have to assemble the GPU and networking software stack themselves. AWS says common multi-node inference patterns include three approaches: - Tensor and pipeline parallelism, which splits a model that does not fit on a single instance. - Data parallelism, which copies the model to increase throughput. - Prefill/decode disaggregation, which places the two inference stages on different nodes.
AWS lists these prerequisites: - An AWS account with billing enabled. - Enough quota to run two g6.8xlarge instances (at least 64 vCPUs). - The AWS CLI, kubectl, eksctl, helm and envsubst tools. AWS's deployment has four steps: 1. Create the EKS cluster (about 15–20 minutes from scratch). 2. Add the GPU worker nodes. 3. Install the KubeRay operator. 4. Deploy the RayService. To avoid ongoing charges, AWS recommends tearing down the resources in reverse order.
All of the above is AWS's own account. Performance and cost results in your own environment may differ.
Frequently asked questions
What is prefill/decode disaggregation?
AWS describes it as placing two stages on different nodes. The prefill stage processes the input prompt, and the decode stage generates tokens one at a time. Separating them means they do not interfere with each other and can scale independently.
What is this container used for?
AWS says the Ray LLM DLC includes vLLM, Ray Serve's HTTP serving layer, NIXL and the EFA stack. It is used to run one model across several machines behind an OpenAI-compatible endpoint.
Do I need to write code to deploy it?
AWS says the entire serving application is just a block of YAML, with no handler class, model archiver or custom image needed. You still need to set up an EKS cluster and the related tools.
Has anyone outside AWS confirmed these figures?
The details here come only from AWS's own blog post, so they are AWS's claims. Results in your own setup may differ.
Browse the latest news in this topic
Lifestyle
Anthropic Pledges 5% of Its Shares to a Public Benefit Fund: What We Know, the Limits, and Open Questions
Anthropic says that, with shareholder approval, it will commit 5% of its shares to a new Public Benefit Fund. This piece covers the company's statements, the fund's structure, how it differs from the existing $200 million research fund, and what it means for readers.
Lifestyle
Anthropic Discloses Internal Shifts: Fewer Junior Technical Roles, More Time in Meetings
AI company Anthropic has published research on employment at its own firm. It says the share of junior staff and new hires in technical teams has fallen sharply, AI agents now work more hours than employees, and employees spend more time in meetings. The report shows job seekers and workers how a heavy AI user is changing hiring and work, but it reflects only that one company.
Lifestyle
Meta Removes Disputed Commercial Terms Language Read as Allowing Scraping of News Sites for AI Training
The Verge reports that Meta, after criticism, revised its commercial terms to remove wording critics read as letting it scrape content from news and other sites to train AI, even overriding site blocks. Meta says the terms apply only to its ads and business products, and site owners can still state preferences via robots.txt.
Lifestyle
Websites Are Blocking AI Agents: Why New Booking and Shopping Features Often Fail
The Verge and TechCrunch report that Amazon, Yelp, Delta Air Lines and other sites are restricting AI agents that act on users' behalf. Some blocks are deliberate; others are anti-bot checks catching agents by mistake. Here is what each side says and what it means for users.