AWS Distributed KV Caching Agent Skill for Self-Hosted LLMs: AWS Says Cost Drops Up to 40% and Time to First Token Up to 80%

AWS offers a distributed KV caching skill in the Agent Toolkit for AWS, letting coding agents such as Kiro and Claude Code help assess, build and debug a shared cache for self-hosted LLMs. AWS says costs fall up to 40% and time to first token up to 80%, with GPU and software version prerequisites.

News date: About 8 min read

Contents
  1. 1. What problem does this skill solve, and how does it differ from the usual approach?
  2. 2. How does the skill work? Four workflows and safety practices
  3. 3. Which environments are supported, and what are the version and scale conditions?
  4. 4. How do I get started?

On its official machine learning blog, AWS introduced a "distributed KV caching" Agent Skill. The skill lets coding agents help assess, build and debug a shared key-value (KV) cache layer for self-hosted LLMs. An Agent Skill is a set of specialized instructions a coding agent can load. The skill is available from the Agent Toolkit for AWS and requires specific GPU, engine and cache versions. It targets long conversations, coding agents and document Q&A services, where a GPU must recompute the entire history when a user returns to an old conversation. The approach aims to reduce that repeated computation. All figures below are AWS's own claims.

What problem does this skill solve, and how does it differ from the usual approach?

AWS points out that when self-hosting LLMs, the small independent cache on each server cannot hold or share the large amount of context that AI agents and chat applications send repeatedly. When a user returns to an old conversation, the GPU server that picks it up has no history in memory. It must recompute the KV attention state for all tokens before it can start generating output. This step is called prefill: the computation in which the model "reads" the whole conversation before answering.

AWS says distributed KV caching adds a shared layer next to the GPU fleet, on top of the local GPU cache. This lets any server reuse KV state for days across sessions, routing changes and server failures. AWS claims this approach can cut cost by up to 40% and TTFT by up to 80%.

How does the skill work? Four workflows and safety practices

AWS demonstrates the skill with four conversations: diagnosing slow first-token time, estimating size and cost, building the cache, and fixing a transfer failure. According to AWS, the agent follows several practices before acting:

  • Applicability check: the agent reads the GPU fleet with a read-only describe-instances call and counts EFA (Elastic Fabric Adapter, AWS's high-speed network interface) interfaces. It only asks for what it cannot see: engine, model, TP (tensor-parallel, the degree to which a model is split across multiple GPUs) size, KV data type, server price and how often prompts repeat. If the skill does not fit, it stops and explains why.
  • Estimate from your own traffic: with short prompts, loading KV can be slower than recomputing. Reads only pay off beyond a certain length (the cutover point, where reading starts to beat recomputing). The agent estimates with KVTrace and stops if the savings would not cover the cache cost. AWS says KVTrace carries no extra charge.
  • Keep the existing stack: the agent adds the LMCache connector to the existing vLLM or SGLang setup and preserves the original routing and engine configuration. All servers use the same KV ID scheme (hash and seed) and an explicit KV data type, to avoid mismatched keys.
  • Validate before going live: the agent clears the local tiers and confirms that two cache-loaded runs give the same output, because a layout mismatch can show normal hit counts but wrong answers. High-impact operations such as creating, changing or deleting resources require explicit approval.

In the first workflow, AWS uses a p5 fleet with a 64K-token context where a returning conversation takes 13 seconds to produce its first token. The agent determines the cause is a cold prefill, meaning the whole history is recomputed from scratch. It first checks options that need no new infrastructure, such as turning prefix caching back on (a built-in cache in which a single GPU reuses identical leading content). If the GPU prefix cache hit rate is already above 90% under session affinity (a conversation is always sent to the same server), a shared KV cache may not be needed.

The second workflow uses a multi-turn RAG (retrieval-augmented generation) assistant on 8 p5 instances with 64,000-token prompts. After looking up GPU and cache node prices, the agent recommends two cache.i8ge.48xlarge nodes. No cache hit rate was provided, meaning the share of recomputed tokens the cache would serve. The agent therefore lays out several scenarios and recommends running KVTrace. AWS explains that KVTrace records KV cache events from vLLM or SGLang and then replays them offline against each candidate capacity. The trace contains only block hashes and token counts, not token IDs or prompt text. If prompts are sensitive, add --anonymize-key-file.

The third workflow is the build. Before creating any resource, the agent asks you to confirm the node count in your Availability Zone and the estimated price. After confirmation, it creates a security group with EFA permissions and a Valkey 9.1 cluster using a custom parameter group and eviction policies. LMCache gets a dedicated cache user, with the password stored in AWS Secrets Manager. The agent starts one LMCache server on each GPU server and validates the cluster with a small test object. It also confirms that repeated prompts load KV state from the cache over RDMA (remote direct memory access, a high-speed way for servers to move data directly). In the fourth workflow, BLOB.RDMA_HELLO succeeds but data transfers fail and rdma_write_errors keeps rising. Here the agent concludes that a security group may be blocking EFA traffic between its own members, and it provides commands to check and fix the rule.

Which environments are supported, and what are the version and scale conditions?

Prerequisites and example compiled from the AWS blog
ItemRequirement or content listed by AWS
Coding agentSupports Agent Skills, such as Kiro and Claude Code; AWS also mentions OpenAI Codex
GPU instancesSupport EFA RDMA write, such as p5
Inference enginevLLM 0.11.1 or later; SGLang 0.5.13 or later
LMCache0.5.8rc1 or later
Cache (if setting up the cache)ElastiCache for Valkey 9.1 or later, using cache.i8ge.48xlarge nodes
Example scale8 p5 instances, 64,000-token prompts → 2 cache.i8ge.48xlarge nodes recommended
Assessment toolKVTrace, which AWS says carries no extra charge

How do I get started?

AWS says you can get the distributed KV caching skill or the AWS MCP Server from the Agent Toolkit for AWS. Then describe your problem, or ask your AI agent to cut the cost and latency of your GPU fleet. For the full setup with every command, see "Getting started with distributed KV caching" in the Amazon ElastiCache documentation. KVTrace also has a GitHub repository.

Frequently asked questions

When might a shared KV cache not be needed?

AWS says that if prefix caching was turned off, turning it back on can speed up back-to-back turns. If the GPU prefix cache hit rate is already above 90% under session affinity, a shared KV cache may not be needed.

Is it suitable for short prompts too?

AWS explains that with short prompts, loading KV state may take longer than recomputing it, and reading only pays off beyond the cutover point. Whether it is worthwhile depends on that length and how often prompts repeat. The agent estimates it with KVTrace.

Does KVTrace collect my prompt content?

According to AWS, the recorder does not write token IDs or prompt text. The trace contains only block hashes and token counts. Because block hashes are derived from token content, add --anonymize-key-file if prompts are sensitive.

Will the agent create or delete resources automatically?

AWS says the skill guides the agent to seek explicit approval for high-impact operations such as creating, changing or deleting resources. Before building, it also asks you to confirm the node count in your Availability Zone and the estimated price.

Browse the latest news in this topic

Found this guide helpful? Share it with someone who needs it

Sources

Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.

Lifestyle