EPD Disaggregation: Scaling Multimodal LLM Serving

In early November 2025, the vLLM project merged pull request 25233 and shipped native encoder disaggregation in release 0.11.1. That change moved EPD disaggregation from a set of research papers into a serving stack that thousands of teams already run in production. For anyone serving large multimodal models at scale, this is one of the most consequential inference shifts of the past year. The reason is simple and specific. Multimodal traffic broke the assumptions that made prefill-decode disaggregation work so well for text-only models.

For two years, the dominant scaling pattern for large language model inference was to split prefill from decode. Prefill is compute-bound and processes the whole prompt at once. Decode is memory-bound and generates tokens one at a time. Separating them onto distinct GPU pools removed interference and let each phase scale on its own curve. That pattern is now the industry default across vLLM, SGLang, TensorRT LLM, and NVIDIA Dynamo. It is deployed at production scale by providers serving frontier models today.

Multimodal models add a third workload that the two-phase model never accounted for. Before a vision language model can generate a single token, every image must pass through a visual encoder. That encoder has a completely different performance profile from prefill and decode. When you run all three stages on the same GPU, one large image request can stall an entire batch of text-only users. This article explains what EPD disaggregation is, how the architecture works, what the benchmarks actually show, where it helps, and how enterprise teams should approach adoption.

Why Multimodal Serving Broke the Prefill Decode Playbook

Large multimodal models look like language models with an extra input path, but their serving behavior is fundamentally different. A text-only request enters prefill, builds a key-value cache, and moves to decode. A multimodal request must first convert pixels or audio into embeddings through a separate encoder network. Only then can the language model treat those embeddings as tokens. That extra stage sits directly on the critical path to the first token.

The encoder is not a minor preprocessing step. It is a full neural network, often a vision transformer, that runs heavy matrix operations over image patches. Its cost scales with image resolution, image count, and input complexity. A single request with four high-resolution images can dominate the compute of an entire serving step. This variability is exactly what production systems struggle to schedule around.

The Three Workloads Hiding Inside One GPU

When you serve a multimodal model on a single instance, you are actually running three distinct workloads that fight for the same hardware. Each one wants a different resource ratio and a different parallelism strategy. Forcing them to share one configuration means none of them runs optimally. This is the core tension that EPD disaggregation resolves.

The three stages have sharply different profiles that matter for hardware planning.

1. The encoder is a one-shot, compute-bound stage that benefits from high parallelism and rewards raw compute throughput on dense image batches.

2. Prefill is bandwidth-hungry and dominated by large general matrix multiplications that build the initial key-value cache for the prompt.

3. Decode is heavily memory-bound, long-lived, and sequential, generating one token per step while streaming output to the user.

Because these three profiles differ, the ideal GPU count and parallelism plan for each stage differ too. A cluster sized for decode will waste money on rare image spikes. A cluster sized for image bursts will sit idle during long text conversations.

How Colocation Creates Cross-Modal Interference

Colocating the encoder with prefill and decode produces a specific failure mode. When a heavy image request arrives, its encoder work preempts ongoing token generation for other users. Text-only requests that need no encoder still wait behind the image job in the queue. The result is jittery, unpredictable latency that violates tail latency targets.

This interference is not theoretical, and it shows up clearly under load. In vLLM benchmarks, colocated multimodal workloads with several images per request stabilized around 12 to 14 queries per second. At that point, the ninety-ninth percentile time per output token spiked by 30% to 50%, breaking service-level objectives. The encoder was starving the decode stage of the compute it needed to stay smooth.

What EPD Disaggregation Actually Is

EPD disaggregation is a serving architecture that separates the visual encoder, the prefill phase, and the decode phase into independently scaled resource pools. Instead of running encode, prefill, and decode on one instance, the encoder becomes its own service. Images are encoded on dedicated hardware; the resulting embeddings are stored in a shared transport layer, and the language model instances pull those embeddings when they run prefill and decode. The three stages now form a pipeline rather than a single blocking sequence.

This is the natural extension of encoder disaggregation applied to the full multimodal pipeline. The predecessor pattern, prefill-decode disaggregation, has already proved that decoupling phases removes interference and enables independent scaling. EPD disaggregation adds the encoder as a first-class stage in that same design philosophy. The insight is that a stage with a distinct performance profile deserves distinct resources.

The immediate payoff is pipelining across requests. The encoder for one request can run while a previous request is already in prefill or decode. Text-only requests bypass the encoder entirely and never queue behind image jobs. Removing that queueing delay is what smooths tail latency and lifts sustainable throughput. At KriraAI, we treat this kind of pipeline decoupling as a core pattern when we design multimodal serving systems for enterprise clients.

Encoder Output Caching and Cross-Request Reuse

A centralized encoder service unlocks a benefit that colocated serving cannot easily provide. Because embeddings are computed in one place, they can be cached and reused across requests. A frequently seen image, such as a company logo, a product photo, or a standard diagram, is encoded once and served from the cache thereafter. Requests that hit the cache pay zero encoder cost and see a direct reduction in time to first token.

This cross-request reuse compounds at scale in real applications. Support tools, document assistants, and catalog search systems, many of them core to modern e-commerce AI solutions, repeatedly process the same images, such as product photos and logos, making encoder caching especially valuable. As the cache hit rate climbs, the encoder pool load falls, and you can serve the same traffic with fewer encoder GPUs. Caching turns a repeated cost into a one-time cost for common visual inputs.

Inside the Architecture: Router, Connectors, and the Encoder Cache

The core technical section of any EPD disaggregation design comes down to three moving parts. There is a proxy and router that orchestrates the request flow. There is a data transfer layer that stores encoder-produced embeddings. There is a connector abstraction that moves those embeddings between encoder workers and language model workers. Understanding how these fit together is what separates a working deployment from a fragile one.

The proxy sits at the front of the system and handles the split. It extracts multimodal inputs from the incoming request and creates one encoder job per input. It dispatches those jobs to encoder instances and waits for completion. Only after the embeddings are stored does it forward the original request, now carrying image hashes rather than raw pixel data, to the prefill and decode pool.

The data transfer layer is the shared medium between the two pools. Encoder workers write embeddings into remote storage, and prefill decode workers read them back. In the vLLM design, this is called the encoder cache, and the bridge to it is a family of connector objects. Connectors come in two roles: a scheduler-side role that decides which embeddings to load or save, and a worker-side role that performs the actual reads and writes.

The Request Lifecycle Step by Step

Tracing a single multimodal request makes the architecture concrete. The lifecycle is deterministic and easy to reason about once the roles are clear. Each step maps to a specific component in the system.

1. The proxy receives the request and extracts its multimodal inputs, then creates one encoder job for each image or audio clip.

2. The encoder scheduler runs those jobs, computes the embeddings, and writes them into remote storage through the connector layer.

3. The encoder workers notify the proxy once every embedding for the request has been stored successfully.

4. The proxy forwards the request, now referencing image hashes instead of pixels, to a prefill and decode instance.

5. The prefill decode instance loads the embeddings from remote storage, injects them into the model runner, and executes prefill and decode normally.

This separation means the language model instance never touches raw image data. It only consumes embeddings, which keeps its execution path close to a pure text workload. That simplicity is part of why the pattern integrates cleanly into existing serving stacks.

Where the KV Cache and Encoder Cache Differ

It is easy to confuse the encoder cache with the key-value cache, but they solve different problems. The key-value cache stores attention states for tokens already processed during prefill and decode. It is per request, short-lived, and central to autoregressive generation. The encoder cache stores image embeddings that can be shared across requests and reused over time.

This distinction matters for capacity planning and memory management. The key-value cache lives on the prefill decode pool and scales with prompt length and concurrency, which is exactly why teams pairing EPD with KV cache compression techniques often see compounding memory savings on the decode side. The encoder cache lives in the transport layer and scales with the diversity of images in your traffic. Sizing them independently is one more degree of freedom that vLLM EPD gives infrastructure teams.

EPD vs PD Disaggregation: What Changed and Why It Matters

Prefill decode disaggregation solved a two-workload problem, and EPD disaggregation solves a three-workload problem. Under the older pattern, teams split compute-bound prefill from memory-bound decode and scaled each independently. That worked because text models have exactly two phases on the critical path. Multimodal models added a third phase that the two-pool design simply had nowhere to put.

The practical difference shows up in how you provision hardware. With prefill decode disaggregation, you tune two pools against two demand curves. With EPD disaggregation, you tune three pools, and crucially, you can route the encoder phase to hardware suited for dense non-autoregressive compute. You stop overprovisioning expensive decode clusters just to absorb occasional image spikes. This is the resource efficiency argument that makes the technique attractive to finance-conscious platform teams.

There is also an important compatibility point that reduces adoption risk. EPD disaggregation does not replace prefill decode disaggregation; it extends it. Teams already running a disaggregated text stack are adding an encoder stage on top of the same architecture. Open-source serving frameworks, including vLLM and SGLang, are evolving their existing prefill-decode logic to incorporate a dedicated encoder stage rather than starting over.

The Performance Numbers: What the Benchmarks Show

Encoder disaggregation delivers its largest gains exactly where colocation hurts most, which is image-heavy traffic. The vLLM team benchmarked the pattern on four A100 80GB GPUs using a Qwen3 VL 4B model. They compared a configuration of one encoder instance and three prefill-decode instances against a plain data-parallel setup of four instances. They measured goodput, defined as the maximum request rate at which both tail latency targets were held.

The tail latency targets in the evaluation were strict and production-realistic. The system had to keep the ninety-ninth percentile time to first token under 20000 milliseconds and the ninety-ninth percentile time per output token under 100 milliseconds. Goodput was only counted while both of those service level objectives were satisfied. This is the right way to measure serving, because average latency hides the spikes that actually break user experience.

Short Text, Image-Heavy Workloads

For short prompts of roughly 400 tokens, the benefit scaled directly with images per request. With a single image, goodput improved modestly, moving from 23 to 24 queries per second. With four images per request, throughput doubled, rising from 6 to 12 queries per second. Tail latency improved as well, with the ninety-ninth percentile time to first token and time per output token often 20% to 50% lower than the colocated baseline.

The stability story is as important as the raw throughput. The colocated baseline destabilized around 12 to 14 queries per second on multi-image traffic. Beyond that point, its tail latency spiked and violated the targets. EPD disaggregation pushed that instability threshold substantially higher and kept latency curves smooth. Removing encoder and decoder interference is what buys that headroom.

Long Text, Decode Dominated Workloads

Longer prompts change the picture because encoding becomes a smaller fraction of the total work. At roughly 2000 input tokens, the system enters a decode-dominated regime. Even here, EPD disaggregation delivered clear gains rather than washing out. The baseline sustained about 8 queries per second with one image and about 4 with three or four images before violating targets.

EPD disaggregation sustained 18, 11, 9, and 8 queries per second across those same settings, which is 2x to 2.5x higher goodput. Decode throughput rose 10% to 30% across multimodal settings. Ninety-ninth percentile time to first token fell 30% to 50%, and time per output token fell 20% to 40% within stable operating regions. The pattern also proved hardware-agnostic, delivering 5% to 20% higher throughput when replicated on Ascend 910B NPUs. Research systems in the same family have reported even larger structural gains, with up to 15 times better memory efficiency and up to 22 times larger batch sizes.

Where EPD Helps, and Where It Does Not

EPD disaggregation is a targeted optimization, not a universal upgrade, and honest engineering means naming its limits. The technique earns its keep when encoding is a meaningful share of your serving cost, the same cost-benefit calculus behind Mixture of Experts inference latency optimizations like predictive expert prefetching, where added complexity only pays off when the targeted bottleneck is real. If your traffic is mostly text with rare images, the added architectural complexity may not pay for itself. The transport layer, the router, and the connector logic all introduce operational surface area that a single instance does not have.

The clearest win case is mixed traffic at volume, where image-heavy requests and text-only requests share the same endpoint. In that setting, letting text requests bypass the encoder is a large and immediate benefit. The clearest marginal case is uniformly long text conversations with occasional single images. There, the decode phase dominates, and the encoder is rarely the bottleneck, so gains shrink toward the lower end of the observed range.

There are real costs to weigh before committing. Moving embeddings between pools requires fast networking, and slow interconnects can erode the benefit. Managing a separate encoder pool adds scheduling, failure handling, and cache invalidation concerns. Teams without the operational maturity to run a disaggregated text stack will find EPD disaggregation harder still. When KriraAI evaluates this pattern for a client, we start by measuring the encoder share of latency before recommending any architectural change.

A Practical Adoption Path for Enterprise Teams

You should adopt EPD disaggregation when your production traffic mixes image-heavy multimodal requests with text-only requests at a meaningful volume. That is the profile where colocation interference is most damaging and where the pattern returns the most value. The decision should be driven by measurement, not by novelty. The right first step is to profile your current serving stack and quantify how much of your tail latency comes from encoding.

Adoption does not have to be all or nothing, and a staged approach reduces risk. You can begin by deploying encoder disaggregation for a single high-volume multimodal endpoint. You keep the rest of your traffic on the existing configuration and compare goodput directly. This gives you a controlled measurement before any broad rollout.

Deciding If Your Workload Justifies EPD

Use a short set of criteria to decide whether the pattern fits your workload before you invest engineering time.

1. Measure the fraction of requests that carry images and how many images each request typically carries.

2. Check whether image-heavy requests currently degrade latency for your text-only users during peak load.

3. Confirm you have fast interconnects, such as high-bandwidth networking, between the machines that would host the encoder and language model pools.

4. Estimate your image reuse rate, since a high rate makes encoder caching far more valuable.

5. Assess your team's existing experience operating prefill decode disaggregation or similar multi-pool serving.

If most of these points in favor, the pattern is likely to pay off. If image traffic is rare or interconnects are slow, a simpler single-instance setup may serve you better. This kind of upfront analysis is exactly the discipline that separates durable systems from fragile ones.

Rolling It Out Without Destabilizing Production

A safe rollout treats the encoder pool as a new dependency and hardens it accordingly. Start with generous headroom on encoder capacity so that a traffic spike does not stall the pipeline. Instrument the embedding transfer path closely, because that network hop is a new potential failure point. Monitor cache hit rates from day one to validate the reuse assumptions that justified the design.

You should also plan for graceful degradation when the encoder pool is under pressure. A well-designed system can fall back to colocated encoding for a subset of requests rather than dropping them. Keeping both paths available during the transition period lets you compare behavior under real load. KriraAI builds and delivers production AI systems for enterprises, and we apply this staged, measured rollout discipline precisely because emerging serving patterns need real-world validation before they carry critical traffic.

Conclusion

Three takeaways matter most for anyone weighing this technique. First, on the mechanics, EPD disaggregation works by treating the visual encoder as a first-class stage, encoding images on dedicated hardware, caching the embeddings, and letting the language model pool consume them without ever touching raw pixels. Second, where it matters, the pattern delivers its strongest gains on mixed, image-heavy traffic, where vLLM measured goodput doubling on short prompts and 2x to 2.5x improvements on longer ones. Third, on what to do, teams should profile the encoder share of their latency, pilot the pattern on one high-volume endpoint, and adopt it only where the measured benefit justifies the added operational surface.

KriraAI stays at the frontier of AI infrastructure because serving architecture is where much of the real cost and reliability of production systems are determined. We build and deliver production AI systems for enterprises, and we track research and open-source developments like EPD disaggregation closely so that we can apply them when they are genuinely ready. Our approach is not to chase novelty but to translate emerging techniques into measurable value, adopting them only after real-world validation on real workloads. If your organization is serving multimodal models and wants to understand what encoder disaggregation could mean for your latency, throughput, and cost, KriraAI can help you evaluate whether this pattern belongs in your stack.