Cutting Mixture of Experts Inference Latency with PEARL Prefetching

Sparse mixture-of-experts models promise dense-network quality at the compute cost of a small model. In practice, that promise breaks at serving time. When experts do not all fit in fast GPU memory, the model must stream expert weights on demand. Under real multi-tenant traffic, the set of needed experts shifts constantly. The result is expert cache thrashing, and the mixture of experts inference latency becomes unpredictable and slow.

Existing serving stacks treat expert loading as a reactive problem. They fetch an expert only after the router selects it, then evict using a least recently used policy. This works when traffic is stable and one domain dominates. It fails when concurrent requests span code, chat, math, and retrieval at once. Under that load, the resident expert set churns, and host-to-device transfers dominate the critical path.

At KriraAI, we approached this from the opposite direction. Instead of reacting to expert demand, we predict it. Our research contribution is PEARL, short for Predictive Expert Affinity and Residency Lookahead. PEARL forecasts which experts each in-flight sequence will need over a short horizon. It then co-schedules requests by expert affinity and manages residency using those forecasts. In our experiments, this cut the p99 inter-token latency by 2.8 times under domain-shifting load.

This post shares the research in full. We first explain the mechanism behind expert cache thrashing. We then present the PEARL architecture component by component. We report our experimental setup, results, ablations, and failure cases. We close with the limitations we still face and the questions we are pursuing next.

The mechanics of expert cache thrashing

Expert cache thrashing is the repeated eviction and reload of the same expert weights within a short window. It happens whenever the working set of active experts exceeds the residency budget in fast memory. On a memory-constrained accelerator, only a fraction of experts can stay resident. The rest live in host memory or a slower tier. Every miss forces a transfer across the interconnect before the layer can compute.

Expert cache thrashing gets worse under multi-tenant LLM serving because the working set widens. A single domain activates a stable, narrow band of experts. Mixing four domains in one batch unions four such bands. The resident budget cannot hold them all, so misses cascade. Latency then tracks interconnect bandwidth rather than compute.

Why reactive loading fails

Reactive loading is the default in most open MoE serving stacks. The router computes gate logits, selects the top experts, and only then requests their weights. If an expert is not resident, the request stalls on a transfer. This places a memory operation directly on the token-critical path. At decode time, where each step is already latency-bound, that stall is expensive.

The deeper flaw is that reactive loading discards information. The hidden state entering the router already carries a strong signal about which expert will fire. Reactive systems ignore that signal until the router formalizes it. By then, there is no time left to prefetch. We saw this as a missed opportunity worth exploiting.

Why static placement breaks under domain shift

Static expert placement pins a chosen subset of experts in fast memory permanently. It assumes the popular experts are stable. That assumption holds only when traffic is stationary. Production traffic is not stationary. Tenant mix, time of day, and prompt type all move the popularity distribution.

We measured expert activation across a mixed trace on a 64-expert model. The top 8 experts by frequency covered 71 percent of activations during code-heavy windows. During math-heavy windows, the same 8 experts covered only 39 percent. Any static pin tuned for one window is wrong for the other. This churn is the structural reason static schemes underperform.

PEARL: predictive expert affinity and residency lookahead

PEARL rests on one empirical observation. Expert activation is not random. It is autocorrelated in time within a sequence and semantically clustered across sequences from the same domain. If activation is predictable, then residency can be planned rather than reacted to. That single shift, from reaction to prediction, is what lets us attack the mixture-of-experts inference latency at its root.

PEARL has three coupled components. The first predicts expert demand per sequence over a short horizon. The second group's requests are broken down into micro-batches that share experts. The third manages fast-memory residency using the predictions. We describe each in turn, then give the training objective and cost.

The Trajectory Expert Predictor

The Trajectory Expert Predictor is a small learned head attached before each MoE layer. It reads the pre-router hidden state and outputs an expert demand forecast. The forecast is a probability per expert for each of the next K decode steps. We implement it as a two-layer gated recurrent unit over the hidden state trajectory. Its output head is a per-expert sigmoid, so the task is multi-label rather than a single softmax.

We chose a recurrent head deliberately. The hidden state trajectory carries momentum that a single-step classifier cannot see. A sequence of writing code tends to keep writing code, and its expert band persists. The recurrent state captures that persistence cheaply. We rejected a transformer head because its cost scaled poorly against the tiny accuracy gain.

The predictor is deliberately small. It adds 0.4 percent to model FLOPs and 0.3 milliseconds per decode step on our hardware. It is trained offline, so it never touches the pretraining budget. We predict a horizon of K equal to 4 decode steps by default. Beyond that horizon, accuracy decays faster than the prefetch benefit grows.

The Affinity Co-scheduler

The Affinity Co-scheduler turns per-sequence forecasts into batches. Its goal is to minimize the union of experts a micro-batch requires. Fewer distinct experts per batch means a smaller resident working set. We formulate batch formation as an online set-cover-style problem. Because decisions must be made in microseconds, we solve it greedily rather than optimally.

The scheduler maintains a candidate pool of ready sequences. It seeds a batch with the sequence whose predicted expert set is largest. It then adds sequences ranked by Jaccard overlap with the batch working set. It stops when the batch fills or overlap drops below a threshold. The threshold trades scheduling latency against residency efficiency.

We add a fairness term to prevent starvation. Each waiting sequence accrues an aging bonus over time. The bonus eventually overrides pure affinity, so no tenant waits indefinitely. This coupling of affinity and aging is the part practitioners most often need to tune. We expose it as a single scalar for that reason.

The Predictive Residency Controller

The Predictive Residency Controller decides which experts stay resident. Reactive systems use least recently used eviction, which only looks backward. Our controller looks forward to using the predictor's forecasts. It prefetches experts along the predicted horizon, a form of predictive expert prefetching, before they are needed. It evicts using a cost-aware score rather than pure recency.

The eviction score approximates the optimal Belady policy. For each resident expert, we estimate two quantities. The first is the predicted time to the next use, taken from the horizon forecast. The second is reload cost, which depends on weight, size,e and transfer tier. We evict the expert with the highest ratio of time to next use over reload cost.

The three components reinforce each other. Better forecasts sharpen both scheduling and eviction. Tighter batches shrink the working set the controller must hold. A well-managed cache, in turn,n lowers the penalty when a forecast misses. This tight coupling is why PEARL outperforms any single trick in isolation.

Training objective and cost

We train the predictor by distillation from observed router decisions. We log gate selections from the frozen base model on a representative trace. Each logged step yields a multi-label target over experts. The loss is a horizon-weighted binary cross-entropy. Nearer horizon steps receive higher weight through a decay factor gamma.

We add a calibration term to the objective. Well-calibrated probabilities matter because the residency controller consumes them as costs. An overconfident forecast triggers wasteful prefetches that pollute the cache. We penalize miscalibration with an expected calibration error regularizer. This kept prefetch precision stable across domains in our runs.

Experimental setup

Models and workloads

We evaluated PEARL on two mixture-of-experts models. The first was Mixtral 8x7B, an 8-expert model with top-2 routing. The second was a custom 64-expert, 32-layer model we trained internally at KriraAI. We chose the pair to test both low and high expert counts. Prediction is easier with 8 experts and harder with 64, so the pair stresses the predictor.

Our workload was a synthetic multi-tenant trace built to mirror production. Requests arrived by a Poisson process across four domains. The domains were code, open chat, mathematical reasoning, and retrieval-augmented answering. We varied the domain mixture over time to induce distribution shift. This shift is exactly the condition that triggers expert cache thrashing.

Baselines, metrics, and hardware

We compared PEARL against four baselines. Each represents a real serving strategy in wide use today. Together, they span reactive, cached, static, and sharded approaches.

1. On-demand reactive loading fetches each expert only after the router selects it.

2. Least recently used caching keeps a fixed budget resident and evicts by recency.

3. Static pinning holds the globally most popular experts in fast memory permanently.

4. Expert parallelism shards experts across devices in the style of DeepSpeed MoE.

We measured five quantities per run. We tracked p50 and p99 inter-token latency to capture tail behavior. We track the expert cache hit rate as the core mechanism metric. We tracked host-to-device transfer volume in gigabytes. We tracked sustained throughput in tokens per second under a fixed latency target.

All runs used a single NVIDIA A100 80GB accelerator. We deliberately capped the resident expert budget at 40 percent of the total experts. This forced genuine memory pressure rather than a trivially cached setup. Host memory served as the slower tier over PCIe. We repeated each configuration across five traces and reported medians.

Results and analysis

Latency and cache behavior

PEARL reduced expert cache thrashing sharply at a fixed residency budget. Miss rate fell from 34.2 percent under least recently used caching to 9.7 percent. That is a 71 percent relative reduction in misses. Equivalently, the expert cache hit rate rose from 65.8 percent to 90.3 percent. Because misses drive transfers, latency followed closely.

We observed a 2.8-fold reduction in p99 inter-token latency under domain-shifting load. The tail improvement mattered more than the median. pP50latency improved by a factor of 1.6 times. The gap tells us PEARL mostly removes the worst stalls, not the typical step. Host-to-device transfer volume fell by 61 percent. Throughput rose by 1.9 times at a fixed p99 target.

The predictor was accurate enough to be useful without being perfect. It recovered the true top 2 expert set with 87.4 percent recall at horizon one. Recall decayed to 62.1 percent at horizon four. This decay is why we cap the default horizon at four. Beyond it, prefetches based on stale forecasts began to pollute the cache.

Ablation study

We ablated each component to isolate its contribution. Each row below removes one part of PEARL. The resulting miss rate shows what that part was worth.

1. Full PEARL reached a 9.7 percent expert cache miss rate.

2. Removing the residency controller and reverting to recency eviction raised misses to 16.4 percent.

3. Removing the co-scheduler while keeping prediction raised misses to 21.8 percent.

4. Removing the predictor and using current-step routing only raised misses to 28.6 percent.

5. The plain least recently used baseline sat at 34.2 percent.

The ordering revealed something we did not expect. The co-scheduler contributed more than the predictor at low residency budgets. Grouping requests by shared experts shrank the working set before any prefetch fired. Prediction then refined an already smaller problem. The two together were superadditive rather than merely additive.

Where PEARL underperforms

PEARL is not free, and it does not always help. On a fully domain-homogeneous trace, thrashing barely occurs. There, the predictor and scheduler add roughly 3 percent overhead for no benefit. We also saw cache pollution under sudden out-of-domain bursts. When traffic jumped to an unseen domain, forecasts miscalibrated, and prefetches wasted bandwidth. The calibration regularizer softened this failure but did not remove it.

What predictable expert demand means for MoE serving

Our central finding reframes MoE as a prediction problem. The field has largely treated expert loading as a caching problem. Caching asks what to keep given past accesses. Prediction asks what to fetch given likely futures. The shift from the first framing to the second is where most of our gains came from.

The result also says something about MoE models themselves. Their routers are more predictable than their design suggests. A mechanism meant to be flexible behaves with strong local regularity. That regularity is a resource, not a nuisance. Serving systems should be built to exploit it.

For teams running production systems, the practical lesson is concrete. Expert offloading does not have to mean unpredictable tail latency. With a lightweight predictor, memory-constrained hardware can serve large MoE models within tight SLOs. This lowers the hardware bar for deploying frontier-scale sparse models. At KriraAI, we see this directly in client deployments where GPU budgets are fixed.

The finding connects to a wider theme in efficient inference. Many bottlenecks that look like hardware limits are really information limits. We were bandwidth-bound not because bandwidth was scarce but because we fetched blindly. Adding a small amount of foresight converted a bandwidth problem into a compute problem. That trade is almost always favorable on modern accelerators.

Limitations and future work

Our approach carries real assumptions that will not hold everywhere. PEARL depends on temporal and semantic autocorrelation in expert activation. Against adversarial or deliberately randomized routing, its advantage collapses. It also needs an offline trace to train the predictor. On a genuinely new domain, the predictor starts cold and forecasts poorly until it adapts.

Several design choices narrow our current claims. We evaluated top-2 routing, where the expert set is small and predictable. Higher top k values widen the set and weaken the prediction signal. Our affinity co-scheduler can also conflict with strict fairness guarantees. Batching by shared experts can delay a lone out-of-domain tenant.

We are pursuing four directions to close these gaps. The first is online adaptation, so the predictor updates during serving. The second is fairness-aware co-scheduling with formal latency bounds per tenant. The third is extending PEARL to expert-choice routing and grouped query MoE variants. The fourth is combining predictive residency with quantized expert weights to further reduce transfer cost.

Conclusion

This research makes three contributions we believe matter. First, we reframed the source of mixture-of-experts inference latency as a prediction failure rather than a caching failure. Second, we designed PEARL, a system that forecasts expert demand, co-schedules by affinity, and manages residency with foresight. Third, we showed that the approach reduces p99 inter-token latency by 2.8x. It also cut expert cache misses by 71 percent under realistic load.

The larger lesson is that MoE routers are far more predictable than their flexibility implies. Serving systems that treat predictability as a resource can run large sparse models on modest hardware. This lowers the cost of deploying frontier-scale models in real products. For enterprises, that difference often decides whether a capable model is affordable to serve at all.

At KriraAI, this work is one piece of a broader research program. We study efficiency, reasoning, and reliability in production AI systems, then apply what we learn to real client deployments. We publish our findings openly because serious applied research should be shared, not hidden. If you are working on MoE serving, expert offloading, or multi-tenant LLM serving, we would like to hear from you. Reach out to discuss the results, challenge them, or explore where PEARL might fit your own systems.