On June 8, 2026, a team spanning NYU, the University of Maryland, Princeton, Columbia, Harvard, Lawrence Livermore National Laboratory, and Modal published a result that quietly resets the economics of long-context inference. Their paper on LLM context compression trains a 0.6B encoder against a 4B decoder on more than 350 billion tokens, at compression ratios of 1:4, 1:8, and 1:16. On RULER at 4k context and LongBench at 64k, the resulting models deliver better accuracy-latency and accuracy-memory curves than KV cache compression baselines, including roughly 8.8 times faster time-to-first-token on RULER at higher ratios while keeping or improving quality.
That number matters because of what it replaces. For two years, the default answer to expensive long context has been KV cache compression: prefill the whole context, then evict, mask, or compact the cache. The approach works on paper. It also forces you to pay the full quadratic prefill before you save anything, and many of these methods are not widely adopted in inference engines such as vLLM or SGLang because their eviction patterns break the assumptions those engines make about cache shape.
Soft-token compression has always been the theoretically cleaner alternative. Encode the raw context into a short sequence of continuous embeddings, hand those to the decoder, and skip the long prefill entirely. Until now, it has lost on quality. Prior soft-token compressors either degraded the base model badly or only worked on narrow, in-domain tasks they had been fine-tuned for.
This blog covers what changed, why it changed, and what an engineering team should actually do about it, including how KriraAI's AI consultancy team approaches evaluating techniques like this before recommending them. We walk through the architecture, the training recipe, the benchmark evidence, the agentic scaffolding, a concrete evaluation plan, and the limitations that the paper names honestly.
KV cache compression reduces memory by discarding cached activations after the model has already computed them. SnapKV drops entries with low aggregated query attention on a per-head basis. KVzip scores each key-value pair by the maximum attention it receives during teacher-forced context reconstruction, then prunes the low scorers. Expected Attention approximates future queries with a Gaussian and prunes keys with the lowest expected contribution.
Each of these is clever. All of them share a structural problem: the compression happens after the expensive part. KV cache compression methods appear as nearly vertical lines on accuracy-versus-latency plots because their compression time is largely independent of the target compression ratio. You choose 16x instead of 4x, and your time-to-first-token barely moves. The compute was already spent.
The deployment story is worse than the latency story. Methods like KVzip allocate eviction budgets non-uniformly across attention heads and layers, so they cannot reduce the sequence-length dimension of the cache at all. Instead, they mask evicted positions during attention, which forfeits the memory and throughput benefits that paged-attention engines exist to provide. Some methods further require prefill passes that are two to three times longer than the original input.
There are second-order costs too. Query-aware methods such as SnapKV produce caches tuned to one prompt, which limits reuse across conversational turns. Offline approaches sidestep this by amortizing compression per corpus. Cartridges distills a fixed-size KV cache per corpus, but an in-context-learning-quality cache takes roughly 30 minutes on an 8xH100 node for an 8B model. That is a viable trade for a static knowledge base and a non-starter for arbitrary user uploads.
The central claim of the paper is not that encoder-decoder compression is new. It is that the encoder-decoder compression was undertrained, and that scale plus a staged recipe closes the quality gap that made everyone abandon it. The authors call the resulting family Latent Context Language Models.
An LCLM is deliberately simple. Given input tokens and a compression ratio N, the encoder maps each contiguous block of N tokens to a single latent vector.
The encoder processes the input in fixed windows of W tokens per forward pass, producing final hidden states for every token in the window.
A pooling operator aggregates those hidden states into one latent token per block of N inputs, so a 16,384-token window at N equal to 16 yields 1,024 latents.
An adapter projects the latent sequence from the encoder hidden dimension into the decoder embedding dimension, since the two models differ in width.
The decoder consumes the projected latents in place of the original tokens, using its standard KV cache and standard decoding path.
The consequence is the whole point. Compression happens before the decoder ever runs, over small parallel windows, using a model roughly seven times smaller than the decoder. Higher compression ratios therefore reduce decoder work directly, which is exactly what KV eviction cannot do. Because the method preserves the standard KV cache structure, prefill and generation can be further accelerated with vLLM, SGLang, and efficient attention implementations.
The training data design is where prior work went wrong, and the paper is specific about it. Earlier approaches used a simple split where the first half of a sequence is compressed and the second half is trainable. LCLM training instead partitions each sequence into alternating compressed and uncompressed segments, wrapping compressed spans in special memory-start and memory-end tokens, and computing next-token prediction loss only on the uncompressed tokens.
Distributing compressed spans throughout the sequence teaches the model to condition on latent context at arbitrary positions rather than only at the beginning. The team pairs this with an auxiliary reconstruction objective that asks the decoder to reproduce compressed text verbatim, drawn from code, prose, math, and LaTeX. Training on reconstruction alone collapses the representation and destroys every other capability, while training on next-token prediction alone yields representations that lack fine-grained fidelity for exact-string tasks. The mixture is what produces a general compressor.
At KriraAI, where we build and deploy production AI systems for enterprise clients, this detail is the one we flag first with engineering teams. The reason earlier context compression failed in production was rarely the architecture. It was a training objective that quietly specialized the model to one task shape.
Rather than assume design choices inherited from prior papers, the authors ran a controlled sweep. They initialized Qwen3-0.6B randomly for both encoder and decoder, trained every variant end-to-end on 38 billion tokens at 16x compression, and compared pre-training loss. The clean-room setup matters because pre-trained encoders carry baked-in biases about pooling and masking.
Most prior soft-token work appends special pooling tokens, EOS or CLS style, and reads their final hidden states as the compressed representation. Mean pooling consistently improves pre-training loss over token-based pooling and is empirically indistinguishable from concatenation at small scale. At full scale, the picture splits by ratio. Concatenation outperforms mean pooling at low compression ratios while mean pooling wins at high ratios, and the gap narrows as context length grows.
Three further results run against received wisdom in this subfield.
Encoder window size matters a lot, and the common choice of setting W equal to N is badly suboptimal. Moving from 16 to 256 produces a large loss improvement, with a smaller additional gain at 1024.
Causal masking in the encoder consistently beats bidirectional masking, even though compression runs on the prompt before decoding and prefix-LM-style bidirectional attention is available.
A plain MLP adapter achieves lower pre-training loss than an attention-based adapter while using less compute, contrary to earlier published findings.
Boundary overlap, where each window sees neighboring tokens, does not improve loss and meaningfully increases encoder compute.
The final scaled configuration uses Qwen3-Embedding-0.6B as the encoder, Qwen3-4B-Instruct-2507 as the decoder, W equal to 1024, causal masking, an MLP adapter, and mean pooling. Initializing the encoder from an embedding model outperforms initializing from a plain language model, though the gap narrows with longer training.
Training itself runs in four stages: adapter warm-up with everything else frozen, encoder unfreezing, end-to-end continual pre-training at a small decoder learning rate, then supervised fine-tuning. Training the full model end-to-end from the start underperforms because the decoder is not yet accustomed to the encoder outputs, and both gradient magnitudes blow up. LoRA and frozen-decoder variants substantially underperformed full-parameter training.
Numbers first, interpretation second. The evaluation suite is RULER, LongBench, LongHealth, and GSM8K, with all measurements on a single H200.
At 16x compression, which discards 93.75 percent of input tokens, LCLM scores 75.06 on RULER at 4k against 94.41 for the uncompressed Qwen3-4B baseline. At 8x, it reaches 85.42, and at 4x, it reaches 91.76. On LongBench English at 4x, LCLM scores 46.04 against the uncompressed 45.21. On LongHealth at 4x, the concatenation variant reaches 82.50 against an uncompressed 75.75, which is a compression method beating its own baseline on clinical document QA, a pattern that also shows up in document-heavy compliance and risk workloads common to enterprise AI in finance.
GSM8K is short and information-dense, and the authors compress the entire prompt rather than just a long context block. This is the setting where compression should fail hardest, and where the baselines collapse completely. At 16x compression, LCLM scores 81.05 exact match. SnapKV scores 0.08. KVzip scores 0.00. Expected Attention scores 0.08. The uncompressed model scores 93.25.
That gap is the strongest evidence in the paper. KV eviction heuristics assume redundancy exists to be evicted. When every token carries a signal, there is nothing to drop, and a heuristic that drops anyway destroys the task. A learned compressor is doing something categorically different: it is re-encoding information, not discarding it.
The scaling behavior is where the deployment case is made. Across context lengths from 4K to 1M, Attention Matching runs out of memory at 1M tokens and fails at 512K due to numerical instability in its linear solver, while every other baseline runs out of memory at both 512K and 1M. LCLM peak memory stays nearly flat for the 16x model between 128K and 512K tokens, because encoder activations dominate before decoder prefill takes over. With an encoder batch size of 128 and a window of 1024, each batched encoder pass processes 131,072 input tokens.
The most commercially relevant section of the paper is the shortest. The authors equip an LCLM with a single EXPAND tool, segment the input into 512-token chunks, compress each chunk, and give every chunk an integer identifier. The model receives the entire compressed corpus in one prompt and can call EXPAND on any chunk to retrieve the original text.
This inverts how retrieval normally works. Conventional agents search first and read second, which fails when the right keyword is not known in advance. The paper's own example is precise: a bug reported in a dashboard login flow may originate in an entitlement module that never mentions dashboard or login. Lexical and semantic search both miss it. An agent that has skimmed the whole repository in compressed form does not.
On needle-in-a-haystack tasks from RULER, the agentic harness substantially improves over the raw 16x-compressed model and, in some settings, matches uncompressed context performance. The mechanism is coarse global visibility plus on-demand fine-grained expansion. For teams building code agents, contract review systems, or multi-document research tools, this is the pattern worth prototyping. KriraAI has been tracking exactly this class of technique for clients whose agents are throttled by retrieval recall rather than by model quality.
The models and code are public. The weights sit on Hugging Face under the latent-context organization, and the code is on GitHub at LeonLixyz/LCLM. Text to compress is wrapped between memory-start and memory-end tokens in the prompt, and the repository ships a reference Hugging Face inference path plus a two-stage vLLM pipeline where the encoder writes latents to a file that the vLLM decoder then reads.
A disciplined evaluation looks like this.
Pick the workload where long-context inference cost is actually hurting, and instrument current time-to-first-token and peak GPU memory at your real p95 context length.
Replay 200 to 500 production prompts through the 4x checkpoint first, since 4x is where quality is closest to uncompressed, and the deployment path is the least surprising.
Measure task accuracy with your own rubric, not RULER, because synthetic retrieval scores do not predict domain performance.
Only then test 8x and 16x, and treat the accuracy drop as a purchasing decision rather than a regression.
If your workload is agentic, build the EXPAND harness before judging the raw 16x numbers, because the compressed-plus-expand configuration is the one that competes with uncompressed context.
Two practical constraints shape the pilot. The published family is a 0.6B encoder paired with a 4B decoder, so if your production model is a frontier-scale MoE, this is not a drop-in swap. You would be evaluating the technique, not adopting these weights. And the vLLM path is two-stage rather than a single fused server, which means your serving layer needs to handle an encode step before the decode request.
The paper is unusually candid, and the limitations are real.
Quality at 16x is not free. RULER at 4k drops from 94.41 to 75.06, and LongBench English drops from 45.21 to 39.08. For exact-recall workloads at high ratios, you are trading measurable accuracy for memory. The 4x tier is close to lossless. The 16x tier is not, and marketing it as such would be dishonest.
Scaling behavior is mixed, and the authors say so. Increasing decoder size produces a much larger drop in pre-training loss than increasing encoder size, but the 8B decoder achieves substantially lower loss without delivering the expected downstream gains. The 0.6B encoder with the 4B decoder scores 75.06 on RULER 4k, while the 0.6B with the 8B configuration scores only 71.20. Lower loss did not translate into better task performance, which the authors attribute partly to a mismatch between their data mixture and the 8B initialization.
There are ecosystem constraints too. The technique requires continual pre-training on hundreds of billions of tokens, which is not something most enterprises will replicate. Full-scale runs used 32 nodes of MI300A GPUs. That places the technique in a category KriraAI thinks about carefully with clients: adopt the released artifacts; do not attempt to reproduce the pipeline unless model training is your business.
Finally, compression applies to input context only in this release. The authors flag extending it to the generated state, including long chain-of-thought and accumulated tool observations, as future work. In long-horizon agents, the generated state often dominates the budget.
The strategic read is that the industry's two answers to long context are converging from opposite directions, echoing a broader shift we've covered in our analysis of inference-time compute scaling, where compute budgets migrate from training to inference. Architecture-side work compresses attention inside the model, a lineage that includes the LatentMoE architecture, which applies the same latent-projection philosophy to Mixture of Experts routing rather than to input context. DeepSeek-V4 interleaves Compressed Sparse Attention and Heavily Compressed Attention across transformer layers, cutting the KV cache to 10 percent of DeepSeek-V3.2 at 1M tokens. LCLMs compress the input before it reaches the model at all. These are orthogonal, and the paper says explicitly that they can in principle be composed.
For system designers, this reframes the question. The choice is no longer between a bigger context window and an RAG pipeline. It is a tiering decision across raw tokens, learned latents, and retrieval, allocated by information density. Everything that must be recalled exactly stays raw. Everything that provides orientation gets compressed. Everything cold stays in an index.
The near-term trajectory is visible in the paper's own conclusion. Adaptive compression that varies granularity by input perplexity would remove the need for explicit expansion tools. Compressing the agent's working history rather than only its inputs would address the budget that actually grows. Neither is shipped. Both are obvious enough that competing labs are almost certainly working on them.
Three things are worth carrying away from this. First, on mechanism: LLM context compression works now because a small causal encoder with mean pooling, an MLP adapter, and 1024-token windows was trained end-to-end on 350 billion tokens with interleaved compressed spans and an auxiliary reconstruction objective. The architecture was never the bottleneck. The training format was.
Second, on where it matters: the value concentrates in long-context inference cost for agentic and document-heavy workloads, where 4x compression is close to free, and the compressed-plus-expand harness solves retrieval failures that keyword and embedding search cannot. It matters much less for short prompts served by small models.
Third, on what to do: run the 4x checkpoint against your own p95 prompts this quarter, measure time-to-first-token and peak memory honestly, and treat higher ratios as a deliberate accuracy purchase. Do not attempt to reproduce the training pipeline.
KriraAI builds and delivers production AI systems for enterprises, and staying at the research frontier is part of that job rather than a side activity. The discipline we hold to is that a technique earns deployment when it produces measurable value, not when it is interesting. Latent context compression is at the stage where it deserves a rigorous pilot and not a rewrite. If long context economics are shaping what your systems can do, we would welcome a conversation with KriraAI about what this technique could mean for your architecture.
KV cache compression reduces memory after the model has already prefilled the entire context by evicting, masking, or compacting cached key-value entries. LLM context compression with an encoder-decoder compressor works before the decoder runs, mapping raw tokens into a shorter sequence of latent embeddings. The practical difference is where the compute is spent. KV eviction still pays the full quadratic prefill, so higher compression ratios barely improve time-to-first-token. Latent compression shortens the sequence the decoder sees, so a 16x ratio directly reduces decoder compute and memory. It also preserves standard cache structure, which keeps it compatible with vLLM and SGLang.
It depends entirely on the ratio and the task. At 4x compression, quality is close to lossless and sometimes better than the uncompressed baseline, with LongBench English at 46.04 versus 45.21 and LongHealth at 82.50 versus 75.75. At 8x, RULER at 4k drops to 85.42 from 94.41. At 16x, which discards 93.75 percent of tokens, RULER at 4k falls to 75.06. On information-dense short prompts, the picture is more favorable relative to alternatives, with GSM8K at 81.05 for 16x compression against near-zero scores for SnapKV, KVzip, and Expected Attention.
Partly, and with caveats. The released checkpoints pair a 0.6B encoder with a 4B decoder, so they suit workloads already served by mid-size models rather than frontier MoE deployments. Where they fit, the gains are substantial: roughly 8.8 times faster time-to-first-token on RULER at 4k, 5.2 times faster on LongBench at 64k, and peak memory that stays flat between 128K and 512K tokens while every KV cache baseline runs out of memory at 512K on a 141GB H200. Teams should benchmark against their own prompts before assuming those figures transfer.
This is one of the paper's counterintuitive findings, and the authors report it as empirical rather than theoretically explained. Compression runs on the prompt before autoregressive decoding begins, so bidirectional attention is architecturally available, and prefix-LM precedent suggests it should help. In the controlled from-scratch sweep at 16x compression, causal masking nonetheless produced consistently lower pre-training loss, and the result held at full scale across the continual pre-training runs. The most plausible reading is representational alignment: a causally masked encoder produces latents whose information ordering matches what a causal decoder expects to consume.
For agents specifically, the compressed-context-plus-expansion pattern is the strongest option in the current literature. The agent receives an entire corpus as compressed latents in 512-token chunks, each with an integer identifier, and calls an EXPAND tool to retrieve the original text on demand. This substantially improves needle-in-a-haystack accuracy over raw 16x compressed context and sometimes matches uncompressed performance. It solves a failure mode that lexical and semantic retrieval cannot: finding relevant code or clauses whose vocabulary never matches the query. Query-aware KV methods like SnapKV are poorly suited here because their caches do not generalize across turns
Ridham Chovatiya is the COO at KriraAI, driving operational excellence and scalable AI solutions. He specialises in building high-performance teams and delivering impactful, customer-centric technology strategies.
