LLM Deployment in Production: Cost, Latency, and Scaling Guide

Picture a freight-audit software company we will call Ridgeway (an illustrative example, not a client), whose new assistant reads a carrier invoice and tells a support rep why one pallet was billed twice. On a laptop the prototype was right most of the time, yet LLM deployment for Ridgeway meant something harder: 40,000 of those questions a day, a first word on screen in under a second, and a monthly bill below $12,000. If you want to see a gap like that closed on a real project, our LLM inference optimization case study walks through one.

Every Ridgeway number below is illustrative arithmetic you can swap for your own, while every outside figure names its source. Engineering leaders usually need three answers before sign-off: what one request costs, what latency the product can promise, and whether renting tokens or running GPUs suits the next year. Each section works one of those through with Ridgeway's invoice traffic as the test case.

What LLM deployment involves when 40,000 questions a day arrive

A model counts as deployed when it carries four commitments at once: a latency promise, a cost ceiling, an engineer who gets paged, and a safe way to ship changes, none of which Ridgeway's notebook prototype had. The weights stayed identical between that prototype and launch, while almost everything around them was rebuilt. That surrounding layer covers the inference engine (vLLM, SGLang or NVIDIA TensorRT-LLM when Ridgeway hosts its own models), a gateway with rate limits, retrieval of carrier rules, and logs that show finance the spend per route.

The four measurements Ridgeway tracks from day one

Ridgeway instruments four measurements before the first rep logs in, because nearly every later serving decision depends on them. Each is tracked per route as a p95 value, the figure 95 percent of requests beat. Averages would hide the slowest twentieth of answers, which are the ones reps complain about.

How one invoice question travels through the serving stack

A production request passes through about 8 stages, and only 2 of them run the model itself. In Ridgeway's stack the other 6 decide whether those 2 are fast and cheap. Here is the path one billing dispute takes.

1. The gateway confirms the rep's login, applies a per-team rate limit, and tags the invoice question with an ID used in every log line.

2. A router sends shipment-status lookups to a small model and billing disputes to the larger one.

3. A response cache returns a stored reply when the same rep asked about the same invoice within the past hour.

4. Retrieval fetches the carrier's rate rules, and Ridgeway's fixed 2,200-token instructions go first in the prompt so the provider can reuse them.

5. The scheduler drops the invoice question into Ridgeway's running batch at the next decode step instead of holding it for a new batch.

6. Prefill reads all 3,000 input tokens in parallel and builds the key-value cache, the model's working memory for this conversation.

7. Decode writes the dispute explanation one token at a time and streams each fragment to the rep.

8. An output check confirms every dollar amount matches the invoice, then the trace is logged with token counts and cost.

Why the 3,000-token prefill and the 350-token decode need different fixes

Prefill is compute-heavy because the GPU handles every input token at once, so Ridgeway's 3,000-token prompts mostly delay the first word. Decode is memory-bound, since each new token means reading the weights plus the growing key-value cache, so long replies mostly hurt completion time and throughput. Trimming the 2,200-token instruction block therefore speeds up the first word, while capping explanations at 250 tokens shortens full answers and frees GPU memory for other reps.

Three hosting models compared for a 40,000-question workload

Teams putting a model in front of users choose between a pay-per-token API, a dedicated endpoint a provider runs for them, or open-weight models on GPUs they control. For Ridgeway the deciding factors are client data rules, traffic shape and in-house skills. The table frames each option through those three factors.

Hybrid setups are common and often the smartest start, with disputes on a frontier API and a small self-hosted model answering the status lookups that make up about 60 percent of Ridgeway's traffic. Contracts can settle the question before cost does, because a shipper moving goods for a defense contractor may forbid invoice text from leaving a named cloud region. Read client contracts first, since careful cost modeling will not rescue an option the legal terms rule out.

Working out LLM inference cost for a 3,000-token prompt

One request costs its input tokens times the input rate plus its output tokens times the output rate, minus caching and batch discounts, so a 3,000-token prompt with a 350-token reply is easy to price before launch. Filling that in early catches most budget surprises. The arithmetic below assumes placeholder rates of $3 per million input tokens and $15 per million output tokens, so swap in your provider's current price list.

Ridgeway's $20,880 monthly bill, line by line

Without discounts Ridgeway's bill reaches $20,880 a month, and caching plus batch pricing brings it to about $11,940 with identical answers for reps. Caching carries the biggest share because the instruction block is long and the same on every call. On most of its models, Anthropic's pricing documentation bills cache reads at one tenth of the base input rate and a five-minute cache write at 1.25 times base, the multipliers applied here, but a timestamp or rep name placed above the instructions breaks every hit.

The overnight job has no live reader, which suits asynchronous pricing. OpenAI's Batch API FAQ describes a 50 percent discount on bulk work given a 24-hour completion window, and other large providers run comparable programs. After both discounts, output becomes Ridgeway's largest line at $6,300, so trimming the average reply from 350 to 250 tokens with a three-line summary format saves roughly $1,800 a month more.

The point where renting GPUs beats renting tokens

Self-hosting saves money when heavy, steady traffic keeps GPUs busy and an open-weight model clears the quality bar, and it loses money when GPUs idle. At a placeholder $2.50 per GPU-hour, two always-on GPUs (the minimum for surviving one failure) cost about $3,600 a month, which undercuts API spend on Ridgeway's status-lookup route if an 8-billion-parameter-class model passes the evaluation set. If disputes need a model spanning eight GPUs, the same arithmetic gives $14,400 a month, above Ridgeway's entire optimized API bill, unless a distilled model trained on Ridgeway's own dispute transcripts can do the job on two.

Engineering time belongs in that comparison too. Someone has to patch drivers, upgrade the serving engine and answer the 2 a.m. page, and in many companies that work costs more than the GPUs. Teams below a few thousand requests a day rarely recover the overhead, which is an honest reason to stay on an API for the first year.

A 700 ms latency budget you can defend in a planning meeting

A defensible latency target splits each response into time to first token and time to finish, then sets a p95 goal for both. Every pipeline stage gets a slice, and for Ridgeway's first streamed word the slices add up to 700 ms. The figures below are illustrative, and writing them down gives product managers a visible millisecond price for every prompt they want to grow.

Six techniques ranked by the milliseconds they buy back

Streaming comes first because a rep reading the opening line of a dispute explanation feels speed long before the reply ends, at no extra compute. Prompt caching follows, since reusing Ridgeway's 2,200-token prefix skips much of the prefill work as well as cutting price. Routing status lookups to a small model shortens the queue for disputes, quantization to 8-bit or 4-bit weights reduces memory traffic per token (invoice arithmetic needs re-testing afterward), and semantic caching, which matches a new question to a stored one with similar meaning, suits policy questions but not invoice totals that differ by a single digit.

Speculative decoding has a lightweight helper model propose a run of tokens that the main model accepts or rejects in one forward pass, so the output stays the same. In the original Google Research paper by Leviathan, Kalman and Matias (ICML 2023), a T5-XXL model ran 2 to 3 times faster than the standard T5X implementation with identical outputs. Gains depend on how often the helper guesses right, so measure the acceptance rate on Ridgeway-style invoice prompts before counting on the speedup.

LLM scaling for a Monday 9 a.m. peak and near-empty nights

Scaling a model server works best when you size for concurrent sequences held in GPU memory rather than requests per second, batch continuously, and add replicas on queue depth instead of CPU load, which for Ridgeway means planning around 20 sequences at peak. CPU and even GPU utilization mislead because a GPU can look busy with an empty queue, while queue depth and key-value cache occupancy track how long Ridgeway's reps actually wait. Loading 16 GB of weights onto a fresh replica can take from tens of seconds to minutes, so one warm spare ahead of the Monday ramp beats scaling up from zero.

Sizing 20 concurrent sequences with one line of arithmetic

Little's law from queueing theory says the average number of requests in flight equals arrival rate multiplied by time spent in the system, which gives Ridgeway a GPU memory target in two steps. If the busiest hour carries 12 percent of daily traffic, that is 4,800 questions an hour, about 1.3 per second, and at roughly 5 seconds per full answer around 7 sequences run at once. Planning for bursts three times that size sets a target of about 20 concurrent sequences.

Each sequence holds its key-value cache in GPU memory, and the size follows from the model's shape. For a model with 32 layers, 8 key-value heads and a head size of 128 at 16-bit precision, every token needs about 128 KB, so a 3,350-token conversation holds roughly 420 MB. Twenty of them need around 8 GB on top of about 16 GB of weights for an 8-billion-parameter model, which fits on one 80 GB accelerator with room for growth.

Why batching and memory layout decide throughput on one 80 GB GPU

How the serving engine packs Ridgeway's 20 sequences together matters more than raw GPU speed. Static batching waits for every sequence in a batch to finish, so one long dispute explanation can stall a dozen short status answers, while continuous batching admits a waiting request at the next decode step. In Anyscale's 2023 benchmark of OPT-13B on an A100 GPU, vLLM using continuous batching reached up to 23 times the throughput of naive serving.

Memory layout is the other half of the story. The vLLM team's SOSP 2023 paper measured earlier serving systems losing 60 to 80 percent of key-value cache memory to fragmentation, and its PagedAttention design, which stores the cache in small blocks that need not sit side by side, cut that waste below 4 percent and lifted throughput 2 to 4 times over FasterTransformer and Orca. Recovered memory becomes extra concurrent sequences, which for a workload like Ridgeway's can postpone buying a second GPU.

Keeping an LLM in production healthy after the first 30 days

Keeping a live model healthy is a continuing release process in which every prompt edit, model upgrade or retrieval change passes a 300-question evaluation gate, a gradual rollout, and monitoring for slipping quality as well as errors. Imagine a carrier switching its fuel surcharge layout, after which Ridgeway's surcharge answers quietly get worse while error rates sit at zero. Infrastructure dashboards would look healthy, and only a graded sample of real answers would reveal the drop.

Slow quality loss like that surcharge drift is a monitoring problem rather than a serving problem. Dedicated work on AI model monitoring and drift detection sits at that layer, checking whether incoming invoices and outgoing answers still resemble what the model was validated on. Watching quality, input patterns and cost per route side by side tends to expose shifts like this within days rather than quarters.

A release gate for every Ridgeway prompt or model change

1. Run the change against a fixed evaluation set of 300 anonymized invoice questions with graded reference answers.

2. Block the release if accuracy on invoice dollar amounts drops, even when the overall score rises.

3. Mirror a slice of live invoice traffic to the new version in shadow mode and compare answers without showing them to reps.

4. Move 5 percent of reps to the new version, then 25 percent, then everyone, checking p95 latency and cost per 1,000 requests each time.

5. Keep the previous prompt and model deployable so one command rolls Ridgeway back.

Wiring those five gates into CI pipelines, version registries and deployment tooling is the heart of LLMOps work. Teams without that tooling often look at MLOps and LLMOps services to get the pipelines built before traffic grows. The aim is a prompt change that feels as routine and reversible as a code change.

Three limits a well-built serving stack will not fix

Good serving infrastructure makes a model faster, cheaper and more dependable, but it cannot make Ridgeway's answers more accurate, hold token prices steady, or keep this year's hardware commitment looking wise. Wrong surcharge figures come from the model and the prompt, so faster decoding only delivers a wrong answer sooner. The 300-question evaluation set and the dollar-amount check stay the main defense.

Agent-style workflows, where the model calls tools and loops several times per task, make per-request cost hard to forecast. One dispute might need two model calls and the next one twelve, so hard caps on steps and tokens per task are the practical guardrail. Pinning a dated model version, where a provider offers one, also stops a silent update from changing answers overnight.

Hardware commitments age quickly while token prices keep falling. Epoch AI's 2025 analysis of API pricing found the cost of reaching a fixed benchmark score dropping between 9 and 900 times per year depending on the task, with a median near 50 times. A three-year GPU reservation signed to beat today's API price can look expensive by its second year, so shorter commitments suit teams still learning their traffic.

A 30-day plan from working prototype to first production traffic

Many teams with a working prototype can reach 5 percent production traffic in about 30 days, and the order of the four weeks matters more than the exact dates. Skipping the measurement week is the costliest shortcut, because teams often learn in month two that real prompts run far longer than the design document assumed. Ridgeway's version of the month looks like this.

1. Days 1 to 7: log realistic invoice prompts, measure token counts per route, and assemble the 300-question evaluation set before touching infrastructure.

2. Days 8 to 14: fill in the cost table with current provider rates, pick a hosting model per route, and write the 700 ms latency budget.

3. Days 15 to 21: build the gateway, router, caching layer and release gate, then load-test at three times the expected peak of 20 concurrent sequences.

4. Days 22 to 30: send 5 percent of reps down the new path, compare cost per 1,000 requests and p95 latency with the budget, and widen exposure only while both hold.

Getting your first route priced against a $12,000 ceiling

The quickest way to lower launch risk is to fill in the cost table and the 700 ms latency budget with your own measured traffic before choosing hardware. Ridgeway's numbers show how far caching, batch pricing and shorter replies can move a bill, but your token counts will tell their own story. If data rules or steady volume point toward running models inside your own environment, our Private LLM deployment services page explains that path.