Optimize LLM Latency: Fix Scheduling and KV Before Model Tuning

The fastest moves to try are state-aware scheduling with admission control and prefill or KV cache cleanup, both before touching the model itself. These two levers can help close the gap between a demo and a production-grade response time. Speculative decoding and quantization help too, but they come with trade-offs worth understanding before you commit to Time To First Token (TTFT) or Time Per Output Token (TPOT) targets.

Key Takeaways

Optimizing large language model inference relies on request scheduling, memory management, and model techniques, with scheduling and KV cache strategies delivering the most significant latency reductions.

Table of Contents

• Key techniques at a glance
• Scheduling, SLOs, and admission control: TTFT vs TPOT
• Prefill and KV cache strategies: LayerKV, disaggregation, and chunking
• Decoding optimizations: speculative decoding and draft models
• Model and hardware optimizations: quantization, parallelism, and offloading
• Batching, chunking, and the throughput vs latency trade-off
• Measuring, benchmarking, and SLO-driven testing
• Decision framework: which levers to use when
• Author's practical notes and one-week sprint checklist
• How I help: fixed-price LLM performance audits and builds
• FAQ
• Sources

Key techniques at a glance

Before going deep, here's the full toolkit in one place so you can jump to what matters for your situation.

• State-aware scheduling and admission control: protects TTFT/TPOT service-level objectives (SLOs) by rejecting or reordering requests before they violate deadlines, at the cost of a lower admission rate under load.
• Layer-wise KV offload (LayerKV-style): cuts queuing caused by KV cache contention, but adds complexity in managing which layers stay resident on GPU.
• Disaggregated prefilling: isolates TTFT tuning from inter-token latency by splitting prefill and decode onto separate instances, though it doesn't raise throughput on its own.
• Chunked prefill: smooths out compute-bound prefill against memory-bound decode, but needs careful token-budget tuning to avoid starving either phase.
• Speculative decoding (EAGLE and draft models): speeds up token generation at low concurrency, but consumes extra GPU memory and compute that can backfire under heavy load.
• Quantization and mixed precision: reduces latency and memory footprint, with a real risk of accuracy drift that needs validation.
• Parallelism choices (tensor, pipeline, data): reduces per-device memory pressure and can shorten compute time, but adds coordination overhead across devices.
• Batching and max_num_batched_tokens tuning: the main throughput versus latency dial, where smaller budgets favor interactivity and larger ones favor volume.
• Measurement and SLO instrumentation: the step that tells you whether any of the above actually worked, rather than just feeling faster.

Each of these gets its own section below, in the order I'd actually tackle them on a real system.

Scheduling, SLOs, and admission control: TTFT vs TPOT

TTFT is the delay between sending a prompt and seeing the first token. TPOT is the average time between each subsequent token. For a chat interface, TTFT shapes how responsive the app feels at the start, and TPOT shapes how natural the streaming output reads. Both deserve their own targets rather than a single blended latency number, because the causes of a TTFT spike and a TPOT spike are usually different.

Scheduling is where most of the tail latency hides. A request queue that treats everything as first-in-first-out will let long prefill jobs block short, latency-sensitive ones. State-aware scheduling fixes this by monitoring request and system state together and reordering work with hierarchical priority rules, so a request close to breaching its deadline jumps ahead of one with slack left, based on SOLA's state-aware scheduling approach.

Formal schedulability analysis takes this further. A Chronos-style admission control model treats each request as a prefill job with a TTFT deadline and a decode task with a TBT bound, and a sound admission algorithm can provably guarantee no TTFT or TBT misses for every accepted request, according to formal schedulability research for LLM inference. The cost is a lower admission rate: you reject some arrivals rather than let everyone degrade together.

To put this into practice:

1. Instrument queueing time separately from compute time so you know where delay actually accumulates.
2. Score incoming requests against a deadline window rather than a static priority tier.
3. Set an admission threshold tied to current queue depth and reject or defer requests that would blow the SLO for everyone else.
4. Log every rejection and missed deadline so you can tune the threshold instead of guessing.

Pro Tip: Use conservative, empirically observed P99 percentiles rather than theoretical worst-case numbers when setting liveness thresholds; real traffic always has fatter tails than synthetic benchmarks suggest.

Prefill and KV cache strategies: LayerKV, disaggregation, and chunking

Every LLM request runs through two phases: prefill, which processes the full prompt and builds the key-value (KV) cache, and decode, which generates tokens one at a time using that cache. As context length grows, KV cache allocation grows with it, and a handful of long-context requests can stall the queue for everyone waiting behind them. This is usually the real cause of a TTFT spike that looks random until you check context lengths.

LayerKV addresses this with layer-wise KV block allocation and offloading, moving less-active layers off the GPU instead of treating the whole cache as one block.

LayerKV reports up to a 69x mean TTFT improvement across 7B to 70B parameter models evaluated, alongside lower SLO violation rates, according to LayerKV's layer-wise KV cache management research. That's a dramatic number from a specific benchmark setup, not a guarantee for every deployment, but it shows how much headroom sits in KV management alone.

Disaggregated prefilling takes a different route: it runs prefill and decode on separate instances entirely, so you can tune TTFT and inter-token latency (ITL) independently rather than fighting over the same GPU cycles. vLLM's disaggregated prefilling documentation describes this as experimental, with KV caches transferred between the prefill and decode workers.

Practical tuning notes:

• Keep the most recently accessed layers resident on GPU and offload colder layers first, rather than an even split.
• Set max_num_batched_tokens smaller when you care about inter-token latency, and larger when TTFT for bursty traffic matters more.
• Watch for preemption or recompute warnings in your logs. Those are early signs that KV pressure, not raw compute, is your bottleneck.

If long contexts are the root cause in your system, my guide on LLM context management walks through auditing where that pressure comes from before you reach for infrastructure changes.

Decoding optimizations: speculative decoding and draft models

Speculative decoding speeds up the decode phase by having a smaller draft model guess several tokens ahead, then having the main model verify them in a single pass instead of generating them one at a time. EAGLE-style and draft-model approaches both work this way, and when the guesses land, you get multiple tokens for close to the cost of one.

The trade-offs are real:

• Running a draft model alongside the main model adds GPU memory overhead that competes with your KV cache budget.
• The speedup is concurrency-dependent. It shows up clearly at low to moderate load and shrinks as GPU utilization climbs.
• Acceptance rate (how often the draft model's guesses are verified correct) determines whether the technique pays for itself at all.

EAGLE-3 speculative decoding typically delivers 2x or more token throughput improvement at low concurrency, according to NVIDIA's speculative decoding guide for TensorRT-LLM, but that same source stresses benchmarking at concurrency=1 specifically to isolate the latency gain from throughput effects. NVIDIA NIM's documentation on speculative decoding adds that the technique can degrade throughput once the GPU is already saturated, since the extra compute for draft generation has nowhere to go.

Before shipping speculative decoding, measure latency at concurrency=1, capture the acceptance rate separately, and then retest under your actual production traffic pattern. A technique that looks great in isolation can regress badly once real concurrent load hits it.

Pro Tip: Build a fallback that disables speculative decoding automatically once GPU utilization crosses a set threshold, so you get the upside at quiet hours without the downside during traffic spikes.

Model and hardware optimizations: quantization, parallelism, and offloading

Once scheduling and KV handling are solid, the next layer is the model and hardware setup underneath it.

Quantization and mixed precision cut memory footprint and often latency, since smaller numeric formats move faster through memory and compute. The trade-off is accuracy drift, which can be subtle enough to pass casual testing and still show up in edge cases, so validate output quality on a representative sample before rolling quantization out broadly.

Parallelism choices affect both memory pressure and latency differently:

• Tensor parallelism splits individual layers across devices, reducing per-device memory and often cutting latency for large models, at the cost of inter-device communication overhead.
• Pipeline parallelism splits layers sequentially across devices, which helps with very large models but can add latency if stages aren't balanced well.
• Data parallelism replicates the whole model across devices to handle more concurrent requests, which helps throughput more than it helps any single request's latency.

GPU memory utilization settings determine how much headroom you leave for KV cache versus model weights. Offloading colder data to CPU or NVMe frees GPU memory for more concurrent KV cache space, but adds latency for whatever gets paged back in. Watch for swap or preemption events in your metrics: frequent preemption usually means memory utilization is set too aggressively for your actual traffic shape, not that you need more GPUs yet.

If you're choosing host hardware for a latency-sensitive deployment, memory bandwidth matters more than most people expect. This comparison of DDR5 and DDR4 server memory is a useful reference for understanding when extra memory channels actually change throughput for memory-bound workloads like decode.

Batching, chunking, and the throughput vs latency trade-off

Batch size and max_num_batched_tokens are the main dial between a system tuned for interactive use and one tuned for bulk throughput. A larger token budget lets the scheduler pack more work into each iteration, which raises throughput and TTFT for bursty traffic, but it slows down inter-token latency for whoever is already streaming a response, since their next token has to wait behind a bigger batch of other work, per vLLM's optimization and tuning guidance.

Chunked prefill splits prefill into smaller pieces and interleaves them with decode steps, rather than letting one long prefill job monopolize the GPU and stall every decode in progress. That same vLLM guidance describes chunked prefill as a way to balance compute-bound prefill against memory-bound decode operations, which is exactly the tension that drives most TPOT spikes.

A practical sequence for tuning this safely:

1. Start with a small max_num_batched_tokens value for interactive, chat-style workloads where TPOT matters most.
2. Raise the value incrementally for batch or offline workloads where total throughput matters more than any single response's smoothness.
3. Pair any increase with admission control so a throughput-tuned system doesn't silently blow its TTFT SLO under load.
4. A/B test changes against replayed production traffic, not synthetic single-request benchmarks, and watch P99 TTFT and TPOT together, not just the average.

Measuring, benchmarking, and SLO-driven testing

None of the techniques above mean anything without measurement that proves they worked. Track these as a baseline:

• TTFT across multiple percentiles, since the tail is where user complaints come from.
• TPOT or time-between-tokens (TBT), tracked across multiple percentiles.
• Queue length and admission or rejection rate, so you can see scheduling pressure directly.
• GPU utilization, to catch when you're compute-bound versus memory-bound.
• Tail compute time, separate from queueing time, to isolate where delay actually originates.

Benchmark with replayed production traces rather than synthetic prompts whenever possible, under realistic concurrency and input-length distributions. Every change, whether it's a scheduling tweak or a quantization swap, deserves a regression test against the same trace before it ships.

Formal admission control models can provably guarantee zero TTFT and TBT misses for accepted requests, according to schedulability analysis for LLM inference, which is a meaningfully stronger claim than the "usually fast enough" standard most teams run with today. Build your pre-deploy tests to check for that kind of guarantee, not just an average latency number that hides the tail.

Decision framework: which levers to use when

Match the lever to your actual constraint rather than reaching for the most interesting technique first.

• Low concurrency, latency-sensitive: speculative decoding and quantization give the clearest wins with the least architectural change.
• High contention, many simultaneous users: scheduling and admission control first, then layer-wise KV offload or disaggregated prefilling.
• Long contexts: LayerKV-style offload and chunked prefill before anything else, since context length is usually the root cause of queuing.

A sprint checklist that holds up across most systems: measure first, protect SLOs with scheduling, cut queueing through KV and prefill fixes, apply speculative decoding selectively where concurrency allows it, then tune model and hardware settings, and only scale hardware once the above is exhausted.

Pro Tip: When you introduce admission control, frame rejected requests to product owners as a reliability feature rather than a failure: a fast, consistent experience for accepted users beats a slow, unpredictable one for everybody.

Author's practical notes and one-week sprint checklist

On a tight timeline, I start with logs and metrics before touching code: queueing time, TTFT and TPOT percentiles, and context-length distribution tell me where the real bottleneck sits. From there, small scheduling changes and KV or prefill tweaks usually move the needle faster than a model swap. An audit typically produces a prioritized fix list, a before-and-after benchmark, and a handoff document so the client's own engineers can keep tuning after I'm gone.

— Hanad Kubat

How I help: fixed-price LLM performance audits and builds

If you'd rather have someone diagnose and fix this directly, I run a fixed-price Prototype Audit that identifies your highest-impact latency fixes, credited against a build if you move forward. From there, I build production AI features, including scheduling, KV, and inference tuning work, at a fixed scope and price, with no agency overhead or offshore markup, and you own the code from the first commit. You can see current project types and pricing on my services page.

FAQ

What is the difference between TTFT and TPOT?

TTFT (Time To First Token) measures the delay before the first token appears, while TPOT (Time Per Output Token) measures the average delay between each subsequent token during generation. They respond to different fixes: TTFT improves with better scheduling and prefill handling, while TPOT improves with decode-phase tuning like batch size and speculative decoding.

Does speculative decoding always improve latency?

No. Speculative decoding typically delivers strong gains at low concurrency, with EAGLE-3 showing 2x or more throughput improvement in that setting, but it adds memory and compute overhead that can degrade throughput once the GPU is saturated. Always benchmark under your real traffic pattern before relying on it.

Should I prioritize scheduling fixes or model-level optimizations first?

Scheduling and admission control usually come first, since they address the system-level bottleneck that causes the worst tail latency, before quantization, parallelism, or other model-level changes. Formal schedulability approaches can guarantee no TTFT or TBT misses for accepted requests, which gives you a measurable foundation before layering on other optimizations.

What does disaggregated prefilling actually change?

Disaggregated prefilling runs prefill and decode on separate instances so each can be tuned independently, which isolates TTFT optimization from inter-token latency tuning. vLLM's documentation notes it's experimental and doesn't improve throughput on its own, so it's best used when tail latency, not raw volume, is the problem.

How much can KV cache management improve latency?

Layer-wise KV cache allocation and offloading can produce large gains in specific benchmark conditions. LayerKV reports up to a 69x mean TTFT improvement across tested model sizes, alongside lower SLO violation rates, though actual results depend heavily on your context lengths and traffic pattern.

Sources

• Frontiers | Formal schedulability analysis for LLM inference: TTFT and TBT deadline guarantees via response-time theory
• LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
• Disaggregated Prefilling (experimental) - vLLM
• Speculative Decoding with TensorRT-LLM — NVIDIA Triton Inference Server

Recommended

• Cut LLM Latency in One Week: Prompt Caching for Engineers
• Cut LLM Costs 50–80%: Engineer Led Two Week TCO Playbook
• Audit First LLM Context Management for Solo Engineers
• Hours to Days: Why Engineers Should Start With RAG Before Fine Tuning