Field notes on inference, evals and small language models.

Your eval set is drifting, and that's fine

Every eval set I've maintained for longer than a quarter slowly stopped describing production traffic. Freezing it forever gives you comparable numbers that no longer mean much; rewriting it every month gives you meaningful numbers you can't compare.

What has worked for me is a two-tier setup: a small frozen anchor set that never changes, and a rolling set rebuilt from a fresh traffic sample each month. Report both. When they disagree, that disagreement is the interesting part.

Speculative decoding on a single consumer GPU

Pairing a 1B draft model with an 8B target gave a 1.7–2.1× speedup on chat-style prompts, and almost nothing on code completion. Acceptance rate is everything: when the draft model guesses badly, you pay for both models and get the speed of one.

Measure acceptance rate per task type before you ship it, not after.

What 4-bit quantization actually costs you

Average benchmark scores barely moved. The damage was concentrated in a few places: long arithmetic, rare languages, and strict output formats. If your feature depends on any of those, average scores will tell you everything is fine right up until it isn't.

This is why I wrote quantdiff: look at where outputs diverge, not at the mean.

Watching the KV cache fill up

Most out-of-memory incidents I've seen during inference were not caused by the weights but by a handful of very long requests sharing a batch with everything else. Plotting cache usage per request over time made the fix obvious: cap context per request and route long ones to a separate queue.

p50 lies: latency budgets for LLM features

Users experience your p95, not your median. For streaming features, time to first token matters more than total generation time; for tool-calling agents, the sum of the slowest steps dominates. Set budgets per stage and alert on percentiles, not averages.