← Back to Blog
For: AI Engineers, ML Engineers, Platform Engineers, AI Systems Architects

Benchmark Small Language Models: The Silent Second Variable

Many model comparisons fail because two things changed, not because a number was wrong. A harness that refuses the comparison catches it in one stack trace.

ai-engineeringllm-deploymentsmall-language-model
#small-language-models#llm-benchmarking#llm-evaluation#inference-latency#quantization#reasoning-models#mlops

This is a guide to benchmarking small language models so that the comparison survives: a quantized checkpoint against its original, one runtime against another, thinking mode on against off. The method is a benchmark harness that records every condition of a run and refuses to compare two runs when more than one condition changed.

Why benchmark comparisons fail: the same LLaMA-65B weights scored 0.637 and 0.488

In June 2023, Hugging Face had to explain why LLaMA-65B ranked lower on its Open LLM Leaderboard than its own paper said it should. Weights and benchmark were identical in every case: Massive Multitask Language Understanding (MMLU), five-shot. Berkeley's original implementation scored the model at 0.636, and Stanford's Holistic Evaluation of Language Models (HELM) scored it at 0.637. EleutherAI's evaluation harness, which the leaderboard used, scored it at 0.488.

Nobody had made an arithmetic mistake. All three implementations formatted the prompt differently (HELM and the harness add a Question: prefix, and the harness adds a Choices keyword), and they also scored answers differently: the original read the probability of the answer letter, while the harness read the probability of the full answer text. Each number was a faithful measurement of a different experiment. In the plain words of the Hugging Face team, evaluations are "strongly tied to their implementations - down to minute details such as prompts and tokenization."

d2
direction: down
w: "LLaMA-65B weights\n(identical in every run)" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
impls: "Same benchmark name: MMLU, 5-shot" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 30
  o: "Original\nletter probability\n0.636" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  h: "HELM\n'Question:' prefix\n0.637" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  e: "Eleuther harness\nfull-answer probability\n0.488" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
w -> impls: "unstated: prompt format + scoring rule"

As the diagram shows, the gap was invisible from the leaderboard: the label on every row said "MMLU", and the variable that moved the score by 15 points was not on the label at all. Readers set the paper's 0.637 against other models' leaderboard rows, so two things changed in that comparison: the model, and the code that evaluated it. Only the first was on anyone's mind.

That was a 65-billion-parameter model on a public leaderboard, with thousands of people looking at it. On small-model teams the same failure happens at a smaller scale, with nobody looking at all. Someone compares a 16-bit checkpoint against its quantized version on a laptop, with a prompt file that a colleague edited on Wednesday, through a chat template whose thinking-mode default nobody checked. A clear winner appears in the spreadsheet. Nobody can say what it measured.

A benchmark comparison is void when a second condition moved

I call the cause the Silent Second Variable: a controlled condition that changed between two benchmark runs without anyone choosing to change it, sitting next to the one variable the comparison was meant to be about.

Statisticians call any such second cause a confounder. The silent second variable is the narrow case that keeps breaking benchmarks: the confounder is a run condition, nobody chose to change it, and so nobody wrote it down.

My claim is that a large share of the small-model comparisons teams make are void for this reason. Not noisy. Void. Each individual number is usually correct, but the comparison between them has no meaning, because a difference produced by two causes cannot be attributed to either one. Standard advice ("benchmark your candidates and compare") does nothing about it, since more benchmarks only produce more numbers with the same defect. And the damage is worse than noise, because the second variable can be larger than the first and take the credit for it. In the runs below, a naive comparison said bf16 weights were four times faster than 32-bit weights. Per token, bf16 was no faster at all on that machine. The four times came from a silent change to thinking mode, which made the average reply five times shorter.

More care will not fix this, because care fails under deadline. What fixes it is a harness that knows which conditions define a measurement, stores them with every result, and raises an exception when two results differ in more than one of them. Not a warning. A warning is read once and filtered forever, while the number escapes into a slide.

d2
direction: down
run_a: "Run A" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
run_b: "Run B" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
changes: "What changed between A and B" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 30
  intended: "Intended\nfp32 -> bf16 weights" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  silent: "Silent\nthinking on -> off\nprompt file 8 -> 10" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
delta: "-5 points accuracy, 4x lower latency\nCause: cannot be attributed" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
run_a -> changes
run_b -> changes
changes -> delta

Both arrows in the diagram feed one delta, and once that happens no arithmetic can split the delta back into its two causes. Your only remedy is to measure again with one of them held fixed.

Experimental science has known this for a long time and calls it controlling your variables. What LLM tooling lacks is enforcement. NVIDIA's NeMo Evaluator documentation says that runs must match on benchmark, dataset slice, prompt template, repeat count and max problems, because "differences in any of these confound the comparison", and then the tool leaves the matching to you. In the rest of this article I build the version that checks.

Where silent second variables come from in small-model work

Small-model work produces more of these than large-model work, because the whole point is to vary the cheap knobs: quantization, runtime, context length, reasoning mode. Each knob you vary on purpose sits among knobs that vary by accident, and these are the ones I see most:

  1. A task set grew under the same name. Someone added the cases that failed in staging, but the file is still called support_faq.jsonl, and every later model now looks worse.
  2. Cold against warm. One run timed the first request, which pays for allocator growth, kernel loading and graph capture, while the other ran after the process had been up for an hour.
  3. Thinking mode flipped. Several current small models carry a reasoning mode and a direct mode in one set of weights, and the default differs by family, by version, and by how you call the chat template.
  4. Runtime or file format differs. A Hugging Face checkpoint measured through transformers is not the GGUF file you will serve through llama.cpp.
  5. A unit changed meaning. Tokens per second across two model families is a ratio of two different units.

Each one appears below on real runs, along with the harness that refuses them.

The wrong way: a benchmark script that cannot notice

Many first benchmarks look like the script below. I ran it on Qwen3-0.6B on a CPU-only laptop (transformers 5.3.0, torch 2.13), and it is correct code: every number it prints is a real measurement. The Monday and Thursday that follow are a staged reproduction of a pattern I have seen on teams. I wrote both calls, and every output is a real run.

python
import jsonimport timeimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizerdef bench(model_id: str, dtype: str, prompt_file: str, **template_kwargs) -> dict:    tok = AutoTokenizer.from_pretrained(model_id)    model = AutoModelForCausalLM.from_pretrained(model_id, dtype=getattr(torch, dtype))    cases = [json.loads(line) for line in open(prompt_file, encoding="utf-8")]    latencies, correct = [], 0    for case in cases:        start = time.perf_counter()        ids = tok.apply_chat_template(            [{"role": "user", "content": case["prompt"]}],            add_generation_prompt=True, return_tensors="pt", return_dict=True,            **template_kwargs,        )        out = model.generate(**ids, max_new_tokens=256, do_sample=False)        text = tok.decode(out[0, ids["input_ids"].shape[-1]:], skip_special_tokens=True)        latencies.append(time.perf_counter() - start)        correct += case["expected"].lower() in text.lower()    return {"accuracy": correct / len(cases),            "avg_latency_s": round(sum(latencies) / len(latencies), 2)}

On Monday it measured 32-bit weights, and on Thursday someone measured 16-bit bfloat16 (bf16) weights to see what the smaller format costs:

python
print(bench("Qwen/Qwen3-0.6B", "float32", "faq_monday.jsonl"))print(bench("Qwen/Qwen3-0.6B", "bfloat16", "faq_thursday.jsonl", enable_thinking=False))
text
{'accuracy': 0.75, 'avg_latency_s': 40.03}{'accuracy': 0.7, 'avg_latency_s': 9.78}

Anyone reading this output would conclude that bf16 is about four times faster and costs five points of accuracy. That sentence would go into a design doc, and it is wrong in both halves.

Three things changed between those calls, and only one of them was chosen. Moving the weights from 32-bit to bf16 was the point. Thursday's call also passes enable_thinking=False, because someone copied it from a snippet that "made Qwen stop rambling", and since Qwen3's chat template thinks by default, Monday measured a model that wrote a reasoning trace before every answer while Thursday measured one that did not. Finally, faq_thursday.jsonl has two more prompts than Monday's file: the two cases that had failed in staging.

When I later varied only the weight format, with everything else held fixed, there was no measurable accuracy difference between bf16 and 32-bit on the same eight prompts. And bf16 did not decode faster on this CPU. Both bf16 runs came in slightly slower, at 4.4 and 4.5 tokens per second, against 4.8 to 5.9 across five 32-bit runs. By elimination, then, the four-times speedup came from turning thinking off, and the harness shows the mechanism directly: with thinking on, replies averaged 208 tokens, and with it off, 40. The naive comparison did not show what bf16 costs. It gave bf16 the credit for a speedup that belonged to a different change.

A fourth variable is one the script cannot see at all. I ran the identical Monday call twice: the first time, with the laptop short of memory, it averaged 100.05 seconds per prompt, and the second time it averaged 40.03. Same code, same weights, same file. Machine load is in no record, mine included, which is why the noise-floor check later in this article matters.

d2
direction: down
script: "bench()" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
kept: "What the result keeps" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 24
  acc: "accuracy" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  lat: "avg_latency_s" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
lost: "What the result throws away" {
  label.near: top-center
  grid-rows: 2
  grid-columns: 2
  grid-gap: 18
  f: "which prompts\n(file contents)" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  t: "thinking mode\n(template kwargs)" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  w: "first-call cost\n(no warmup)" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  s: "prefill vs decode\n(one blended time)" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
script -> kept
script -> lost

What the script returns is two numbers. Four facts that decide what those numbers mean exist only in the code and the file system on the day it ran, and as the diagram shows, none of them reaches the result.

How to benchmark small language models: a run record and a comparison that refuses

Compared with the script, the harness below does five extra things: it lets the runner report its own timings, fingerprints both the task set and the prompts the model actually receives, reads the runtime and hardware from the machine instead of trusting a typed label, binds the measurement to every condition that produced it, and refuses to compare two records that differ in more than one condition. I adapted it from Chapter 2 of my book, Right-Sized AI, with prompt fingerprints, thinking mode and warmup added.

A runner protocol and a fingerprinted task set

python
import hashlibimport statisticsfrom dataclasses import dataclassfrom typing import Protocol@dataclass(frozen=True)class Generation:    """One completion, timed by the runner: only it can see the first token."""    text: str    completion_tokens: int  # generated token ids, counted directly, EOS included    ttft_s: float    total_s: float    peak_memory_gib: floatclass Runner(Protocol):    name: str    def render(self, prompt: str) -> str: ...  # the exact text the model will see    def generate(self, prompt: str, max_tokens: int) -> Generation: ...@dataclass(frozen=True)class Task:    prompt: str    expected: str | None = Nonedef _sha(parts) -> str:    digest = hashlib.sha256()    for part in parts:        digest.update(part.encode("utf-8") + b"\x00")    return digest.hexdigest()[:12]@dataclass(frozen=True)class TaskSet:    name: str    tasks: tuple[Task, ...]    def fingerprint(self) -> str:        # A set that gains one example is a different set.        # A name does not notice. A hash does.        return _sha(p for t in self.tasks for p in (t.prompt, t.expected or ""))def prompt_fingerprint(runner: Runner, task_set: TaskSet) -> str:    # Hashes what the model actually receives: chat template, system prompt,    # thinking flags. A /think buried in a system prompt changes this hash.    return _sha(runner.render(t.prompt) for t in task_set.tasks)

Of everything in the harness, the fingerprint is the cheapest protection. While the name support-faq stayed the same, the Monday file hashes to 487828267c4b and the Thursday file to c6dd143e36f1. It hashes the expected answers too, and it is sensitive to order. Reordering prompts changes nothing statistically, but it means someone edited the file, and I want to be told.

The task-set fingerprint hashes what you wrote. prompt_fingerprint hashes what the model received, after the chat template, the system prompt and the thinking flag have all done their work. That second hash is the one that catches a /think sitting in somebody's system prompt, which the task set never sees. It needs one method on the runner, render, which returns the exact text a prompt becomes.

d2
direction: down
mon: "faq_monday.jsonl\n8 tasks" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
thu: "faq_thursday.jsonl\n10 tasks (2 added)" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
ids: "Identity of each set" {
  label.near: top-center
  grid-rows: 2
  grid-columns: 2
  grid-gap: 16
  n1: "name: support-faq" {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
  h1: "hash: 487828267c4b" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  n2: "name: support-faq" {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
  h2: "hash: c6dd143e36f1" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
mon -> ids
thu -> ids

In the diagram, the grey column is what most benchmark spreadsheets record, and the right column is what the harness compares.

TTFT vs decode rate: split the latency

Our first script reports one blended latency, and that number hides the most useful split in inference. Time to first token (TTFT) is how long the user waits for anything to appear. Prefill dominates it: the whole prompt is processed in parallel, as matrix-matrix work that saturates compute. Decode rate is the pace of every token after the first, one per forward pass, as matrix-vector work bound by memory bandwidth, and NVIDIA's inference optimization guide describes the same two phases (I cover them in more depth in how LLM inference and serving work). They respond to opposite fixes: shorter prompts or different batching fix a slow prefill, while smaller weights or more bandwidth fix a slow decode. Average them and you learn that something is slow, and nothing about what.

d2
direction: down
req: "One request, in time order" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 0
  pre: "Prefill\nwhole prompt in parallel\ncompute-heavy, sets peak memory" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  first: "first\ntoken" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  dec: "Decode\none token per step\nbound by memory bandwidth" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
m: "What the harness reports" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 30
  ttft: "TTFT = prefill + first token" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  rate: "decode rate = (tokens - 1) / decode time" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
req -> m

As the timeline shows, the decode rate subtracts one token because the first token arrives at the end of prefill, so counting it as a decode event inflates the rate, and inflates it most on short replies.

Serious tools already make this split: llama.cpp's llama-bench reports prompt processing (pp) and text generation (tg) as separate tests, and vLLM's vllm bench serve reports TTFT, time per output token and inter-token latency by default. Your own harness should not be less careful than the tools you would compare it with.

python
def final_answer(text: str) -> str:    # Score only the answer. A trace cut off before </think> has no answer.    if "<think>" in text and "</think>" not in text:        return ""    return text.split("</think>")[-1]@dataclass(frozen=True)class Measurement:    quality: float    ttft_p50_s: float    ttft_max_s: float  # with 8-10 prompts, "p95" would be this number anyway    decode_tok_s: float    mean_reply_tokens: float    capped_share: float  # replies that hit max_tokens    peak_memory_gib: float    samples: intdef measure(runner: Runner, task_set: TaskSet, *,            max_tokens: int, warmup_runs: int) -> Measurement:    # Load cost, allocator growth and first-call kernels land in the warmup runs,    # which are thrown away.    for _ in range(warmup_runs):        runner.generate(task_set.tasks[0].prompt, max_tokens)    gens = [runner.generate(t.prompt, max_tokens) for t in task_set.tasks]    scored = [        float(t.expected.lower() in final_answer(g.text).lower())        for t, g in zip(task_set.tasks, gens)        if t.expected is not None    ]    ttfts = [g.ttft_s for g in gens]    # Pooled, not averaged per request: a 4-token reply must not weigh as much    # as a 400-token one. The first token belongs to prefill, not decode.    decode_tokens = sum(max(g.completion_tokens - 1, 0) for g in gens)    decode_s = sum(max(g.total_s - g.ttft_s, 0.0) for g in gens)    return Measurement(        quality=statistics.fmean(scored) if scored else float("nan"),        ttft_p50_s=statistics.median(ttfts),        ttft_max_s=max(ttfts),        decode_tok_s=decode_tokens / decode_s if decode_s > 0 else 0.0,        mean_reply_tokens=statistics.fmean(g.completion_tokens for g in gens),        capped_share=statistics.fmean(g.completion_tokens >= max_tokens for g in gens),        peak_memory_gib=max(g.peak_memory_gib for g in gens),  # a maximum, never a mean        samples=len(gens),    )

Five choices in measure are deliberate. Peak memory is a maximum, because an average of high-water marks is a number no moment of the run ever reached. Quality is nan when nothing was scorable, because "nothing was scored" and "everything was wrong" must not print the same number. With eight to ten prompts, a "p95" is the maximum under a statistical name, so the field is honestly called ttft_max_s. Quality is scored on the text after the reasoning trace only: a reply cut off before </think> has no answer, and a check that reads the trace will credit an answer the model only considered. And reply length and the share of replies that hit max_tokens sit next to the timings, because, as the thinking section shows, they explain more than the timings do.

The warmup boundary: cold and warm are different measurements

A warmup boundary is the line you draw between costs paid once per process and the behavior you actually want to report. Above it sit weight loading, allocator growth, library loading and graph capture; below it sit the requests your users will send. On a GPU the first-call costs are documented and large: PyTorch's benchmarking recipe shows a first matrix multiply running far slower because cuBLAS has to load, and vLLM captures its CUDA graphs on the first forward pass. Without a boundary those costs are silently averaged into the result, and on a short benchmark they can dominate it.

On my CPU run I could not measure the effect at all, and I report that as a result. With weights loaded outside the timer, a run with no warmup (warmup_runs=0) decoded at 6.0 tokens per second with a median TTFT of 0.74 seconds. Five identical runs with two warmup calls ranged from 4.8 to 5.9 tokens per second and from 0.75 to 0.98 seconds. The cold run sat at the fast edge of the warm range, not outside it on the slow side. On this machine, the movement between identical runs is larger than any cold-start cost, and I cannot claim one. A record is what makes that judgment possible: the warmup setting is stored, the noise floor is measured, and the comparison says what it can and cannot support.

d2
direction: down
a: "Process start" {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
b: "Load weights\n(outside the timer: runner.load())" {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
c: "Warmup runs\nallocator growth, first-call kernels\ndiscarded" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
d: "Measured runs\nsteady state" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
a -> b -> c
c -> d: "warmup boundary"

Which side of the boundary you want depends on the deployment rather than on measurement taste. A server that stays loaded all day should be measured warm, while an edge device that powers up for a single question should be measured cold, because every request it serves is a first request. Either choice can be right. What goes wrong is making no choice, which is what a script with no warmup parameter does by default. Even vLLM's vllm bench serve defaults --num-warmups to 0, so a good tool measures cold unless you ask otherwise, and for that reason warmup_runs is a field in the record below and not a detail of the script.

The run record and the one-variable rule

python
# What a measurement is *of*. Two records may differ in exactly one of these.CONTROLLED = (    "model", "model_revision", "quantization", "runtime", "hardware",    "prompt_fingerprint", "max_tokens", "batch_size", "warmup_runs",)# When the rendered prompts differ, these say why. If none of them changed,# something unrecorded did: a template update, a system-prompt flag.EXPLAINS_PROMPT = ("thinking", "task_set_fingerprint")@dataclass(frozen=True)class RunRecord:    model: str    model_revision: str  # the Hub commit, not the repo name    quantization: str    runtime: str  # read from the installed packages, never typed by hand    hardware: str    prompt_fingerprint: str  # what was actually sent    thinking: bool  # what was requested    task_set_fingerprint: str    max_tokens: int    batch_size: int    warmup_runs: int    measurement: Measurement    # Informational: context for a human, never part of the comparison.    recorded_at: str = ""    notes: str = ""class IncomparableRecords(Exception):    pass@dataclass(frozen=True)class Comparison:    varied: tuple[str, ...]    quality: float    ttft_p50_s: float    decode_tok_s: float    mean_reply_tokens: floatdef varied_fields(before: RunRecord, after: RunRecord) -> tuple[str, ...]:    out: list[str] = []    for f in CONTROLLED:        if getattr(before, f) == getattr(after, f):            continue        if f == "prompt_fingerprint":            causes = [c for c in EXPLAINS_PROMPT                      if getattr(before, c) != getattr(after, c)]            out += causes or ["prompt_fingerprint"]        else:            out.append(f)    return tuple(out)def compare(before: RunRecord, after: RunRecord) -> Comparison:    varied = varied_fields(before, after)    if len(varied) > 1:        raise IncomparableRecords(            f"{len(varied)} controlled fields vary ({', '.join(varied)}); "            "hold all but one fixed and measure again"        )    a, b = before.measurement, after.measurement    return Comparison(        varied=varied,  # empty means a repeatability check: the harness's noise floor        quality=b.quality - a.quality,        ttft_p50_s=b.ttft_p50_s - a.ttft_p50_s,        decode_tok_s=b.decode_tok_s - a.decode_tok_s,        mean_reply_tokens=b.mean_reply_tokens - a.mean_reply_tokens,    )

Splitting controlled fields from informational ones is the whole design. Controlled fields say what the measurement is of, and informational fields help the human who reads the record in six months. With that split in place, the one-variable rule becomes a single if: a pair of records can disagree on one controlled field at most, and the comparison is labelled with that field.

Three controlled fields are read, never typed, because a typed label is the field most likely to be stale. runtime comes from the installed transformers and torch versions, model_revision from the Hub commit the weights were loaded from, and hardware from the machine:

python
def environment(runner: TransformersRunner) -> dict:    import os    import platform    import torch    import transformers    return {        "model_revision": runner._model.config._commit_hash or "unknown",        "runtime": f"transformers-{transformers.__version__}+torch-{torch.__version__}",        "hardware": f"{platform.processor()} x{os.cpu_count()}",    }

varied_fields treats the rendered prompt specially. When the prompts differ because thinking or the task set changed, the comparison names that cause, so one intended change is counted once. When the prompts differ and neither recorded cause changed, the comparison names prompt_fingerprint itself: something changed what the model saw, and no field says what.

d2
direction: down
recs: "Two run records" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 40
  r1: "before" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  r2: "after" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
gate: "How many CONTROLLED fields differ?" {shape: diamond; style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
out: "Outcome" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 24
  zero: "0: repeatability check\n(your noise floor)" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  one: "1: attributable delta\nlabelled with that field" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  many: "2 or more:\nIncomparableRecords" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
recs -> gate
gate -> out

As the diagram shows, the guard has three outcomes, and only one of them is an error. Zero differences is the most underused measurement in benchmarking: the same configuration run twice, which tells you how much your harness moves on its own.

I chose an exception over a warning on purpose, and it is the part of this design people most often object to. A warning is printed once, then scrolled past, then filtered by everyone who runs the script daily, and the number it warned about is copied into a document within the hour. An exception costs a few seconds of annoyance. In return, you never ship a conclusion that the data could not support.

The guard has a limit, and it is the same limit the problem has: it can only see what the record holds. A variable that is not a field is silent to the harness too. Reading the runtime, revision and hardware from the machine, and hashing the rendered prompts, moves several of the usual suspects into the record, but it does not close the set. Machine load, power plan, background processes and the order the runs happen in are all missing, and the 100-second Monday run was exactly that kind of variable. My hardware field is a CPU name and a core count, with no memory size or power state. Repeated runs are the backstop: whatever the record cannot name shows up as noise, and the noise floor measures how much of it there is.

Timing a real model: the streamer thread

measure trusts the runner to report TTFT, because only the runner can see the first token. A blocking model.generate() hands back control only when the reply is complete, so all it can time is the whole request. To see where prefill ends you have to watch tokens arrive, and that means streaming.

python
import mathimport timefrom dataclasses import dataclass, fieldfrom threading import Threadfrom typing import Anyfrom harness import Generation@dataclassclass TransformersRunner:    model_id: str    thinking: bool  # set on every call, never left to a chat-template default    system_prompt: str | None = None    device: str = "cpu"    dtype: str = "bfloat16"    name: str = ""    _model: Any = field(default=None, repr=False)    _tok: Any = field(default=None, repr=False)    def __post_init__(self) -> None:        mode = "think" if self.thinking else "direct"        self.name = self.name or f"transformers:{self.model_id}:{mode}"    def load(self) -> None:        # Separate from __init__ so the caller decides where load cost lands.        import torch        from transformers import AutoModelForCausalLM, AutoTokenizer        self._tok = AutoTokenizer.from_pretrained(self.model_id)        self._model = AutoModelForCausalLM.from_pretrained(            self.model_id, dtype=getattr(torch, self.dtype)        ).to(self.device)        self._model.eval()    def render(self, prompt: str) -> str:        if self._tok is None:            self.load()        messages = [{"role": "user", "content": prompt}]        if self.system_prompt:            messages.insert(0, {"role": "system", "content": self.system_prompt})        return self._tok.apply_chat_template(            messages, tokenize=False, add_generation_prompt=True,            enable_thinking=self.thinking,        )    def generate(self, prompt: str, max_tokens: int) -> Generation:        import torch        from transformers import TextIteratorStreamer        if self._model is None:            self.load()        if self.device.startswith("cuda"):            # Without the reset you read the session's peak, not this request's.            torch.cuda.reset_peak_memory_stats()        text = self.render(prompt)        inputs = self._tok(text, return_tensors="pt").to(self._model.device)        prompt_len = inputs["input_ids"].shape[-1]        streamer = TextIteratorStreamer(            self._tok, skip_prompt=True, skip_special_tokens=True        )        result: dict[str, Any] = {}        def work() -> None:            try:                result["ids"] = self._model.generate(                    **inputs, max_new_tokens=max_tokens, streamer=streamer,                    do_sample=False,                )            except BaseException as exc:  # surface it; never leave the reader blocked                result["error"] = exc                streamer.end()        worker = Thread(target=work)        started = time.perf_counter()        worker.start()        chunks: list[str] = []        ttft_s = math.nan        for chunk in streamer:  # the main thread holds the clock            if not chunks:                ttft_s = time.perf_counter() - started            chunks.append(chunk)        worker.join()        total_s = time.perf_counter() - started        if "error" in result:            raise result["error"]        return Generation(            text="".join(chunks),            # Count the ids the model produced. Re-tokenizing the decoded text            # drops EOS and need not round-trip.            completion_tokens=int(result["ids"].shape[-1] - prompt_len),            ttft_s=ttft_s,            total_s=total_s,            peak_memory_gib=(                torch.cuda.max_memory_allocated() / 2**30                if self.device.startswith("cuda")                else math.nan  # "not measured" is true; a process RSS figure is not            ),        )
d2
direction: down
main: "Main thread (holds the clock)" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  s: "started = perf_counter()" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  f: "first chunk -> ttft_s" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  j: "join -> total_s" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
worker: "Worker thread: model.generate()\nprefill, then decode\nnever touches the clock" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
streamer: "TextIteratorStreamer\n(queue of text chunks)" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
worker -> streamer: "push chunks"
streamer -> main: "iterate"

In the diagram, every timer sits on the main thread: the worker does all the model work, and the main thread stamps two moments, the first chunk out of the streamer and the join.

Five details in that runner each prevent a specific false number. Generation runs on a worker thread, which is the pattern in the transformers TextIteratorStreamer documentation, so the main thread can observe the first chunk. If the worker fails, it ends the streamer and the main thread re-raises the error, instead of waiting forever on a queue that will never fill. The token count comes from the ids the model generated, end-of-sequence token included, because re-tokenizing the decoded text drops that token and need not give back the same ids. On CUDA, torch.cuda.max_memory_allocated() returns the peak since the program started unless you call reset_peak_memory_stats() first, so without the reset the fifth request reports the second request's peak. Sampling is off, because sampling adds variance that the harness would report as a property of the model. And peak memory is nan off CUDA, because a process memory reading includes the interpreter and the tokenizer. When the harness has no way to measure a value, it should print nan and admit it. A plausible wrong number is worse than a missing one, because every comparison downstream inherits it and nobody can tell.

The comparison guard on real small-model benchmark runs

I re-ran the Monday and Thursday conditions through the harness, as run records instead of dicts: M is 32-bit weights, thinking on, the 8-prompt set, no warmup; G is bf16, thinking off, the 10-prompt set, no warmup. Then I asked for the comparison the design doc wanted:

text
compare(M, G)  IncomparableRecords: 3 controlled fields vary (quantization, thinking,  task_set_fingerprint); hold all but one fixed and measure again

That one line holds the whole Monday-to-Thursday story. Without the guard, discovering it took a second round of measurement; with the guard, the delta never exists to be quoted.

Holding everything else fixed gives comparisons the harness accepts, each one labelled with the single field that varied. A1 to A5 are the same 32-bit configuration run five times. The output is reformatted to fit: TTFT is the median, decode is in tokens per second, and reply is the mean reply length in tokens.

text
[A1] float32  think=0 warm=2  quality 0.88  ttft 0.98s  decode 4.78  reply 29[A2] float32  think=0 warm=2  quality 0.88  ttft 0.76s  decode 5.86  reply 29[A3] float32  think=0 warm=2  quality 0.88  ttft 0.75s  decode 5.86  reply 29[A4] float32  think=0 warm=2  quality 0.88  ttft 0.93s  decode 5.15  reply 29[A5] float32  think=0 warm=2  quality 0.88  ttft 0.90s  decode 5.25  reply 29[B1] bfloat16 think=0 warm=2  quality 0.88  ttft 1.07s  decode 4.38  reply 29[B2] bfloat16 think=0 warm=2  quality 0.88  ttft 1.05s  decode 4.46  reply 29[D]  float32  think=0 warm=0  quality 0.88  ttft 0.74s  decode 6.02  reply 29[H]  float32  think=0 warm=2  quality 0.88  ttft 0.79s  decode 5.89  reply 29[C]  float32  think=1 warm=2  quality 0.50  ttft 0.77s  decode 5.11  reply 208compare(A1, A2)  varied: (none) - repeatability check  quality +0.00   ttft -0.22 s   decode +1.08   reply +0compare(A1, B1)  varied: quantization  quality +0.00   ttft +0.09 s   decode -0.40   reply +0compare(B1, B2)  varied: (none) - repeatability check  quality +0.00   ttft -0.02 s   decode +0.08   reply +0compare(A1, C)   varied: thinking  quality -0.38   ttft -0.21 s   decode +0.33   reply +179compare(D, A1)   varied: warmup_runs  quality +0.00   ttft +0.24 s   decode -1.24   reply +0compare(A1, H)   varied: prompt_fingerprint  quality +0.00   ttft -0.19 s   decode +1.11   reply +0

Read the five identical runs first, because they decide how to read everything else. Quality and reply length did not move at all: decoding is greedy, so the text was the same every time, and only the clock moved. But the clock moved a lot. Decode ranged from 4.78 to 5.86 tokens per second, about 20%, and median TTFT from 0.75 to 0.98 seconds. I have not found the cause; on a laptop, thermal and power management are the usual suspects, and neither is in the record.

That range changes how the single comparisons read. compare(A1, B1) says bf16 cost 0.40 tokens per second, which is well inside the noise. Set B1 against A2 and the same drop looks like 25%. One pair of runs cannot settle a difference this size. The five-run picture is clearer: both bf16 runs were slower than all five 32-bit runs, on decode and on TTFT, by about one noise-floor width. I read that as likely, not established. The likely reason is the hardware. This laptop's Tiger Lake CPU has AVX-512 but no native bf16 arithmetic (no AVX512-BF16 and no Advanced Matrix Extensions (AMX)), so the bf16 matrix work probably runs through conversion to 32-bit. On a CPU with those instructions, or on a GPU, expect the opposite result, which is one more reason hardware is a controlled field.

The pair that a careless reader would quote as a noise floor is compare(B1, B2): 0.08 tokens per second. Two runs that happened to agree make the harness look ten times steadier than it is.

The H run is the guard working on the variable this article is named for. H is the A configuration with /think added to the system prompt and thinking=False left as it was, so no recorded field changed. The rendered prompts did change, and the comparison names prompt_fingerprint as the varied field because neither recorded cause explains it. On Qwen3-0.6B the hidden flag happened to change nothing: replies stayed at 29 tokens. On SmolLM3, whose card says the system prompt wins, the same edit would turn thinking back on. The guard does not need to know which model behaves which way. It reports that the prompts changed for a reason nobody wrote down.

Thinking mode changes the workload, so treat it as a controlled field

Comparing A1 against C gives the strangest result in this article, and the most useful. Turning thinking on barely moved the per-token rates: decode was 5.11 tokens per second, inside the 32-bit range, and TTFT stayed inside it too. What moved was reply length, from 29 tokens to 208. The naive script's 40.03 against 9.78 seconds is the same effect: the harness records M and G are the naive pair, and their replies averaged 208 and 40 tokens. At 5 to 6 tokens per second, that difference in length is the whole difference in latency. Most of it is a reasoning trace the user may never see.

Thinking mode therefore changes the workload, not the speed, and a comparison across modes is a comparison between two different jobs done by the same weights. That is why thinking is a controlled field in RunRecord, and why reply length sits next to the timings in Measurement.

The quality number needs the same care. With thinking on, quality fell from 0.88 to 0.50, but half the thinking replies hit the 256-token cap before the trace closed, and a reply with no answer scores zero. Most of that drop belongs to the pair of thinking and max_tokens, not to thinking alone. The naive script scored the same Monday condition at 0.75, because its substring check found expected answers inside unfinished traces and gave credit for them.

Every family exposes the switch differently, and the defaults are not stable across versions. These are from the model cards as of 29 October 2026:

ModelDefaultHow to turn thinking off
SmolLM3-3Bthinking on/no_think in the system prompt, or enable_thinking=False. The system-prompt flag overrides the argument.
Qwen3.5-4Bthinking onenable_thinking=False in the chat template. The Qwen3 /think and /nothink soft switches are not officially supported.
Llama-3.3-Nemotron-Super-49B-v1not stateddetailed thinking off in the system prompt. Recommended sampling: greedy when off; temperature 0.6 and top-p 0.95 when on.

Two rows in that table contain a second trap. On SmolLM3 the system prompt beats the argument, so a run can pass enable_thinking=False and still think, because someone's system prompt carries /think. On Nemotron the card recommends different sampling for each mode, so following the card changes two controlled conditions at once. And a Qwen3 prompt that relies on /nothink is not guaranteed to work on Qwen3.5.

d2
direction: down
weights: "One set of weights" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
modes: "Two behaviours" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 30
  think: "Thinking\nlong trace, then answer\nmore tokens, more cache" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  direct: "Direct\nanswer only\nfewer tokens" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
set_by: "Set by any of:\ntemplate argument\nsystem-prompt flag\nmodel-family default\nruntime default" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
weights -> modes
set_by -> modes

In the diagram, the yellow box is the problem: four different places can set the mode, and a record that names only the model has recorded none of them. Set the mode explicitly on every run and store it, the way RunRecord.thinking does.

Why tokens per second is not comparable across model families

A token is not a fixed amount of text. One sentence tokenizes to different counts under different tokenizers, so two models with equal decode rates can be delivering very different amounts of work. Meta reported that Llama 3's 128K-vocabulary tokenizer produces up to 15% fewer tokens than Llama 2's, and across families and languages the gap is much larger. I counted tokens for the same four strings with the Qwen3 and Gemma 3 tokenizers:

python
from transformers import AutoTokenizerSAMPLES = {    "english prose": "The invoice was rejected because the purchase order number did "                     "not match the vendor record, so the payment run skipped it.",    "python code": "def total(items: list[dict]) -> float:\n    return sum(i['qty'] * "                   "i['price'] for i in items if i.get('active'))",    "json": '{"order_id": 48213, "status": "rejected", "reason": "po_mismatch", '            '"retry": false}',    "hindi": "भुगतान रोक दिया गया क्योंकि खरीद आदेश संख्या मेल नहीं खाती थी।",}toks = {"qwen3": AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B"),        "gemma3": AutoTokenizer.from_pretrained("unsloth/gemma-3-270m-it")}print(f"{'sample':14} {'chars':>5} {'qwen3':>6} {'gemma3':>6} {'ratio':>6}")for label, text in SAMPLES.items():    n = {k: len(t(text, add_special_tokens=False)["input_ids"]) for k, t in toks.items()}    print(f"{label:14} {len(text):5} {n['qwen3']:6} {n['gemma3']:6} "          f"{n['qwen3'] / n['gemma3']:6.2f}")
text
sample         chars  qwen3 gemma3  ratioenglish prose    122     23     23   1.00python code      110     33     37   0.89json              82     30     32   0.94hindi             62     59     15   3.93

On English prose the two tokenizers tie, but on this one Hindi sentence Qwen3 needs 3.93 times as many tokens as Gemma 3. One sentence is not a corpus, so measure the ratio on your own text before you rely on it. A Qwen model decoding Hindi at 40 tokens per second would deliver about as much text per second as a Gemma model at about 10, so on a product that serves Hindi, a tokens-per-second comparison between these two families tells you almost nothing about the user's wait.

d2
direction: down
text: "Same Hindi sentence\n62 characters" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
tk: "Tokenizer output" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 30
  q: "Qwen3: 59 tokens" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  g: "Gemma 3: 15 tokens" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
unit: "Within one model: tokens/s is valid\nAcross families: measure characters/s or time to a full reply" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
text -> tk
tk -> unit

Tokens per second is a valid measurement within one model and an invalid one across families. When you compare families, measure something the user experiences: characters per second, or time to complete a realistic response on your own inputs. In record terms, model differing is the one variable, and the unit has to survive that change.

How small a difference your benchmark task set can detect

While the guard stops comparisons that cannot be attributed, it does nothing about comparisons that are too small to see. That is a separate check, and it needs two numbers.

First comes your noise floor. Run the same configuration several times and compare the records; since no controlled field varies, every delta in the result is movement from your own harness and machine.

On my laptop, five identical runs gave a decode range of 1.08 tokens per second and a median-TTFT range of 0.23 seconds, with no change in quality. Those are the thresholds below which I am not allowed to believe a delta from this harness on this machine. A single repeat would have reported 0.08 tokens per second, which is why two runs are not enough. Yours will differ, and you cannot know them without running the check. Teams that skip it end up trusting differences that are smaller than their harness's own variation.

Second is the resolution of your task set. On n examples with baseline accuracy p, treat a quality difference as unresolved unless it exceeds about 2.8 x sqrt(2p(1-p)/n), where 2.8 is 1.96 for 95% confidence plus 0.84 for 80% power. This is not a significance test. It is the smallest true difference the set would detect four times out of five, from the unpaired, conservative form of the power calculation in Evan Miller's Adding Error Bars to Evals. Inside one harness both models answer the same items, so Miller's paired analysis will usually give a smaller threshold. The formula also uses a normal approximation, which is poor below about 30 examples; there, use an exact test or a bootstrap instead.

Examples (n)Smallest reliably detectable difference at p = 0.7
1057 points
5026 points
20013 points
1,0006 points

Apply it to the thinking result above. Eight prompts at a baseline of 0.88 gives a threshold of about 46 points, and the measured drop was 37.5 points (printed as -0.38). So although the guard said the comparison was attributable, the resolution check says it is still not evidence, and eight prompts is below the size where the formula is reliable anyway. One answer moves eight-example accuracy by 12.5 points. Two decimal places on eight examples is false precision, which is one more reason to keep the task set's size in the record next to the score.

When you mean to change two things at once

Sometimes the honest comparison is between bundles. You may want to compare "Qwen in bf16 on transformers" against "the same model as a 4-bit GGUF on llama.cpp", because those are the two deployments on the table, and you have no plan to serve 4-bit weights through transformers. The guard will refuse that, and it should. Choosing between such deployments is a system decision, which is the subject of design patterns for SLM-first systems.

Instead of turning the guard off, make the bundle the variable. Give the record a deployment name (hf-bf16, gguf-q4km-llamacpp) and compare on it, with the parts kept as informational fields, so that the comparison says what it is about: two deployments, not quantization, not runtime. What you cannot do is quote the result as "4-bit costs X points", because the record now says plainly that it measured something else.

The one-variable rule has a second limit: it cannot see interactions. The thinking result above depends on max_tokens, and at a larger cap more traces would close, so the quality drop might shrink. A single comparison cannot tell you. When you suspect an interaction, run the small grid, here thinking on and off at two max_tokens values, and compare each cell with its neighbours. Each comparison is still one variable. The grid is what shows the dependence.

Existing benchmark harnesses and what they miss

None of the parts is new on its own. Biderman et al.'s Lessons from the Trenches shows prompt format alone moving GPT-NeoX-20B on ARC-Easy from 72.4% to 26.5%, and recommends releasing exact prompts and code. Sclar et al. measured formatting changes worth up to 76 accuracy points on LLaMA-2-13B. Miller's paper supplies the statistics, and Agrawal et al. catalogue evaluation anti-patterns in LLM serving systems as a checklist. NeMo Evaluator's documentation states the matching rule. On the tooling side, EleutherAI's lm-evaluation-harness writes its full configuration next to each result, and experiment trackers such as MLflow and Weights & Biases store the parameters of every run. All of them make the conditions available. None of them refuses a comparison when the conditions differ.

What I have not found is a harness that enforces the rule at the moment of comparison: that stores the conditions as part of every result and refuses to produce a delta when two of them differ. Every source above relies on a careful human, and the Silent Second Variable is a name for the failure that careful humans keep producing anyway. A guard at the moment of comparison is the cheapest place to stop it.

Benchmark harness checklist for your next model comparison

  1. Write down the one variable the comparison is about before you run anything.
  2. Fingerprint the task set. Adding examples creates a new set, so rename it and run the baseline again on the new version; one extra run buys a comparison that holds.
  3. Record every controlled field with every result: model and revision, quantization, runtime version, hardware, a fingerprint of the rendered prompts, thinking mode, task-set fingerprint, max tokens, batch size, warmup runs. Read what you can from the machine instead of typing it.
  4. Make the comparison refuse. Two records that differ in more than one controlled field raise, not warn.
  5. Report TTFT and decode rate separately, and never average per request when you can pool.
  6. Choose the warmup boundary from the deployment. Warm for a resident server, cold for a device that wakes per request. Store the choice.
  7. Set thinking mode explicitly on every run, and check whether a system-prompt flag overrides your argument.
  8. Do not compare tokens per second across families. Use characters per second or time to a full reply.
  9. Measure your noise floor with several zero-variable runs, five rather than two, before you believe any delta.
  10. Check the delta against your task set's resolution. At a baseline of 0.7 on 50 examples, the smallest difference you can reliably detect is about 26 points. Compute it for your own baseline and size.

A model comparison usually fails quietly, with correct numbers and a spreadsheet that looks finished. Explaining the MMLU incident took a public leaderboard and a team of evaluation specialists; on your laptop, the same failure costs a lost week, or a quantized model shipped because of a delta that was really a prompt file. Store the conditions, refuse the comparison, and the Silent Second Variable becomes a stack trace instead of a decision.

About the book: Right-Sized AI

This article is adapted from Chapter 2, "Your First Right-Sized Model", of my book Right-Sized AI: Small Language Models in Production. The chapter builds the full harness, its tests, and the memory-prediction check that goes with it. The rest of the book follows one method from constraint to device: pick the smallest model that can plausibly do the job, then fine-tune, distil, quantize, serve and ship it, with an evaluation gate at each step.

Right-Sized AI: Small Language Models in Production, by Ranjan Kumar

Available on Kindle and in paperback: right-sized-ai.ranjankumar.in

For the sizing arithmetic that comes before measurement, see the Footprint Multiple.

References

AI Engineering

LLM Deployment

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:

Books by Ranjan Kumar

Right-Sized AI cover

Right-Sized AI

Explore the Book
Harness Engineering for Production AI Systems cover

Harness Engineering

Explore the Book
The 7 GenAI Architectures cover

The 7 GenAI Architectures

Explore the Book
Building Real-World Agentic AI Systems with LangGraph cover

Building Real-World Agentic AI Systems

Explore the Book
The ChatML Handbook cover

The ChatML Handbook

Explore the Book
The Chat Templates Handbook cover

The Chat Templates Handbook

Explore the Book

Comments