How to Catch Chat Template Regressions with Golden-Token Tests

What you'll build: golden-token tests that fail when a chat template changes

A chat template can change what a model receives without raising anything. Here is the golden-token regression test this tutorial builds, run on a three-line "whitespace only" edit to Qwen3's template, the kind of diff a reviewer approves in seconds:

plain: char 46system: char 100multi_turn: char 100tool_round_trip: char 703reasoned_tool_call: char 703no_think: char 46expected: ·is·my·order?<|im_end|>\n<|im_start|>assistant\nactual:   ·is·my·order?<|im_end|>\n\n<|im_start|>assistant\n

All six recorded conversations failed. The edit added a newline after every user turn, so the model now sees a blank line it was never trained on before <|im_start|>assistant. The template still renders, the model still answers, and nothing in Transformers or the server reports it. Only a stored copy of the old output can tell you it moved. That stored copy is what the check is built on: you record, once, the exact string a template renders and the exact token IDs an engine sends for a small set of conversations, then compare every later change against that record. This intermediate tutorial is for engineers who own a served model's chat template, or the CI around it.

By the end you will have templatestudio/golden.py, a JSON file of reviewed goldens, and a pytest test that fails CI when either the rendered string or the reference tokenizer's token IDs change; a separate script checks the same goldens through llama-server. You will also walk through four incidents from chapter 11 of The Chat Templates Handbook, each from its first symptom to the check that catches it: the whitespace edit above, a misspelled enable_thinking switch, a doubled <bos> from a client that renders its own prompt, and a fine-tune that never stops, which no golden can catch. Budget about 45 minutes. Everything runs on a CPU with no API key, no account and no Hugging Face token.

Verified against Python 3.12.2, transformers==5.22.1, tokenizers==0.23.3, huggingface-hub==2.4.0, jinja2==3.1.6, pytest==9.2.0, and llama.cpp b12283 (llama-server) on 2026-11-11. The two llama-servers were started by hand, as printed in Prerequisites; serve_check.py confirms their build and models.

This is Part 11 of Chat Templates in Practice and continues the project from Part 10, How to Check the Token IDs llama-server Sends the Model. Part 10 built adapters that ask an engine for token IDs. This part stores what they return and turns any later difference into a failing test.

Prerequisites: the Part 10 project, pytest 9.2.0 and two llama-servers

You need the chat-templates-part1/ project from the earlier parts, with its virtual environment and these files:

• templatestudio/__init__.py (Part 1, empty)
• templatestudio/render.py (Part 1, last changed in Part 6): load_tokenizer, render, render_ids, visible_whitespace, first_divergence
• templatestudio/messages.py (Part 2, last changed in Part 7)
• templatestudio/tools.py (Part 5): tool_schema, tool_call_message, tool_result_message
• templatestudio/serving.py (Part 10): TokenRenderer (any engine adapter with a name and an ids(conversation, **kwargs) method), TransformersTokens, LlamaCppTokens, ClientCompletionTokens, request_tool
• at the project root, from Part 10: endpoints.py (GEMMA_URL, GEMMA_TOKENIZER, QWEN_URL), session.py (get_weather) and serve_check.py

Step 13 is optional. It also needs templatestudio/jinja_env.py from Part 3, and three things from Part 8: templatestudio/authoring.py, support_data.py, and the saved support-model/ folder with its trained_row.json. authoring.py imports only Transformers and the earlier parts' modules, so step 13 needs none of Part 8's training packages. Use a support-model/ saved by a Part 8 run on the same Transformers pin, 5.22.1. If a file is missing, copy it from the complete code of the part that last changed it: Part 5, Part 6, Part 7, Part 8 or Part 10. This part adds one module, templatestudio/golden.py, two data files (golden_cases.py and whitespace_commit.py), a goldens/ folder, a tests/ folder, a saved tokenizer folder and thirteen scripts. It changes no earlier file.

Go to the project folder, activate the environment, and turn off the Transformers advisory warnings. Run the cd from the folder that contains chat-templates-part1/:

cd chat-templates-part1source .venv/bin/activateexport TRANSFORMERS_NO_ADVISORY_WARNINGS=1

On Windows, activate with .venv\Scripts\activate in PowerShell or source .venv/Scripts/activate in Git Bash, and set the variable in PowerShell with $env:TRANSFORMERS_NO_ADVISORY_WARNINGS = "1". Every script in this part runs from the project root, because the goldens file is opened by the relative path goldens/qwen3.json.

The Python pins from Parts 1 to 10 are unchanged: Python 3.12 (verified on 3.12.2), transformers 5.22.1, tokenizers 0.23.3, huggingface-hub 2.4.0 and jinja2 3.1.6. Nothing in this part needs PyTorch. Transformers 5.22.2 (2026-11-06) hardens the Jinja sandbox against inputs this tutorial never sends, and the series moves to it in one step at the end. This part adds one Python package, pytest, for step 9:

python -m pip install pytest==9.2.0

Check both versions before step 1:

python -c "import transformers as t, pytest as p; print(t.__version__, p.__version__)"

5.22.1 9.2.0

Steps 8 and 9 need the two llama-servers from Part 10, on llama.cpp build b12283. Start them from the folder you extracted llama.cpp into, each in its own terminal, and leave both running:

./llama-server -hf ggml-org/gemma-3-270m-it-GGUF:Q8_0./llama-server -hf Qwen/Qwen3-0.6B-GGUF:Q8_0 --port 9932

The Gemma command has no --port because 9931 is llama-server's built-in default port since llama.cpp v0.7.0; no environment variable or config file sets it. The Qwen3 server needs a port of its own, so its command passes one. On Windows, run .\llama-server.exe with the same arguments. b12283 is no longer the newest build (b12284 followed the same day), and a later build can change the server's half of the render, which is what steps 8 and 9 test. Both models are in your Hugging Face cache from Part 10, and so are the two tokenizers this part loads, Qwen/Qwen3-0.6B and unsloth/gemma-3-270m-it. Then confirm both servers from the project terminal with Part 10's serve_check.py:

python serve_check.py

http://localhost:9931: ok, build b12283-f3b407ca1  model: ggml-org/gemma-3-270m-it-GGUF:Q8_0  template starts: {{ bos_token }}http://localhost:9932: ok, build b12283-f3b407ca1  model: Qwen/Qwen3-0.6B-GGUF:Q8_0  template starts: {%- if tools %}

Every other step runs without a server. A few code blocks below start with a comment such as # templatestudio/golden.py (add at the end) or (replace check_goldens). That line tells you where the code goes. It is not part of the code, so do not paste it.

How a golden-token test works: record once, compare on every change

A golden is the exact output of a deterministic function for one fixed input, stored so a later run can be compared with it. A chat template is that kind of function. For a given conversation and a given set of switches, it has exactly one correct output, down to the last newline. It is a snapshot test, specialised for chat templates. Chapter 11 of the book calls the test built on this the golden-token test: for a fixed set of conversations, store the token IDs the engine sends the model, store the rendered string too so a person can read what changed, and fail on any change to either.

Each golden case has two halves, and they answer different questions. The string half re-renders the case with the tokenizer's template and compares characters. It runs anywhere, and when it fails it tells you where in the text the change is. The token half asks an engine for the case's token IDs and compares them with stored IDs. It is the only half that sees what happens after the string is finished, such as a server adding a second <bos>. The diagram shows the whole lifecycle you build in this tutorial:

direction: down
classes: {
  input: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
once: "1. Record once" {
  grid-rows: 1
  grid-gap: 24
  label.near: top-center
  cases: "Golden cases\nconversations + switches" {class: input}
  record: "record from a template\nyou checked by eye" {class: act}
  file: "goldens/qwen3.json\nreviewed in code review" {class: store}
}
halves: "2. Check both halves" {
  grid-rows: 1
  grid-gap: 30
  label.near: top-center
  str: "String half\nrender with the tokenizer,\ncompare characters" {class: check}
  tok: "Token half\nask an engine for IDs,\ncompare token by token" {class: check}
}
pass: "Empty list: pass" {class: good}
fail: "GoldenFailure\ncase, unit, position" {class: bad}
once -> halves: "on every change: template edit,\ntokenizer bump, engine upgrade, server flag"
halves -> pass: "nothing moved"
halves -> fail: "something moved"

Notice where the goldens file sits: between recording and checking, as data a person reviews. Recording is not part of the check. A check that re-records whenever it fails passes forever and tests nothing, so step 12 makes re-recording a separate script that refuses to write until you have read the diff.

Chapter 11 also gives a five-stage procedure for a template bug, and the incidents in this tutorial follow it. Reproduce the failing conversation with the model's own tokenizer or the engine you serve. Make it visible with Part 1's visible_whitespace. Diff against the last known-good output; the first divergence points at the bug. Bisect a large change until one piece is left. Lock the fix with a golden case, so the same bug fails CI if it comes back. Steps 4, 5 and 7 label each stage as they reach it.

Step 1: Define a golden case for a chat template

Goal: create templatestudio/golden.py with the two data types every later step uses.

Why this step: a golden test needs a fixed description of what to render and what it must produce, and it needs a failure report that says where the output moved. A GoldenCase holds one conversation and the switches it is rendered with, plus its two goldens, which stay None until you record them. The one design decision here is the kwargs field. The same dict reaches the string render and the engine, so a case recorded with enable_thinking=False is checked with that switch on both sides, and the two halves can never test different requests. GoldenFailure carries a short window around the first difference rather than the whole output, because a 900-character tool prompt printed twice hides the one character that changed.

direction: down
classes: {
  field: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  gold: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  use: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
case: "GoldenCase" {
  grid-rows: 2
  grid-columns: 3
  grid-gap: 16
  label.near: top-center
  name: "name" {class: field}
  conversation: "conversation" {class: field}
  kwargs: "kwargs\ntools, enable_thinking, ..." {class: field}
  expected: "expected\nthe rendered string" {class: gold}
  ids: "expected_ids\na tuple of token IDs" {class: gold}
  engine: "engine\nwho recorded the IDs" {class: field}
}
string: "String half\nrender(tokenizer, conversation, **kwargs)" {class: use}
token: "Token half\nengine.ids(conversation, **kwargs)" {class: use}
case -> string: "same kwargs"
case -> token: "same kwargs"

The yellow fields are the goldens. Everything else describes the request, and both halves read it identically.

Create templatestudio/golden.py. This first block holds the module's whole import block, so later steps only append functions:

# templatestudio/golden.py"""Golden-token tests: record what a template renders and what the enginesends for fixed conversations, then fail on any change to either."""from __future__ import annotationsimport jsonfrom dataclasses import asdict, dataclass, field, replacefrom pathlib import Pathfrom typing import Anyfrom transformers import PreTrainedTokenizerBasefrom templatestudio.messages import Conversationfrom templatestudio.render import first_divergence, renderfrom templatestudio.serving import TokenRendererCHAR_WINDOW = 24  # characters shown either side of a string failureTOKEN_WINDOW = 4  # token IDs shown, starting at the first one that differs@dataclass(frozen=True)class GoldenCase:    """One fixed conversation and the exact output it must keep producing.    kwargs reach both the string render and the engine, so a case with    {"enable_thinking": False} is checked with that switch on both sides.    expected is the rendered string; expected_ids are the token IDs one    engine sent for it. Either stays None until it has been recorded.    """    name: str    conversation: Conversation    kwargs: dict[str, Any] = field(default_factory=dict)    expected: str | None = None    expected_ids: tuple[int, ...] | None = None    engine: str = ""  # the engine that recorded expected_ids@dataclass(frozen=True)class GoldenFailure:    name: str    unit: str  # "char" for the string half, "token" for the ID half    position: int  # first index where the output left the golden    expected: str  # the golden near that position: chars around it, IDs from it    actual: str  # the same window of what was produced insteaddef render_case(tokenizer: PreTrainedTokenizerBase, case: GoldenCase) -> str:    """The string half: the case rendered by the tokenizer's template."""    return render(tokenizer, case.conversation, **case.kwargs)

render_case is Part 1's render with the case's switches passed through. render already defaults to add_generation_prompt=True, which is what a served request uses, and a case that needs the other value puts it in kwargs, as step 13 does.

Run it: create golden_types.py, which builds one case and prints each field:

# golden_types.py"""Build one case and print each field, to see what a case holds."""from templatestudio.golden import GoldenCasecase = GoldenCase("plain", [{"role": "user", "content": "Hi"}])for name, value in vars(case).items():    print(f"{name:<13} {value!r}")

python golden_types.py

name          'plain'conversation  [{'role': 'user', 'content': 'Hi'}]kwargs        {}expected      Noneexpected_ids  Noneengine        ''

What just happened: templatestudio/golden.py imports cleanly, so render.py, messages.py and serving.py are in place; step 2 also needs tools.py and session.py. Both goldens are None because nothing has been recorded yet. The empty kwargs dict means this case renders with no switches. Step 2 fills in expected.

Step 2: Record golden strings and save them as a reviewed JSON file

Goal: record the rendered string for six conversation shapes and write them to goldens/qwen3.json.

Why this step: the cases should cover the shapes that break, not only a happy path. Chapter 11's advice is to seed them from its failure catalog, the chapter's table of every template failure in the book and the check that catches each one: a system message, several turns, a tool round trip, a turn with kept reasoning, and every template switch you set in production, at the value you set it to. The goldens go into a JSON file because a person has to review them. When a golden changes on purpose, the change shows up in code review as a diff someone can read, and code review is where a bad template edit should be stopped. indent=2 and ensure_ascii=False keep that diff line by line and readable.

direction: right
classes: {
  input: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
}
cases: "golden_cases.py\nCASES, 6 cases" {class: input}
record: "record_goldens\nfills expected" {class: act}
save: "save_goldens\nindented JSON" {class: act}
file: "goldens/qwen3.json" {class: store}
cases -> record -> save -> file

Add the recording and file functions to the module:

# templatestudio/golden.py (add at the end)def record_goldens(    tokenizer: PreTrainedTokenizerBase, cases: list[GoldenCase]) -> list[GoldenCase]:    """Fill in expected for every case from the current template. Run it on    a template you have checked by eye, never to make a failing check pass."""    return [replace(case, expected=render_case(tokenizer, case)) for case in cases]def save_goldens(path: str | Path, cases: list[GoldenCase]) -> None:    """Write cases as indented JSON, so a changed golden reviews as a diff."""    data = [asdict(case) for case in cases]    text = json.dumps(data, indent=2, ensure_ascii=False)    # newline="\n" writes the same bytes on Windows, Linux and macOS.    Path(path).write_text(text + "\n", encoding="utf-8", newline="\n")def load_goldens(path: str | Path) -> list[GoldenCase]:    """Read cases back. JSON has no tuples, so expected_ids comes back as a    list; turning it into a tuple again makes a loaded case equal the saved one."""    data = json.loads(Path(path).read_text(encoding="utf-8"))    cases = []    for item in data:        if item["expected_ids"] is not None:            item["expected_ids"] = tuple(item["expected_ids"])        cases.append(GoldenCase(**item))    return cases

One line in load_goldens needs explaining. JSON has no tuple type, so the token IDs you store in step 6 come back as a list, and a tuple never equals a list in Python: (1, 2) == [1, 2] is False. The check itself does not care, because token_failure in step 6 compares lists. The tuple(...) call is there so a loaded case matches its tuple[int, ...] type and equals the case you saved, which matters as soon as you compare cases.

Create golden_cases.py with the six cases. It reuses Part 10's get_weather tool and request_tool, and Part 5's message helpers:

# golden_cases.py"""Part 11: the conversation shapes the Qwen3 goldens cover."""from session import get_weatherfrom templatestudio.golden import GoldenCasefrom templatestudio.serving import request_toolfrom templatestudio.tools import tool_call_message, tool_result_message, tool_schemaORDER = [{"role": "user", "content": "Where is my order?"}]SUPPORT = [    {"role": "system", "content": "You are a support agent."},    *ORDER,    {"role": "assistant", "content": "Let me check."},    {"role": "user", "content": "It is order 1042."},]WEATHER = [{"role": "user", "content": "What is the weather in Paris?"}]CALL = tool_call_message("get_weather", {"city": "Paris"})REASONED_CALL = {**CALL, "reasoning_content": "The user wants the weather."}TOOLS = [request_tool(tool_schema(get_weather))]# The tokenizer folder your service loads; step 10 saves it, CI tests it.SHIPPED = "qwen3-tokenizer"CASES = [    GoldenCase("plain", ORDER),    GoldenCase("system", SUPPORT[:2]),    GoldenCase("multi_turn", SUPPORT),    GoldenCase(        "tool_round_trip",        [*WEATHER, CALL, tool_result_message("get_weather", "sunny")],        kwargs={"tools": TOOLS},    ),    GoldenCase(        "reasoned_tool_call",        [*WEATHER, REASONED_CALL, tool_result_message("get_weather", "sunny")],        kwargs={"tools": TOOLS},    ),    GoldenCase("no_think", ORDER, kwargs={"enable_thinking": False}),]

request_tool reduces the schema to the keys every engine reads, and drops the return key llama-server ignores, as Part 10 found. That is what lets step 9 check these same cases through the server. The reasoned_tool_call case carries reasoning_content on the assistant turn that calls the tool. Qwen3 keeps reasoning for turns after the last user message, as Part 6 showed, so this case puts a <think> block in the golden. Create a folder for the goldens, then golden_record.py:

mkdir goldens

# golden_record.py"""Record a golden string for every case and write goldens/qwen3.json."""from golden_cases import CASESfrom templatestudio.golden import record_goldens, save_goldensfrom templatestudio.render import load_tokenizerqwen = load_tokenizer("Qwen/Qwen3-0.6B")cases = record_goldens(qwen, CASES)save_goldens("goldens/qwen3.json", cases)for case in cases:    print(f"{case.name:<20} {len(case.expected):>4} chars")

Run it:

python golden_record.py

plain                  68 charssystem                122 charsmulti_turn            213 charstool_round_trip       904 charsreasoned_tool_call    950 charsno_think               87 chars

Expected file: goldens/qwen3.json starts like this:

[  {    "name": "plain",    "conversation": [      {        "role": "user",        "content": "Where is my order?"      }    ],    "kwargs": {},    "expected": "<|im_start|>user\nWhere is my order?<|im_end|>\n<|im_start|>assistant\n",    "expected_ids": null,    "engine": ""  },

What just happened: six goldens are on disk, recorded from Qwen3-0.6B's own template. expected_ids is still null; step 6 records it. The plain case's 68 characters are the whole prompt for one user message, ending with the generation prompt <|im_start|>assistant\n.

Step 3: Check a chat template against its goldens

Goal: add the string half of the check and run it against the unchanged template.

Why this step: a test has to pass on the code it was recorded from before its failures mean anything. If the check failed here, the cause would be the check itself, or a case whose render is not stable, and every later failure would be noise. string_failure uses Part 1's first_divergence, which works on any two sequences, strings or lists of token IDs, and returns None when they are equal. Here it reports the first character that moved, and slices CHAR_WINDOW characters either side of it.

direction: down
classes: {
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
golden: "case.expected\nfrom goldens/qwen3.json" {class: store}
now: "render_case(tokenizer, case)\ntoday's template" {class: act}
diff: "first_divergence\n(Part 1)" {class: check}
none: "None: no failure" {class: good}
fail: "GoldenFailure\nunit char, position,\n24 chars either side" {class: bad}
golden -> diff
now -> diff
diff -> none: "equal"
diff -> fail: "differs at N"

Add the string half at the end of templatestudio/golden.py:

# templatestudio/golden.py (add at the end)def string_failure(case: GoldenCase, actual: str) -> GoldenFailure | None:    """Compare one render with its golden string; None if they are equal."""    assert case.expected is not None    position = first_divergence(case.expected, actual)    if position is None:        return None    start = max(position - CHAR_WINDOW, 0)    end = position + CHAR_WINDOW    return GoldenFailure(        name=case.name,        unit="char",        position=position,        expected=case.expected[start:end],        actual=actual[start:end],    )def check_goldens(    tokenizer: PreTrainedTokenizerBase, cases: list[GoldenCase]) -> list[GoldenFailure]:    """Re-render every case that has a golden string. Empty means pass."""    failures = []    for case in cases:        if case.expected is None:            continue        failure = string_failure(case, render_case(tokenizer, case))        if failure is not None:            failures.append(failure)    return failures

Create golden_check.py:

# golden_check.py"""Load the goldens and check the current template against them."""from templatestudio.golden import check_goldens, load_goldensfrom templatestudio.render import load_tokenizerqwen = load_tokenizer("Qwen/Qwen3-0.6B")cases = load_goldens("goldens/qwen3.json")failures = check_goldens(qwen, cases)print(f"{len(cases)} cases, {len(failures)} failures")

Run it:

python golden_check.py

6 cases, 0 failures

What just happened: the six goldens match the template they were recorded from, and that gives you a baseline. Recording and checking both go through render_case, so the check adds no difference of its own. A later failure means the template or tokenizer moved, and it turns this 0 into a list of failures, each with a case name and a character position.

Step 4: Catch a whitespace-only template change with a golden string

Goal: apply a three-line tidy-up to Qwen3's template and watch the string goldens catch it.

Why this step: chapter 11's first incident starts with a tidy-up. Someone fixes the indentation of a shared template, the diff looks like whitespace, review approves it, and quality drops a little on every model that uses the template. A user reported that kind of drop after a whitespace change to Phi-3-mini's template, in a Hugging Face discussion. The trap is in how Jinja handles whitespace. Transformers renders with trim_blocks and lstrip_blocks, so the spaces at the start of a tag's line are removed either way. The newline before a tag is removed only by the - in {%-. Re-indenting a line is harmless. Dropping that dash is not, and in a diff the two look the same. Part 3 showed what one missing dash does to a single render; here a golden catches it across six conversations. Part 4 pointed to this kind of template-drift test as the next step after its family table.

direction: down
grid-rows: 1
grid-gap: 40
classes: {
  code: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
before: "Before: {%- elif" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  a: "user turn ends: <|im_end|> + newline" {class: code}
  b: "{%- strips the newline\nbefore the elif tag" {class: good}
  c: "<|im_end|>\\n<|im_start|>assistant" {class: good}
}
after: "After: {% elif" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  a: "user turn ends: <|im_end|> + newline" {class: code}
  b: "the template's own newline\nnow lands in the output" {class: bad}
  c: "<|im_end|>\\n\\n<|im_start|>assistant" {class: bad}
}

The newline in question is the one after the user branch's output line in Qwen3's template. It belongs to the user branch, so once the dash on the next tag is gone, every user turn ends with one extra newline. Create whitespace_commit.py with the edit as three hunks, each one line of the template and its replacement. Two hunks only re-indent a line. The third looks like a re-indent too, but it also drops the dash:

# whitespace_commit.py"""Part 11: a tidy-up of Qwen3's template that reads as whitespace only.Each hunk replaces one line of the template with a re-indented copy."""HUNKS = [    (        '    {%- set index = (messages|length - 1) - loop.index0 %}',        '  {%- set index = (messages|length - 1) - loop.index0 %}',    ),    (        '        {%- set content = message.content %}',        '    {%- set content = message.content %}',    ),    (        '    {%- elif message.role == "assistant" %}',        '    {% elif message.role == "assistant" %}',    ),]def apply_hunks(template: str, hunks: list[tuple[str, str]]) -> str:    """Apply each (old line, new line) hunk; every old line must be there."""    for old, new in hunks:        assert template.count(old) == 1, f"hunk not found once: {old!r}"        template = template.replace(old, new)    return template

Create golden_commit.py. It applies the hunks to the loaded tokenizer in memory, the reproduce stage, and prints the first failure's window through Part 1's visible_whitespace, the make it visible stage. Step 11 applies the same hunks to a template file on disk. The script calls sys.stdout.reconfigure(encoding="utf-8") so the middle dots visible_whitespace prints show on a Windows console, as in Part 1:

# golden_commit.py"""Apply the whitespace commit to Qwen3's template and check the goldens."""import sysfrom templatestudio.golden import check_goldens, load_goldensfrom templatestudio.render import load_tokenizer, visible_whitespacefrom whitespace_commit import HUNKS, apply_hunkssys.stdout.reconfigure(encoding="utf-8")qwen = load_tokenizer("Qwen/Qwen3-0.6B")qwen.chat_template = apply_hunks(qwen.chat_template, HUNKS)failures = check_goldens(qwen, load_goldens("goldens/qwen3.json"))for failure in failures:    print(f"{failure.name}: {failure.unit} {failure.position}")first = failures[0]# Each window ends in a newline already, so print adds none.print("expected:", visible_whitespace(first.expected), end="")print("actual:  ", visible_whitespace(first.actual), end="")

Run it:

python golden_commit.py

plain: char 46system: char 100multi_turn: char 100tool_round_trip: char 703reasoned_tool_call: char 703no_think: char 46expected: ·is·my·order?<|im_end|>\n<|im_start|>assistant\nactual:   ·is·my·order?<|im_end|>\n\n<|im_start|>assistant\n

What just happened: every case failed at the end of its first user turn. plain and no_think fail at character 46, the end of <|im_start|>user\nWhere is my order?<|im_end|>\n. system and multi_turn fail at 100, after the system turn, and the tool cases at 703, after the long # Tools system block. Comparing the two windows is the diff stage. visible_whitespace shows a middle dot for each space and \n for each newline, and the actual window has a line the expected window does not: the extra newline. Here the run disagrees with the book. The book says the extra newline appears before the first assistant header only. On Qwen3-0.6B it appears after every user turn, because the newline belongs to the user branch.

Step 5: Bisect a template change to the hunk that moved the output

Goal: find which of the commit's three hunks changed the output, the bisect stage.

Why this step: a failing golden tells you that the output moved and where. It does not tell you which part of a change moved it. With a large diff, the fastest route is to bisect: apply part of the change, check, and narrow it down. This commit has three independent hunks, so applying each one alone is enough. A real refactor with dozens of hunks would split them in halves instead.

direction: down
classes: {
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
orig: "original template" {class: act}
hunks: "one hunk at a time" {
  grid-rows: 1
  grid-gap: 24
  label.near: top-center
  h1: "hunk 1: re-indent\n0 of 6 fail" {class: good}
  h2: "hunk 2: re-indent\n0 of 6 fail" {class: good}
  h3: "hunk 3: drop a dash\n6 of 6 fail" {class: bad}
}
orig -> hunks: "apply + check_goldens"

Create golden_bisect.py. It keeps the original template and applies each hunk to a fresh copy:

# golden_bisect.py"""Apply the commit's hunks one at a time to find the one that moves output."""from templatestudio.golden import check_goldens, load_goldensfrom templatestudio.render import load_tokenizerfrom whitespace_commit import HUNKS, apply_hunksqwen = load_tokenizer("Qwen/Qwen3-0.6B")original = qwen.chat_templatecases = load_goldens("goldens/qwen3.json")for number, hunk in enumerate(HUNKS, start=1):    qwen.chat_template = apply_hunks(original, [hunk])    failures = check_goldens(qwen, cases)    print(f"hunk {number}: {len(failures)} of {len(cases)} cases fail")    if failures:        print("  -", hunk[0].strip())        print("  +", hunk[1].strip())

Run it:

python golden_bisect.py

hunk 1: 0 of 6 cases failhunk 2: 0 of 6 cases failhunk 3: 6 of 6 cases fail  - {%- elif message.role == "assistant" %}  + {% elif message.role == "assistant" %}

What just happened: two hunks changed indentation only, and lstrip_blocks removes indentation, so they moved nothing. The third removed one character, the -, and moved every case. The fix is to restore the dash. You found it without reading the template's 89 lines; the goldens pointed at the line. The lock stage is already done. The six goldens caught the bug and stay in the file, so the same edit fails again if anyone repeats it. For a bug no golden caught, lock means adding the case that would have caught it.

Step 6: Add golden token IDs from the reference engine

Goal: add the token half of the check, and record each case's token IDs from the Transformers reference.

Why this step: a string golden checks the template. It cannot check anything that happens after the string is finished. Part 10 called that point the serving boundary and showed what happens there: the server parses the request, may drop a field, and tokenizes with its own rules about <bos>. Token IDs are what the model actually receives, so they are what the test should assert. The IDs here come from TransformersTokens, Part 10's reference adapter, which runs the model's own template and tokenizer. Steps 8 and 9 check llama-server against IDs recorded the same way.

direction: down
classes: {
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  eng: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
ref: "TransformersTokens\nthe reference (Part 10)" {class: eng}
file: "goldens/qwen3.json\nexpected + expected_ids" {class: store}
check: "check_goldens(..., engine=)" {
  grid-rows: 1
  grid-gap: 30
  label.near: top-center
  tok: "token_failure\nfirst token that moved,\n4 IDs from there" {class: check}
  str: "string_failure\nfirst character that moved" {class: check}
}
ref -> file: "record_golden_ids"
file -> check: "every change"

Replace check_goldens in templatestudio/golden.py with the version below. Its first if is new: when you pass an engine, every case with stored IDs is also sent through it:

# templatestudio/golden.py (replace check_goldens)def check_goldens(    tokenizer: PreTrainedTokenizerBase,    cases: list[GoldenCase],    *,    engine: TokenRenderer | None = None,) -> list[GoldenFailure]:    """Re-render every case that has a golden string and, given an engine,    re-send every case that has golden IDs. Empty means pass."""    failures = []    for case in cases:        if engine is not None and case.expected_ids is not None:            failure = token_failure(case, engine.ids(case.conversation, **case.kwargs))            if failure is not None:                failures.append(failure)        if case.expected is not None:            failure = string_failure(case, render_case(tokenizer, case))            if failure is not None:                failures.append(failure)    return failures

Then add the recording function and the token comparison at the end of the file:

# templatestudio/golden.py (add at the end)def record_golden_ids(engine: TokenRenderer, cases: list[GoldenCase]) -> list[GoldenCase]:    """Fill in expected_ids for every case from what the engine sends."""    return [        replace(            case,            expected_ids=tuple(engine.ids(case.conversation, **case.kwargs)),            engine=engine.name,        )        for case in cases    ]def token_failure(case: GoldenCase, actual: list[int]) -> GoldenFailure | None:    """Compare one engine's IDs with the golden IDs; None if they are equal."""    assert case.expected_ids is not None    expected = list(case.expected_ids)    position = first_divergence(expected, actual)    if position is None:        return None    end = position + TOKEN_WINDOW    return GoldenFailure(        name=case.name,        unit="token",        position=position,        expected=str(expected[position:end]),        actual=str(actual[position:end]),    )

Create golden_record_ids.py. It loads the string goldens, adds the IDs and saves the file again:

# golden_record_ids.py"""Add the reference engine's token IDs to every golden case."""from templatestudio.golden import load_goldens, record_golden_ids, save_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensqwen = load_tokenizer("Qwen/Qwen3-0.6B")cases = record_golden_ids(TransformersTokens(qwen), load_goldens("goldens/qwen3.json"))save_goldens("goldens/qwen3.json", cases)for case in cases:    print(f"{case.name:<20} {len(case.expected_ids):>4} ids from {case.engine}")

Run it:

python golden_record_ids.py

plain                  13 ids from transformerssystem                 24 ids from transformersmulti_turn             47 ids from transformerstool_round_trip       198 ids from transformersreasoned_tool_call    208 ids from transformersno_think               17 ids from transformers

Now replace golden_check.py so it checks both halves:

# golden_check.py"""Load the goldens and check the template, and the reference engine's IDs."""from templatestudio.golden import check_goldens, load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensqwen = load_tokenizer("Qwen/Qwen3-0.6B")cases = load_goldens("goldens/qwen3.json")failures = check_goldens(qwen, cases, engine=TransformersTokens(qwen))print(f"{len(cases)} cases, {len(failures)} failures")

Run it:

python golden_check.py

6 cases, 0 failures

What just happened: every case now carries both goldens, and expected_ids in goldens/qwen3.json is a list of numbers instead of null. The reference reproduced its own recording exactly, so from here on a failure means the template, the tokenizer or the engine moved, not noise in the check. A token failure with no string failure points past the string, at what happens after rendering. Note the plain case's length, 13 token IDs. It comes back in the next step.

Step 7: Catch a misspelled enable_thinking switch that silently does nothing

Goal: send the no_think case with its switch misspelled, enable_thinkng=False, and see what catches it.

Why this step: chapter 11's next incident starts with a configuration change meant to turn Qwen3's thinking off. Replies still arrive with reasoning, and no error appears anywhere, because nothing checks the name of a template switch. Transformers passes every extra keyword argument into the template and never warns about one the template does not read. Qwen3's template tests enable_thinking is defined and enable_thinking is false. A misspelled name leaves enable_thinking undefined, so the template renders exactly what it renders when you pass nothing. The same kind of mistake shows up in public. On the vLLM forum, a user set {"thinking": false} and the model kept thinking. The reply pointed out that Qwen3's switch is named enable_thinking, not thinking. That thread's model was a thinking-only variant, which reasons whatever the switch says, so the corrected name did not turn its thinking off either.

direction: down
grid-rows: 1
grid-gap: 40
classes: {
  input: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
right: "enable_thinking=False" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  k: "template reads the switch" {class: input}
  r: "17 IDs, ending\n<think> \\n\\n </think> \\n\\n" {class: good}
}
typo: "enable_thinkng=False" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  k: "template never reads it" {class: input}
  r: "13 IDs, the same as\npassing no switch at all" {class: bad}
}

Create golden_typo.py. It takes the recorded no_think case and changes only its kwargs:

# golden_typo.py"""Send the no_think case with the switch's name misspelled."""from dataclasses import replacefrom templatestudio.golden import check_goldens, load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensqwen = load_tokenizer("Qwen/Qwen3-0.6B")case = next(c for c in load_goldens("goldens/qwen3.json") if c.name == "no_think")typo = replace(case, kwargs={"enable_thinkng": False})for failure in check_goldens(qwen, [typo], engine=TransformersTokens(qwen)):    print(f"{failure.unit} {failure.position}")    print(f"  expected {failure.expected!r}")    print(f"  actual   {failure.actual!r}")

Run it:

python golden_typo.py

token 13  expected '[151667, 271, 151668, 271]'  actual   '[]'char 68  expected '>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'  actual   '>\n<|im_start|>assistant\n'

What just happened: reading these two failures is the diff stage again. The token half reports position 13, where four IDs are missing: 151667 is <think>, 271 is \n\n, and 151668 is </think>. Together they form the empty reasoning block the switch exists to produce. The string half reports the same change as characters, at position 68. Note where the typo lives: in the test case itself. A golden sends only its own case's kwargs. It catches a typo in production code only when the case's kwargs come from that code. So build each case's switches by calling the same function your service uses to build its requests, not by typing them by hand. Then a typo there reaches the case.

A note on the book's version. Chapter 11 says only a token golden catches this incident. Its example builds a token-only case, so there the claim holds. Here both halves catch it, because both render with the same kwargs. The token half is the one that matters when the switch lives in a server's configuration, such as llama-server's --chat-template-kwargs startup flag, which a string render in your test never sees:

• A server flag that changes the prompt: step 9's goldens fail, because the server's IDs move.
• A misspelled server flag: nothing changes, so every golden still passes. Catching it needs a golden recorded with the value you intend and sent with no switch in the request, so only the server's flag can produce the expected IDs. That needs different switches for recording and for sending, which this part's one-kwargs design does not have; treat it as an extension.

Step 8: Catch a double BOS in llama.cpp by checking every production path

Goal: check one Gemma golden through both ways a request can reach llama-server.

Why this step: the next incident from chapter 11 is the doubled <bos> from Part 10. A token golden only sees the path it runs through, and a service often has more than one path: the chat endpoint, which receives messages, and an older client that renders the prompt string itself and posts it to /completion. Gemma 3's template writes <bos> into the string, and /completion adds another when it tokenizes. The rule has two parts. Check one golden through every production path. Take the expected IDs from the reference, never from the path under test, because a golden recorded from the broken path would lock the bug in.

direction: down
classes: {
  eng: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
ref: "TransformersTokens(gemma)\nthe reference" {class: eng}
golden: "golden: 14 IDs\none <bos>" {class: store}
paths: "Each production path" {
  grid-rows: 1
  grid-gap: 30
  label.near: top-center
  chat: "LlamaCppTokens\nmessages to the chat path\n0 failures" {class: good}
  client: "ClientCompletionTokens\nclient's string to /completion\nfails at token 1: second <bos>" {class: bad}
}
ref -> golden: "record_golden_ids"
golden -> paths: "check_goldens(engine=path)"

Create golden_paths.py. It records the golden from the reference, then checks it through each path. functools.partial(render, gemma) is the client's own render function, as in Part 10:

# golden_paths.py"""Check one Gemma golden through every path a request takes to the model.The golden IDs always come from the reference, never from a path under test."""from functools import partialfrom endpoints import GEMMA_TOKENIZER, GEMMA_URLfrom golden_cases import ORDERfrom templatestudio.golden import GoldenCase, check_goldens, record_golden_idsfrom templatestudio.render import load_tokenizer, renderfrom templatestudio.serving import (    ClientCompletionTokens,    LlamaCppTokens,    TransformersTokens,)gemma = load_tokenizer(GEMMA_TOKENIZER)reference = TransformersTokens(gemma)[case] = record_golden_ids(reference, [GoldenCase("order", ORDER)])print(f"golden: {len(case.expected_ids)} ids from {case.engine}")paths = [    LlamaCppTokens(GEMMA_URL),    ClientCompletionTokens(partial(render, gemma), GEMMA_URL),]for path in paths:    failures = check_goldens(gemma, [case], engine=path)    print(f"{path.name}: {len(failures)} failures")    for failure in failures:        print(f"  token {failure.position}")        print(f"    expected {failure.expected}")        print(f"    actual   {failure.actual}")

Run it with the Gemma server running on port 9931:

python golden_paths.py

golden: 14 ids from transformersllama.cpp: 0 failuresclient + /completion: 1 failures  token 1    expected [105, 2364, 107, 10936]    actual   [2, 105, 2364, 107]

What just happened: the chat path matches the reference token for token. The client path fails at token 1: the golden has 105, <start_of_turn>, and the path has 2, a second <bos>. The case has no expected string, because it was built with record_golden_ids alone, so only the token half runs and no char failure appears. A test that only checked the chat path would pass before and after this bug, which is why the per-path check is the part to keep. golden_paths.py records the golden from the reference in the same run to keep the step short, and for seeing the bug that is enough. For a CI check, store the case with step 2's save_goldens in a file of its own, such as goldens/gemma3.json, and load it, so the check compares against a reviewed file rather than a fresh render.

Step 9: Check golden token IDs through llama-server

Goal: check all six Qwen3 goldens, recorded from Transformers in step 6, through the Qwen3 llama-server.

Why this step: the goldens in goldens/qwen3.json came from the reference. Your service sends requests through llama-server, which parses the request, renders its own copy of the template from the GGUF file, and tokenizes with its own rules. Part 9's compare_engines compared two Jinja engines' strings; this is the same idea at the token level, against the stored goldens rather than a second live render. If the server's half of the render ever disagrees with the reference, for example after a llama.cpp upgrade, this is the check that fails.

Create golden_server.py:

# golden_server.py"""Check the Qwen3 goldens, recorded from Transformers, through llama-server."""from endpoints import QWEN_URLfrom templatestudio.golden import check_goldens, load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import LlamaCppTokensqwen = load_tokenizer("Qwen/Qwen3-0.6B")cases = load_goldens("goldens/qwen3.json")failures = check_goldens(qwen, cases, engine=LlamaCppTokens(QWEN_URL))print(f"{len(cases)} cases through llama.cpp, {len(failures)} failures")

Run it with the Qwen3 server running on port 9932:

python golden_server.py

6 cases through llama.cpp, 0 failures

What just happened: all six Qwen3 cases, including both tool cases and the reasoning turn, reach the model through llama-server b12283 exactly as Transformers builds them. The tool cases pass because golden_cases.py uses Part 10's request_tool, which removes the return key llama-server would otherwise drop. Add a case with the full tool_schema output and this check fails at the token where the two engines part, which Part 10 measured at token 106 for its get_weather request: the Part 10 finding, now locked as a check. Run it against the same llama.cpp build, with the same flags, that production runs. A token golden checked against a different build is a check of a different system.

Step 10: Run golden-token tests in CI with pytest

Goal: turn the goldens into a pytest test that checks the tokenizer folder your service ships.

Why this step: goldens that nobody runs catch nothing. They belong in the pipeline that every template edit, tokenizer upgrade and engine upgrade already passes through. pytest's parametrize gives each golden case its own test, so a failure names the case. The test should check the template you actually deploy. In a real service that is a tokenizer folder in your repository or your model artifact, not the copy on the Hugging Face Hub. So this step saves the Qwen3 tokenizer into qwen3-tokenizer/, the folder golden_cases.py names as SHIPPED, and the test loads it from there. save_pretrained writes the template to its own file, chat_template.jinja, which is the file a template commit edits.

direction: down
classes: {
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
hub: "Qwen/Qwen3-0.6B\nyour Hugging Face cache" {class: act}
shipped: "qwen3-tokenizer/\nchat_template.jinja + tokenizer files" {class: store}
test: "tests/test_goldens.py\none test per case" {class: check}
goldens: "goldens/qwen3.json" {class: store}
hub -> shipped: "golden_ship.py"
shipped -> test: "load_tokenizer(SHIPPED)"
goldens -> test: "load_goldens"

Create golden_ship.py, which saves the tokenizer:

# golden_ship.py"""Save the Qwen3 tokenizer, with its chat template, into the folder CI tests."""from pathlib import Pathfrom golden_cases import SHIPPEDfrom templatestudio.render import load_tokenizerfiles = load_tokenizer("Qwen/Qwen3-0.6B").save_pretrained(SHIPPED)print(f"saved to {SHIPPED}/:", ", ".join(sorted(Path(f).name for f in files)))

python golden_ship.py

saved to qwen3-tokenizer/: chat_template.jinja, tokenizer.json, tokenizer_config.json

Create the test file tests/test_goldens.py:

mkdir tests

# tests/test_goldens.py"""Fail CI when the shipped template or its token IDs move off the goldens."""import pytestfrom golden_cases import SHIPPEDfrom templatestudio.golden import check_goldens, load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensCASES = load_goldens("goldens/qwen3.json")@pytest.fixture(scope="module")def shipped():    return load_tokenizer(SHIPPED)@pytest.mark.parametrize("case", CASES, ids=[case.name for case in CASES])def test_golden(shipped, case):    failures = check_goldens(shipped, [case], engine=TransformersTokens(shipped))    assert not failures, "; ".join(f"{f.unit} {f.position}" for f in failures)

ids=[case.name for case in CASES] names each test after its case. Without it, pytest builds IDs only from simple values such as strings and numbers, so a GoldenCase parameter becomes case0, case1, and so on. The tokenizer fixture uses scope="module", so the tokenizer loads once for all six tests. The assertion message lists each failure's unit and position, so a failing test says where the output moved.

Run it:

python -m pytest -q tests

......                                                                   [100%]6 passed in 3.75s

Your timing will differ, and pytest pads the first line to your terminal's width. Run pytest as python -m pytest, from the project root: that puts the project folder on Python's import path, so templatestudio and golden_cases import. The When it breaks section shows what a bare pytest prints.

What just happened: you have one test per golden, run against the tokenizer folder you ship. python -m pytest -q tests, run from the project root, is the command your CI job runs; this part adds no workflow file, so wire that one command into whatever CI you use. Commit goldens/qwen3.json and qwen3-tokenizer/ next to the code that serves the model. The test checks the reference engine only, so it needs no server; step 9's script is the server check, to run against your serving build.

Step 11: Make a chat template regression test fail on a template commit

Goal: apply the step 4 whitespace edit to the shipped template file, as a commit would, and watch the tests fail.

Why this step: a test that has only ever passed has not shown it can fail. This step applies the same three hunks as step 4, this time to qwen3-tokenizer/chat_template.jinja on disk, which is what a real template commit does. No git commit is involved: the script edits the file, and step 12 restores it. Nothing else changes: same goldens, same test, same Python. If the tests still passed, they would not be testing the file you ship.

direction: down
classes: {
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
commit: "golden_commit_file.py\nthe three step 4 hunks" {class: act}
file: "qwen3-tokenizer/chat_template.jinja" {class: store}
tests: "python -m pytest\n6 failed" {class: bad}
commit -> file: "edit on disk"
file -> tests: "load_tokenizer(SHIPPED)"

Create golden_commit_file.py. It reuses whitespace_commit.py from step 4:

# golden_commit_file.py"""Commit the whitespace tidy-up to the shipped template file on disk."""from pathlib import Pathfrom golden_cases import SHIPPEDfrom whitespace_commit import HUNKS, apply_hunkspath = Path(SHIPPED) / "chat_template.jinja"edited = apply_hunks(path.read_text(encoding="utf-8"), HUNKS)path.write_text(edited, encoding="utf-8", newline="\n")print(f"applied {len(HUNKS)} hunks to {path.as_posix()}")

Run it, then run the tests. --tb=no -rN keeps pytest's report to its pass/fail line; the detail comes from the next script:

python golden_commit_file.pypython -m pytest -q --tb=no -rN tests

applied 3 hunks to qwen3-tokenizer/chat_template.jinja

FFFFFF                                                                   [100%]6 failed in 3.73s

What just happened: the edit reached the file the tests load, and all six tests failed. Without --tb=no, each failing test's message names where its output moved; for plain it reads AssertionError: token 9; char 46. The next step reads those failures in full and decides what to do with them.

Step 12: Read a failing golden's diff, and re-record goldens only on purpose

Goal: print every failure, decide the change is a regression, and revert it.

Why this step: a failing golden means one of two things: an accidental regression, or an intended change nobody has approved yet. Re-recording without reading the diff turns the first into the second, and that is the one way a golden test can be defeated. Snapshot-testing plugins such as syrupy re-record with a single flag. After the first recording in steps 2 and 6, golden_update.py is the only script that should rewrite the goldens. It prints the failures first and writes only when you pass --accept.

direction: down
classes: {
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
red: "6 failed" {class: bad}
read: "read the diff\npython golden_update.py" {class: check}
fix: "regression: revert or fix" {class: bad}
accept: "intended: --accept,\ncommit the new goldens" {class: store}
red -> read
read -> fix: "not wanted"
read -> accept: "wanted, reviewed"

Create golden_update.py. It loads the shipped tokenizer, prints every failure, and refuses to write without --accept:

# golden_update.py"""Re-record the goldens from the shipped tokenizer, after showing the diff.Run with --accept once you have read the diff and want the new output."""import sysfrom golden_cases import SHIPPEDfrom templatestudio.golden import (    check_goldens,    load_goldens,    record_golden_ids,    record_goldens,    save_goldens,)from templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokenssys.stdout.reconfigure(encoding="utf-8")qwen = load_tokenizer(SHIPPED)engine = TransformersTokens(qwen)cases = load_goldens("goldens/qwen3.json")failures = check_goldens(qwen, cases, engine=engine)for failure in failures:    print(f"{failure.name}: {failure.unit} {failure.position}")    print(f"  expected {failure.expected!r}")    print(f"  actual   {failure.actual!r}")if not failures:    print("nothing changed; goldens left as they are")elif "--accept" not in sys.argv:    print(f"{len(failures)} changes; rerun with --accept to record them")else:    recorded = record_golden_ids(engine, record_goldens(qwen, cases))    save_goldens("goldens/qwen3.json", recorded)    print(f"recorded {len(cases)} cases")

python golden_update.py

plain: token 9  expected '[198, 151644, 77091, 198]'  actual   '[271, 151644, 77091, 198]'plain: char 46  expected ' is my order?<|im_end|>\n<|im_start|>assistant\n'  actual   ' is my order?<|im_end|>\n\n<|im_start|>assistant\n'system: token 20  expected '[198, 151644, 77091, 198]'  actual   '[271, 151644, 77091, 198]'system: char 100  expected ' is my order?<|im_end|>\n<|im_start|>assistant\n'  actual   ' is my order?<|im_end|>\n\n<|im_start|>assistant\n'multi_turn: token 20  expected '[198, 151644, 77091, 198]'  actual   '[271, 151644, 77091, 198]'multi_turn: char 100  expected ' is my order?<|im_end|>\n<|im_start|>assistant\nLe'  actual   ' is my order?<|im_end|>\n\n<|im_start|>assistant\nL'tool_round_trip: token 159  expected '[198, 151644, 77091, 198]'  actual   '[271, 151644, 77091, 198]'tool_round_trip: char 703  expected 'her in Paris?<|im_end|>\n<|im_start|>assistant\n<t'  actual   'her in Paris?<|im_end|>\n\n<|im_start|>assistant\n<'reasoned_tool_call: token 159  expected '[198, 151644, 77091, 198]'  actual   '[271, 151644, 77091, 198]'reasoned_tool_call: char 703  expected 'her in Paris?<|im_end|>\n<|im_start|>assistant\n<t'  actual   'her in Paris?<|im_end|>\n\n<|im_start|>assistant\n<'no_think: token 9  expected '[198, 151644, 77091, 198]'  actual   '[271, 151644, 77091, 198]'no_think: char 46  expected ' is my order?<|im_end|>\n<|im_start|>assistant\n<t'  actual   ' is my order?<|im_end|>\n\n<|im_start|>assistant\n<'12 changes; rerun with --accept to record them

Each case fails twice, once for each half that check_goldens checks. The halves share kwargs, so both failures describe the same request: one change, reported in tokens and in characters. The token half shows something the string half cannot. At token 9 of plain, ID 198, one newline, became 271, two newlines in one token. The prompt has the same number of tokens and one of them is different, so a check that compared token counts would have passed.

This change is a regression, so do not pass --accept. Revert it by saving the tokenizer again, which rewrites chat_template.jinja from your cache, and run the tests:

python golden_ship.pypython -m pytest -q tests

saved to qwen3-tokenizer/: chat_template.jinja, tokenizer.json, tokenizer_config.json

......                                                                   [100%]6 passed in 3.88s

What just happened: you saw the whole loop a template change goes through: the commit, a red CI run naming every case, a diff you can read, and a revert. Had the change been intended, python golden_update.py --accept would re-record all six cases from the edited template, print recorded 6 cases, and rewrite goldens/qwen3.json. Its diff would show a \n added after every user turn in the six expected strings, two of them in multi_turn, and the matching 198 IDs turning into 271. That diff goes into the same pull request as the template change. If a pull request touches both a template and its goldens, read the goldens' diff first.

Step 13 (optional): Test a fine-tune's training row and its stop tokens

Goal: use Part 8's saved training row as a token golden, then find a failure that every golden passes.

Why this step: Part 8 saved support-model/trained_row.json, the token IDs the trainer actually trained on. Because they come from training itself, they are the strongest golden you can have for a fine-tune: a served prompt that matches them is the format the model learned. The last incident from chapter 11 is the one no golden catches, a fine-tune whose answers run on into invented turns. The prompt is correct and the IDs are correct. What is wrong is the stop list. The model writes its end-of-turn token, <|end|>, the marker Part 8 added for this fine-tune, and generation carries on, because that ID is missing from eos_token_id in the generation_config.json the engine loaded. A golden checks what goes into the model, and the stop list is not part of the input. You need this step only if you have Part 8's support-model/ folder.

direction: down
grid-rows: 1
grid-gap: 40
classes: {
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
input: "What goes in: the prompt" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  c: "check_goldens\ntrained_row.json IDs" {class: check}
  r: "0 failures" {class: good}
}
output: "When it stops: the stop list" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  c: "check_stop_alignment\n(Part 8)" {class: check}
  r: "<|end|> 151672 not in\neos_token_id" {class: bad}
}

Create golden_trained.py. The case uses kwargs={"add_generation_prompt": False}, because a training row is the whole conversation with no prompt for a next turn, and one kwargs dict carries that to both halves. The second check compares the shipped stop list with the base model's, which is what a fine-tune ships with when someone forgets to update generation_config.json:

# golden_trained.py"""Part 8's saved model: its trained row as a golden, and its stop set."""import copyimport jsonfrom pathlib import Pathfrom transformers import GenerationConfigfrom support_data import SUPPORT_CONVERSATION, SUPPORT_TEXTfrom templatestudio.authoring import check_stop_alignmentfrom templatestudio.golden import GoldenCase, check_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokenstokenizer = load_tokenizer("support-model")row = json.loads(Path("support-model/trained_row.json").read_text(encoding="utf-8"))case = GoldenCase(    "trained_row",    SUPPORT_CONVERSATION,    kwargs={"add_generation_prompt": False},    expected=SUPPORT_TEXT,    expected_ids=tuple(row["input_ids"]),    engine="trl",)failures = check_goldens(tokenizer, [case], engine=TransformersTokens(tokenizer))print(f"trained_row golden: {len(failures)} failures")shipped = GenerationConfig.from_pretrained("support-model")base_stops = copy.deepcopy(shipped)base_stops.eos_token_id = [151645, 151643]  # the base model's stop listfor label, config in (("shipped", shipped), ("base stops", base_stops)):    print(f"{label}: {check_stop_alignment(tokenizer, config)}")

Run it:

python golden_trained.py

trained_row golden: 0 failuresshipped: []base stops: ["'<|end|>' (id 151672) is not in eos_token_id [151645, 151643]"]

What just happened: the served template reproduces the trained row exactly, token for token, so the golden passes. With the base model's stop list, the same model would never stop on <|end|>, and only check_stop_alignment says so. This incident therefore ends in a check, not a golden: run check_stop_alignment in CI next to the goldens. The same failure happened in public: when Llama 3 was released, its generation config did not list <|eot_id|>, and users reported that the model kept writing new turns for the user.

When it breaks: errors from golden-token tests

The full AssertionError line, when the first hunk is no longer in the template, is AssertionError: hunk not found once: ' {%- set index = (messages|length - 1) - loop.index0 %}'.

Fix: llama.cpp double BOS, "Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token"

llama-server logs llama_vocab::impl::tokenize: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want? when a client posts a string that already starts with <bos>. Step 8 catches it: send messages to the chat endpoint instead.

Fix: enable_thinking not working, a misspelled template switch is silently ignored

Transformers and the template never warn about a switch name the template does not read, so enable_thinkng=False renders exactly like no switch. Step 7 shows the token golden failing at position 13 when the four empty-reasoning IDs go missing.

How the golden-token test fits together: cases, goldens file, halves and paths

Here is what you built, with the names from your code. The cases and the goldens file are data. golden.py has two halves that read the same case. The engines in the bottom row are the paths a request takes to the model, all checked against IDs recorded from the reference:

direction: down
classes: {
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  store: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  code: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  eng: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  ci: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}
cases: "golden_cases.py\nCASES" {class: data}
file: "goldens/qwen3.json" {class: store}
golden: "templatestudio/golden.py" {
  grid-rows: 1
  grid-gap: 24
  label.near: top-center
  rec: "record_goldens\nrecord_golden_ids" {class: code}
  chk: "check_goldens\nstring_failure, token_failure" {class: code}
  io: "save_goldens\nload_goldens" {class: code}
}
engines: "TokenRenderer adapters (Part 10)" {
  grid-rows: 1
  grid-gap: 24
  label.near: top-center
  tf: "TransformersTokens\nreference" {class: eng}
  cpp: "LlamaCppTokens\nchat path" {class: eng}
  client: "ClientCompletionTokens\n/completion path" {class: eng}
}
ci: "tests/test_goldens.py\ngolden_update.py --accept" {class: ci}
shipped: "qwen3-tokenizer/\nthe template you ship" {class: store}
ci -> shipped: "load_tokenizer(SHIPPED)"
cases -> golden: "conversations + kwargs"
golden -> file: "write"
file -> golden: "read"
golden -> engines: "engine.ids(conversation, **kwargs)"
ci -> golden: "runs check_goldens"

The arrow from the goldens file back into golden.py is the important one. The check reads goldens; it never writes them. After the first recording, the only writer is golden_update.py --accept, which a person runs after reading the failures, and its output is reviewed like any other change. The tests load qwen3-tokenizer/, the template you ship, so a commit to that folder is exactly what they check.

Complete code for the golden-token test: golden.py, cases, scripts and test

The full module, with every step's code merged:

# templatestudio/golden.py"""Golden-token tests: record what a template renders and what the enginesends for fixed conversations, then fail on any change to either."""from __future__ import annotationsimport jsonfrom dataclasses import asdict, dataclass, field, replacefrom pathlib import Pathfrom typing import Anyfrom transformers import PreTrainedTokenizerBasefrom templatestudio.messages import Conversationfrom templatestudio.render import first_divergence, renderfrom templatestudio.serving import TokenRendererCHAR_WINDOW = 24  # characters shown either side of a string failureTOKEN_WINDOW = 4  # token IDs shown, starting at the first one that differs@dataclass(frozen=True)class GoldenCase:    """One fixed conversation and the exact output it must keep producing.    kwargs reach both the string render and the engine, so a case with    {"enable_thinking": False} is checked with that switch on both sides.    expected is the rendered string; expected_ids are the token IDs one    engine sent for it. Either stays None until it has been recorded.    """    name: str    conversation: Conversation    kwargs: dict[str, Any] = field(default_factory=dict)    expected: str | None = None    expected_ids: tuple[int, ...] | None = None    engine: str = ""  # the engine that recorded expected_ids@dataclass(frozen=True)class GoldenFailure:    name: str    unit: str  # "char" for the string half, "token" for the ID half    position: int  # first index where the output left the golden    expected: str  # the golden near that position: chars around it, IDs from it    actual: str  # the same window of what was produced insteaddef render_case(tokenizer: PreTrainedTokenizerBase, case: GoldenCase) -> str:    """The string half: the case rendered by the tokenizer's template."""    return render(tokenizer, case.conversation, **case.kwargs)def record_goldens(    tokenizer: PreTrainedTokenizerBase, cases: list[GoldenCase]) -> list[GoldenCase]:    """Fill in expected for every case from the current template. Run it on    a template you have checked by eye, never to make a failing check pass."""    return [replace(case, expected=render_case(tokenizer, case)) for case in cases]def save_goldens(path: str | Path, cases: list[GoldenCase]) -> None:    """Write cases as indented JSON, so a changed golden reviews as a diff."""    data = [asdict(case) for case in cases]    text = json.dumps(data, indent=2, ensure_ascii=False)    # newline="\n" writes the same bytes on Windows, Linux and macOS.    Path(path).write_text(text + "\n", encoding="utf-8", newline="\n")def load_goldens(path: str | Path) -> list[GoldenCase]:    """Read cases back. JSON has no tuples, so expected_ids comes back as a    list; turning it into a tuple again makes a loaded case equal the saved one."""    data = json.loads(Path(path).read_text(encoding="utf-8"))    cases = []    for item in data:        if item["expected_ids"] is not None:            item["expected_ids"] = tuple(item["expected_ids"])        cases.append(GoldenCase(**item))    return casesdef string_failure(case: GoldenCase, actual: str) -> GoldenFailure | None:    """Compare one render with its golden string; None if they are equal."""    assert case.expected is not None    position = first_divergence(case.expected, actual)    if position is None:        return None    start = max(position - CHAR_WINDOW, 0)    end = position + CHAR_WINDOW    return GoldenFailure(        name=case.name,        unit="char",        position=position,        expected=case.expected[start:end],        actual=actual[start:end],    )def check_goldens(    tokenizer: PreTrainedTokenizerBase,    cases: list[GoldenCase],    *,    engine: TokenRenderer | None = None,) -> list[GoldenFailure]:    """Re-render every case that has a golden string and, given an engine,    re-send every case that has golden IDs. Empty means pass."""    failures = []    for case in cases:        if engine is not None and case.expected_ids is not None:            failure = token_failure(case, engine.ids(case.conversation, **case.kwargs))            if failure is not None:                failures.append(failure)        if case.expected is not None:            failure = string_failure(case, render_case(tokenizer, case))            if failure is not None:                failures.append(failure)    return failuresdef record_golden_ids(engine: TokenRenderer, cases: list[GoldenCase]) -> list[GoldenCase]:    """Fill in expected_ids for every case from what the engine sends."""    return [        replace(            case,            expected_ids=tuple(engine.ids(case.conversation, **case.kwargs)),            engine=engine.name,        )        for case in cases    ]def token_failure(case: GoldenCase, actual: list[int]) -> GoldenFailure | None:    """Compare one engine's IDs with the golden IDs; None if they are equal."""    assert case.expected_ids is not None    expected = list(case.expected_ids)    position = first_divergence(expected, actual)    if position is None:        return None    end = position + TOKEN_WINDOW    return GoldenFailure(        name=case.name,        unit="token",        position=position,        expected=str(expected[position:end]),        actual=str(actual[position:end]),    )

The project files this part adds or uses, with earlier parts' files that this part imports:

chat-templates-part1/├── templatestudio/│   ├── __init__.py          Part 1│   ├── render.py            Part 1 (last changed in Part 6)│   ├── messages.py          Part 2 (last changed in Part 7)│   ├── jinja_env.py         Part 3 (step 13 only)│   ├── tools.py             Part 5│   ├── authoring.py         Part 8 (step 13 only)│   ├── serving.py           Part 10│   └── golden.py            this part├── goldens/│   └── qwen3.json           steps 2 and 6├── tests/│   └── test_goldens.py      step 10├── endpoints.py             Part 10├── session.py               Part 10├── serve_check.py           Part 10├── qwen3-tokenizer/         step 10 (the tokenizer CI tests)├── support_data.py          Part 8 (step 13 only)├── support-model/           Part 8 (step 13 only)├── golden_types.py          step 1├── golden_cases.py          step 2├── golden_record.py         step 2├── golden_check.py          steps 3 and 6├── whitespace_commit.py     step 4├── golden_commit.py         step 4├── golden_bisect.py         step 5├── golden_record_ids.py     step 6├── golden_typo.py           step 7├── golden_paths.py          step 8├── golden_server.py         step 9├── golden_ship.py           steps 10 and 12├── golden_commit_file.py    step 11├── golden_update.py         step 12└── golden_trained.py        step 13

The cases, the commit used in steps 4, 5 and 11, and the test are the other three files you reuse:

# golden_cases.py"""Part 11: the conversation shapes the Qwen3 goldens cover."""from session import get_weatherfrom templatestudio.golden import GoldenCasefrom templatestudio.serving import request_toolfrom templatestudio.tools import tool_call_message, tool_result_message, tool_schemaORDER = [{"role": "user", "content": "Where is my order?"}]SUPPORT = [    {"role": "system", "content": "You are a support agent."},    *ORDER,    {"role": "assistant", "content": "Let me check."},    {"role": "user", "content": "It is order 1042."},]WEATHER = [{"role": "user", "content": "What is the weather in Paris?"}]CALL = tool_call_message("get_weather", {"city": "Paris"})REASONED_CALL = {**CALL, "reasoning_content": "The user wants the weather."}TOOLS = [request_tool(tool_schema(get_weather))]# The tokenizer folder your service loads; step 10 saves it, CI tests it.SHIPPED = "qwen3-tokenizer"CASES = [    GoldenCase("plain", ORDER),    GoldenCase("system", SUPPORT[:2]),    GoldenCase("multi_turn", SUPPORT),    GoldenCase(        "tool_round_trip",        [*WEATHER, CALL, tool_result_message("get_weather", "sunny")],        kwargs={"tools": TOOLS},    ),    GoldenCase(        "reasoned_tool_call",        [*WEATHER, REASONED_CALL, tool_result_message("get_weather", "sunny")],        kwargs={"tools": TOOLS},    ),    GoldenCase("no_think", ORDER, kwargs={"enable_thinking": False}),]

# whitespace_commit.py"""Part 11: a tidy-up of Qwen3's template that reads as whitespace only.Each hunk replaces one line of the template with a re-indented copy."""HUNKS = [    (        '    {%- set index = (messages|length - 1) - loop.index0 %}',        '  {%- set index = (messages|length - 1) - loop.index0 %}',    ),    (        '        {%- set content = message.content %}',        '    {%- set content = message.content %}',    ),    (        '    {%- elif message.role == "assistant" %}',        '    {% elif message.role == "assistant" %}',    ),]def apply_hunks(template: str, hunks: list[tuple[str, str]]) -> str:    """Apply each (old line, new line) hunk; every old line must be there."""    for old, new in hunks:        assert template.count(old) == 1, f"hunk not found once: {old!r}"        template = template.replace(old, new)    return template

# tests/test_goldens.py"""Fail CI when the shipped template or its token IDs move off the goldens."""import pytestfrom golden_cases import SHIPPEDfrom templatestudio.golden import check_goldens, load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensCASES = load_goldens("goldens/qwen3.json")@pytest.fixture(scope="module")def shipped():    return load_tokenizer(SHIPPED)@pytest.mark.parametrize("case", CASES, ids=[case.name for case in CASES])def test_golden(shipped, case):    failures = check_goldens(shipped, [case], engine=TransformersTokens(shipped))    assert not failures, "; ".join(f"{f.unit} {f.position}" for f in failures)

The other scripts appear in full in the steps that create them. golden_check.py is shown twice: the step 6 version replaces the step 3 version.

Where to go next: minimize failing conversations, add goldens per model, guard goldens edits

• Shrink a failing conversation automatically. Step 5 bisected the template. The other half of bisecting is the input: write minimize(tokenizer, case) that drops one message at a time and keeps the drop whenever the golden still fails, so a long agent transcript shrinks to the two or three turns that trigger the change. This is the book's second exercise.
• Keep one goldens file per served model. Step 2's six shapes are written for Qwen3. Record the same shapes for every model you serve, goldens/gemma3.json and so on, plus a multimodal case if you serve a vision model. For a vision model, the token golden should hold the IDs after the processor expands the image placeholder, which Part 7 showed is where the image cost lives. Part 7 left these expanded-ID goldens to this part. They are not built here because they need PyTorch and a processor, which this part otherwise avoids; they are the next case to add.
• Flag commits that change a template and its goldens together. A pull request that edits both is the one case where the golden test can be defeated quietly. Add a CI check that fails such a change unless the commit message carries an explicit acknowledgment. Chapter 11 of The Chat Templates Handbook sets this as its third exercise, and also carries the full failure catalog that maps every failure in the book to the check that owns it. Its companion repository has the chapter's version of golden.py at the v1.2/ch11-debugging tag.

For the same record-and-compare discipline applied to an agent's behaviour, see contract tests for a LangGraph agent with pytest.

Part 12 moves from accidental changes to deliberate ones: a template written by someone hostile, and what the Jinja sandbox does and does not stop.

References

• Transformers 5.22.1, apply_chat_template source: extra keyword arguments are passed to the template with no check of their names.
• Transformers releases, including 5.22.2 and PR #50431, "Harden our Jinja sandboxing".
• Transformers: writing a chat template.
• pytest: calling pytest through python -m pytest, which adds the current folder to the import path.
• pytest: managing traceback output, for --tb=no and -r in step 11.
• pytest 9.2.0 release notes and the pytest API reference for parametrize and fixture.
• syrupy, a pytest snapshot plugin that re-records snapshots with --snapshot-update.
• Jinja whitespace control: trim_blocks, lstrip_blocks and the - in {%-, the mechanism behind step 4.
• Python 3.12 json module: tuples are encoded as JSON arrays and decoded as lists.
• Python 3.12 dataclasses module: replace and asdict.
• llama.cpp b12283 release and its server README: /tokenize, /apply-template, /completion and --chat-template-kwargs.
• llama.cpp tests/test-chat-template.cpp at b12283: llama.cpp's own expected-output tests for chat templates.
• Qwen3-0.6B tokenizer_config.json, which holds the chat template the step 4 hunks edit.
• GGUF models served in steps 8 and 9: ggml-org/gemma-3-270m-it-GGUF and Qwen/Qwen3-0.6B-GGUF.
• tokenizers 0.23.3 and huggingface-hub 2.4.0, the other pinned packages.
• TRL, the trainer that produced Part 8's trained_row.json used in step 13.
• Transformers GenerationConfig, whose eos_token_id is the stop list step 13 checks.
• unsloth/gemma-3-270m-it, the ungated tokenizer used as step 8's reference.
• Qwen3-0.6B generation_config.json and model card.
• Meta-Llama-3-8B-Instruct discussion #4: the model did not stop on <|eot_id|> with the default generation config.
• vLLM forum: thinking token limit setting: a thinking key that the template never read.
• Phi-3-mini-4k-instruct discussion #54: a template whitespace change and a report of worse instruction following.
• The Chat Templates Handbook, chapter 11, "Debugging Broken Templates": the failure catalog, the golden-token test and the four incidents.
• Template Studio, tag v1.2/ch11-debugging: the book's companion code.

Related Articles

Llms

• How to Test a Chat Template Across Jinja Engines
• How to Check the Token IDs llama-server Sends the Model
• How to Gate Chat Template Changes in CI with One Audit
• How to Write a Chat Template for a Fine-Tuned Model and Test It