How to Check the Token IDs llama-server Sends the Model

What you'll build: a token-ID check for the server you deploy

Your template test passes in a notebook, and the server still sends the model something else. This intermediate tutorial is for engineers who serve a model in one place and test its template in another. It builds a check that asks a running inference server for the token IDs it builds from a conversation, then compares them with the IDs Transformers builds, one token at a time. Here is its report on a client that renders Gemma 3's prompt itself and posts the string to llama-server:

client's string starts: '<bos><start_of_turn>'server's prompt starts: '<start_of_turn>user\n'transformers: [2, 105, 2364]llama.cpp: [2, 105, 2364]client + /completion: [2, 2, 105]client + /completion leaves transformers at token 1  lengths: 22 vs 23

The client's prompt starts with two <bos> tokens, ID 2, where the model was trained on one. Nothing raised. The model answers anyway, from a prompt shape it was never trained on and that Unsloth's Gemma 3 guide warns against. No test that compares rendered strings can see it, because the second <bos> is added after the string is finished.

By the end you will have two new modules. templatestudio/serving.py gets token IDs from three places (Transformers, llama-server's own template and tokenizer, and a client that renders the prompt itself) and reports the first token where any of them disagree. templatestudio/prefix.py checks that every request in a conversation starts with the request before it, token for token, which is what an engine's prefix cache needs to reuse its work. You will run both against two real llama-servers, Gemma 3 270M and Qwen3-0.6B, on your own machine. The adapter in this part is for llama.cpp; a vLLM one is a single class, described in Where to go next.

Verified against Python 3.12.2, transformers==5.22.1, tokenizers==0.23.3, huggingface-hub==2.4.0, jinja2==3.1.6, and llama.cpp b12283 (llama-server) on 2026-11-11. Two things on this page were done by hand rather than by a script: starting the two servers, exactly as printed below, which serve_check.py then confirms; and copying the server log line in step 7 from the server's terminal, where your timestamp will differ.

The check finds three problems in this tutorial, and none of them is a bug in a chat template:

• A tool list the server reads differently. Transformers writes a tool's return type into the prompt; llama-server drops it. Step 4 finds the token where the two prompts part, and step 5 fixes the request.
• A doubled <bos>. A client that renders the prompt itself and posts the string to /completion leaves the tokenizing to a server that has no way to tell the template has run. Steps 6 and 7.
• A prefix that moves. A client that reorders its tool list between two requests makes the second request start differently from the first, and the engine recomputes everything after the first token that moved. Steps 8 and 9.

This is Part 10 of Chat Templates in Practice and continues the project from Part 9, How to Test a Chat Template Across Jinja Engines. Budget about 40 minutes, most of it downloading, and about 1 GB of disk for llama.cpp and two small models. Everything runs on a CPU: no GPU, no API key, no account and no Hugging Face token.

Prerequisites: download llama.cpp b12283 and start two servers

You need the project from the earlier parts: the chat-templates-part1/ folder with its virtual environment, and these files from it:

• templatestudio/__init__.py (Part 1, empty)
• templatestudio/render.py (Part 1, last changed in Part 6)
• templatestudio/messages.py (Part 2, last changed in Part 7)
• templatestudio/tools.py (Part 5)

If one is missing, copy it from the complete code of the part that last changed it: Part 5, Part 6 or Part 7. This part adds two modules, templatestudio/serving.py and templatestudio/prefix.py, two data files, endpoints.py and session.py, and ten scripts. It changes no earlier file.

Go to the project folder, activate the environment, and turn off the Transformers advisory warnings. Run the cd from the folder that contains chat-templates-part1/:

cd chat-templates-part1source .venv/bin/activateexport TRANSFORMERS_NO_ADVISORY_WARNINGS=1

On Windows, activate with .venv\Scripts\activate in PowerShell or source .venv/Scripts/activate in Git Bash, and set the variable in PowerShell with $env:TRANSFORMERS_NO_ADVISORY_WARNINGS = "1". Every script in this part runs from the project root.

The Python pins from Parts 1 to 9 are unchanged: Python 3.12 (verified on 3.12.2), transformers 5.22.1, tokenizers 0.23.3, huggingface-hub 2.4.0 and jinja2 3.1.6. Nothing in this part needs PyTorch. No new Python package is installed either, since the HTTP calls use the standard library's urllib. Transformers 5.22.2 (2026-11-06) hardens the Jinja sandbox against inputs this tutorial never sends. The pins stay at 5.22.1 so every part's output stays comparable, and the series moves to 5.22.2 in one step at the end.

This part adds one program, llama.cpp's llama-server, pinned to build b12283 (2026-11-11). llama.cpp publishes a numbered build for almost every commit, and /props reports the build a server runs, so the pin is checkable.

• Download: open the b12283 release page and take the archive for your machine: llama-b12283-bin-win-cpu-x64.zip on Windows, llama-b12283-bin-macos-arm64.tar.gz on an Apple Silicon Mac, llama-b12283-bin-ubuntu-x64.tar.gz on Linux. Those three are 23.9 MB, 16.1 MB and 22.2 MB on the release page.
• Unpack: extract it into a folder of its own, outside the project (on macOS and Linux, tar -xzf the archive into that folder). The archive holds llama-server (llama-server.exe on Windows) and the libraries it loads. The Windows archive unpacks flat, with llama-server.exe at the top level. If your archive has a build/bin/ folder instead, run the commands below from there. If macOS refuses to open the downloaded program, clear its quarantine flag on the folder, for example xattr -dr com.apple.quarantine ~/llama-b12283; on Linux, run chmod +x llama-server if the file lost its execute bit. The same release also has archives for Intel Macs (macos-x64), Linux on ARM (ubuntu-arm64) and Windows on ARM (win-cpu-arm64).
• Pick a newer build only on purpose: a later build can change the server's half of the render, which is the thing this part measures. The token positions in steps 4 and 9 are the numbers most likely to move, and step 5 shows how to find what changed.

Keep the terminal where you activated the environment for every python command in this part, and open two new terminals for the servers. In each, cd into the folder you extracted, for example cd ~/llama-b12283 (on Windows, cd $HOME\llama-b12283). A GGUF file is llama.cpp's single-file model format: the weights plus metadata, including a copy of the chat template and the tokenizer's settings, such as whether to add <bos>. The first time it runs, -hf downloads one from Hugging Face; b12283 stores it in your Hugging Face cache. That is about 292 MB for Gemma 3 270M and 639 MB for Qwen3-0.6B. Both repositories are ungated:

./llama-server -hf ggml-org/gemma-3-270m-it-GGUF:Q8_0./llama-server -hf Qwen/Qwen3-0.6B-GGUF:Q8_0 --port 9932

On Windows, run .\llama-server.exe with the same arguments; Windows may ask whether to allow it through the firewall, and the server only needs to listen on your own machine. The :Q8_0 suffix picks the 8-bit quantized file in each repository; the token IDs do not depend on which quantization you pick. The first server takes llama-server's default port, 9931. The second needs --port, because two servers cannot share a port. Wait until each terminal prints server is listening, and leave both running for the rest of the tutorial. The Transformers side needs one more tokenizer, Gemma 3 270M's, about 33 MB, which downloads the first time step 6 runs if it is not in your cache. The Qwen3 tokenizer is already there from Part 1.

Many guides still use 8080, which is where older llama.cpp builds listened. The default moved to 9931 in v0.7.0 (2026-10-15), and b12283 is a later build, so it listens on 9931. The official Docker image still listens on 8080 inside the container, because its Dockerfile sets LLAMA_ARG_PORT=8080. If you run the image, publish it to 9931 with -p 9931:8080, or edit endpoints.py below.

Create endpoints.py. Every script in this part reads its addresses and tokenizer names from it:

# endpoints.py"""Part 10: the two llama-servers, and the Transformers tokenizer thatholds the same model's chat template for each one."""GEMMA_URL = "http://localhost:9931"  # llama-server's default portQWEN_URL = "http://localhost:9932"GEMMA_TOKENIZER = "unsloth/gemma-3-270m-it"QWEN_TOKENIZER = "Qwen/Qwen3-0.6B"

Create session.py with the conversations and the two tools every engine gets. Steps 4, 8 and 9 use the later names; they are defined here so every script imports its data from one file:

# session.py"""Part 10: the conversations and tools sent to every engine."""QUESTION = [    {"role": "system", "content": "You are terse."},    {"role": "user", "content": "What is 2 + 2?"},]# Three requests: after each user turn, the client sends everything so far.SESSION = [    *QUESTION,    {"role": "assistant", "content": "4."},    {"role": "user", "content": "And times 3?"},    {"role": "assistant", "content": "12."},    {"role": "user", "content": "Minus 5?"},]def get_weather(city: str) -> str:    """Get the current weather for a city.    Args:        city: The city name.    """    return "sunny"def get_time(city: str) -> str:    """Get the local time in a city.    Args:        city: The city name.    """    return "noon"

Then create serve_check.py. It asks each server for its health and for /props, which reports the build, the model, and the chat template llama-server read from the GGUF file:

# serve_check.py"""Ask each llama-server which build it runs and which template it loaded."""import jsonimport urllib.requestfrom endpoints import GEMMA_URL, QWEN_URLdef get(url: str) -> dict:    with urllib.request.urlopen(url, timeout=5) as response:        return json.load(response)for url in (GEMMA_URL, QWEN_URL):    status = get(f"{url}/health")["status"]    props = get(f"{url}/props")    print(f"{url}: {status}, build {props['build_info']}")    print(f"  model: {props['model_alias']}")    print(f"  template starts: {props['chat_template'].splitlines()[0]}")

Check both servers before you go on:

python serve_check.py

Expected output:

http://localhost:9931: ok, build b12283-f3b407ca1  model: ggml-org/gemma-3-270m-it-GGUF:Q8_0  template starts: {{ bos_token }}http://localhost:9932: ok, build b12283-f3b407ca1  model: Qwen/Qwen3-0.6B-GGUF:Q8_0  template starts: {%- if tools %}

If either line is not b12283-f3b407ca1, you are running another build, and the numbers below can differ. If the script ends with urllib.error.URLError and a connection refused message, that server is not running on that port. When it breaks has a row for each.

The two template lines are already a clue. Gemma 3's template starts by writing bos_token itself, which matters in step 6. Qwen3's starts with a branch on tools, which matters in step 4.

How an inference server turns messages into token IDs: the serving boundary

When you call apply_chat_template in a notebook, one program renders the template into a string and tokenizes that string, and you can test either job. An inference server does both with its own code: its own template engine, its own copy of the template, its own tokenizer, and its own rules for which special tokens to add. The Chat Templates Handbook, the book this series accompanies, calls the line between your code and that pipeline the serving boundary. Everything you tested in Parts 1 to 9 happens on your side of it. What the model receives is decided on the server's side.

The diagram shows the path a chat request takes through llama-server, and where each of this tutorial's checks reads it:

direction: down
classes: {
  client: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  stage: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  probe: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  model: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
client: "Your client\nmessages + tools + kwargs" {class: client}
server: "llama-server" {
  label.near: top-left
  parse: "1. Parse the request\n(tools reduced to its own struct)" {class: stage}
  template: "2. Render the GGUF's template\n(its own Jinja engine)" {class: stage}
  tokenize: "3. Tokenize\n(add_special, parse_special)" {class: stage}
  apply: "POST /apply-template\nreturns stages 1-2" {class: probe}
  tok: "POST /tokenize\nruns stage 3" {class: probe}
  parse -> template -> tokenize
  template -> apply {style.stroke-dash: 3}
  tokenize -> tok {style.stroke-dash: 3}
}
model: "Model\nreceives token IDs" {class: model}
client -> server.parse: "the serving boundary"
server.tokenize -> model

A template test checks stage 2 with someone else's engine. It cannot see stage 1, where the server turns your JSON into its own structures and can drop a field. Nor can it see stage 3, where the server decides whether to add a <bos> and whether <|im_start|> in the text means the special token or twelve characters. Only a comparison of token IDs covers all three. llama-server exposes the two endpoints you need to get them without generating a single token: /apply-template runs stages 1 and 2 and returns the prompt string, and /tokenize runs stage 3 on any string you send.

Part 9 measured the parity gap between two Jinja engines with compare_engines. compare_token_renders, in step 4, makes the same comparison one stage later, on token IDs instead of strings, against the engine you deploy.

Token IDs only compare when both sides use the same vocabulary. Every comparison in this tutorial pairs a GGUF with the Transformers tokenizer of the same model: Qwen3-0.6B with Qwen/Qwen3-0.6B, and Gemma 3 270M with unsloth/gemma-3-270m-it, an ungated copy of Google's tokenizer files.

Step 1: Get token IDs from apply_chat_template as the reference

Goal: wrap apply_chat_template in a class whose one method returns token IDs, so a server can stand in for it later.

Why this step: you need a reference to compare every server against, and an interface every engine shares. The reference is Transformers. The template you tested in earlier parts runs there, and the model's authors usually tested with it. Part 1's render_ids already tokenizes the way the model expects: apply_chat_template with tokenize=True tokenizes the rendered string with add_special_tokens=False, because the template has written every special token already. TokenRenderer is a Protocol, Python's way of saying "anything with a name and an ids method", so the comparison in step 4 never needs to know which engine it is calling.

direction: down
classes: {
  proto: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  impl: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  later: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
}
proto: "TokenRenderer\nname, ids(conversation)" {class: proto}
tf: "TransformersTokens\n(this step)" {class: impl}
cpp: "LlamaCppTokens\n(step 2)" {class: later}
client: "ClientCompletionTokens\n(step 6)" {class: later}
proto <- tf: "implements"
proto <- cpp
proto <- client

Create templatestudio/serving.py:

# templatestudio/serving.py"""Token IDs from the engine itself: what the model receives after theserver's half of the render, which a template-string test never runs."""from __future__ import annotationsimport jsonimport urllib.requestfrom collections.abc import Callablefrom dataclasses import dataclassfrom typing import Any, Protocolfrom transformers import PreTrainedTokenizerBasefrom templatestudio.messages import Conversation, assert_validfrom templatestudio.render import first_divergence, render_ids# (url, JSON payload) -> parsed JSON reply. Tests can pass a fake.Transport = Callable[[str, dict], dict]def http_post(url: str, payload: dict, timeout: float = 30.0) -> dict:    """POST a JSON body and return the parsed JSON reply (stdlib only)."""    request = urllib.request.Request(        url,        data=json.dumps(payload).encode("utf-8"),        headers={"Content-Type": "application/json"},    )    with urllib.request.urlopen(request, timeout=timeout) as response:        return json.loads(response.read().decode("utf-8"))class TokenRenderer(Protocol):    """One engine's answer to: which token IDs does this request become?"""    name: str    def ids(        self,        conversation: Conversation,        tools: list[dict] | None = None,        **template_kwargs: Any,    ) -> list[int]: ...class TransformersTokens:    """The reference: apply_chat_template with tokenize=True, which does not    add special tokens a second time."""    name = "transformers"    def __init__(self, tokenizer: PreTrainedTokenizerBase) -> None:        self.tokenizer = tokenizer    def ids(        self,        conversation: Conversation,        tools: list[dict] | None = None,        **template_kwargs: Any,    ) -> list[int]:        return render_ids(            self.tokenizer, conversation, tools=tools, **template_kwargs        )

http_post is the module's one network function: it sends JSON and reads JSON. Step 2 is the first to call it, but it goes in now, with the imports, so that the module's first block stays the one every later step adds to. Transport is its type, so a test can pass a fake that returns canned replies without a server.

Create serve_reference.py:

# serve_reference.py"""The reference: the token IDs Transformers gives Qwen3 for one question."""from endpoints import QWEN_TOKENIZERfrom session import QUESTIONfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensqwen = load_tokenizer(QWEN_TOKENIZER)reference = TransformersTokens(qwen)ids = reference.ids(QUESTION)print(f"{reference.name}: {len(ids)} token IDs")print(ids[:9])print([qwen.decode([i]) for i in ids[:9]])

Run it:

python serve_reference.py

Expected output:

transformers: 25 token IDs[151644, 8948, 198, 2610, 525, 50537, 13, 151645, 198]['<|im_start|>', 'system', '\n', 'You', ' are', ' terse', '.', '<|im_end|>', '\n']

What just happened: the reference is a list of 25 integers. The first nine decode to the system turn. <|im_start|> is one token, ID 151644, not twelve characters, and so is <|im_end|>. Qwen3 has no <bos>, so the first ID is the template's own first token. This list is what the server must reproduce.

Step 2: Ask llama-server for the token IDs it builds

Goal: add a LlamaCppTokens engine that gets the token IDs from llama-server itself.

Why this step: you want the IDs the server builds with its own template, engine and tokenizer, not a local guess at them. No single llama-server endpoint returns a chat request's token IDs, so LlamaCppTokens chains two. /apply-template takes the same messages, tools and chat_template_kwargs a chat request takes, runs them through the same request parser and the GGUF's template, and returns {"prompt": ...}. /tokenize turns that prompt into {"tokens": [...]}. The two switches in the /tokenize body copy the chat path, which tokenizes the rendered prompt with special tokens parsed and the model's own special tokens added. /tokenize defaults add_special to false, so the adapter must say true. Step 3 shows what each switch changes.

direction: down
classes: {
  py: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  ep: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
ids: "LlamaCppTokens.ids" {class: py}
apply: "POST /apply-template" {class: ep}
prompt: "prompt string" {class: data}
tok: "POST /tokenize\nadd_special, parse_special" {class: ep}
tokens: "token IDs" {class: data}
ids -> apply: "messages, tools,\nchat_template_kwargs"
apply -> prompt -> tok -> tokens

Add LlamaCppTokens to the end of templatestudio/serving.py. A block whose first line ends in (add at the end) goes after the last line of that file, separated from it by two blank lines. In such a block the first line only says where the code goes: do not paste it.

# templatestudio/serving.py (add at the end)class LlamaCppTokens:    """llama-server: /apply-template renders with the GGUF's own template,    then /tokenize turns the prompt into IDs with add_special and    parse_special on, the way the server's chat path tokenizes."""    name = "llama.cpp"    def __init__(self, base_url: str, transport: Transport = http_post) -> None:        self.base_url = base_url.rstrip("/")        self.transport = transport    def ids(        self,        conversation: Conversation,        tools: list[dict] | None = None,        **template_kwargs: Any,    ) -> list[int]:        body: dict[str, Any] = {"messages": conversation}        if tools:            body["tools"] = tools  # the request field, so the tool path runs        if template_kwargs:            body["chat_template_kwargs"] = template_kwargs        prompt = self.transport(f"{self.base_url}/apply-template", body)["prompt"]        reply = self.transport(            f"{self.base_url}/tokenize",            {"content": prompt, "add_special": True, "parse_special": True},        )        return [int(t) for t in reply["tokens"]]

tools goes into the request as its own field, not into the template's keyword arguments, so the server's tool-parsing code runs on it as it would for a real chat request. Step 4 shows why that matters.

Create serve_llama.py:

# serve_llama.py"""The same question through llama-server's own template and tokenizer."""from endpoints import QWEN_TOKENIZER, QWEN_URLfrom session import QUESTIONfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import LlamaCppTokens, TransformersTokensreference = TransformersTokens(load_tokenizer(QWEN_TOKENIZER))engine = LlamaCppTokens(QWEN_URL)expected = reference.ids(QUESTION)actual = engine.ids(QUESTION)print(f"{reference.name}: {len(expected)} tokens, {engine.name}: {len(actual)} tokens")print("identical:", expected == actual)

Run it:

python serve_llama.py

Expected output:

transformers: 25 tokens, llama.cpp: 25 tokensidentical: True

What just happened: llama-server read Qwen3's template out of the GGUF, rendered it with its own C++ Jinja engine, tokenized the result with its own tokenizer, and produced the same 25 IDs as Transformers. That match is not automatic. The GGUF carries a copy of the template made when the model was converted, and a different engine runs it. For this conversation, all three stages agree.

Step 3: See how parse_special turns marker text into special tokens

Goal: tokenize the same prompt with parse_special on and off, and see what the server does with special-token text.

Why this step: a rendered prompt is a string, and the string <|im_start|> is ambiguous. It can mean the special token the model was trained on, or the twelve characters a user could type. parse_special decides which: on, the tokenizer matches special-token text to its single ID; off, it treats the text as ordinary characters. add_special decides something else, namely whether the tokenizer adds the model's own special tokens, such as <bos>, in front of the text. Qwen3 adds none, so it does not change this step's numbers; step 6 measures it on Gemma 3. Get either switch wrong in an adapter and you get IDs the server never sends.

direction: down
classes: {
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
text: "the text <|im_start|>" {class: data}
on: "parse_special on\n1 token: 151644" {class: good}
off: "parse_special off\n6 tokens: < | im _start | >" {class: bad}
text -> on
text -> off

Create serve_switches.py. It calls the two endpoints directly, so you see the raw replies:

# serve_switches.py"""What /tokenize's two switches do to a rendered Qwen3 prompt."""from endpoints import QWEN_URLfrom session import QUESTIONfrom templatestudio.serving import http_postprompt = http_post(f"{QWEN_URL}/apply-template", {"messages": QUESTION})["prompt"]for parse_special in (True, False):    body = {"content": prompt, "add_special": True, "parse_special": parse_special}    tokens = http_post(f"{QWEN_URL}/tokenize", body)["tokens"]    print(f"parse_special={parse_special}: {len(tokens)} tokens")body = {"content": "<|im_start|>", "parse_special": False, "with_pieces": True}pieces = http_post(f"{QWEN_URL}/tokenize", body)["tokens"]print("<|im_start|> as text:", [p["piece"] for p in pieces])

Run it:

python serve_switches.py

Expected output:

parse_special=True: 25 tokensparse_special=False: 46 tokens<|im_start|> as text: ['<', '|', 'im', '_start', '|', '>']

What just happened: with parse_special on, the prompt is 25 tokens, the same as step 1. With it off, it is 46. The prompt's five special-token strings, three <|im_start|> and two <|im_end|>, were split into ordinary tokens, 21 more in total. On its own, <|im_start|> becomes the six pieces the last line shows. Inside the prompt some pieces merge with the character next to them, such as .< and >\n, so the total is not five times six. A model given the 46-token version sees text that looks like its markers and is not, a prompt it was never trained on, and nothing reports an error. The with_pieces switch is a third one, and only for you: it returns each token's text next to its ID, which is the fastest way to read what a server did to a string.

Step 4: Compare token IDs across engines for three request shapes

Goal: write compare_token_renders, and run it on a plain request, a request with thinking turned off, and a request with a tool.

Why this step: step 2 compared one conversation by hand with ==. A test needs a report it can repeat over many request shapes, naming the engine and the first token where it leaves the reference. One passing shape tells you nothing about the next, because each reaches a different branch of the template and a different part of the server's request parser. The index comes from first_divergence, Part 1's helper, which works on lists of IDs as well as on strings. compare_token_renders validates the conversation with Part 2's assert_valid first, so a malformed conversation fails on your side of the boundary, with your error.

direction: down
classes: {
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  eng: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
conv: "one conversation\n+ tools + kwargs" {class: data}
ref: "renderers[0]\n(the reference)" {class: eng}
other: "every other engine" {class: eng}
diff: "first_divergence(expected, actual)\n-> TokenDivergence or nothing" {class: check}
conv -> ref
conv -> other
ref -> diff: "expected"
other -> diff: "actual"

Add the comparison to the end of templatestudio/serving.py:

# templatestudio/serving.py (add at the end)@dataclass(frozen=True)class TokenDivergence:    reference: str    engine: str    position: int  # first token index where the two differ    reference_len: int    engine_len: intdef compare_token_renders(    renderers: list[TokenRenderer],    conversation: Conversation,    tools: list[dict] | None = None,    **template_kwargs: Any,) -> list[TokenDivergence]:    """Send one conversation through every engine and report each engine    whose token IDs differ from the first one (the reference). IDs only    compare when every engine serves the same model."""    assert_valid(conversation)    reference, *others = renderers    expected = reference.ids(conversation, tools=tools, **template_kwargs)    divergences: list[TokenDivergence] = []    for engine in others:        actual = engine.ids(conversation, tools=tools, **template_kwargs)        position = first_divergence(expected, actual)        if position is None:            continue        divergences.append(            TokenDivergence(                reference=reference.name,                engine=engine.name,                position=position,                reference_len=len(expected),                engine_len=len(actual),            )        )    return divergences

Create serve_compare.py. tool_schema is Part 5's helper, which builds a tool's JSON schema from a Python function's type hints and docstring:

# serve_compare.py"""One question, three request shapes, two engines serving Qwen3."""from endpoints import QWEN_TOKENIZER, QWEN_URLfrom session import QUESTION, get_weatherfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import (    LlamaCppTokens,    TransformersTokens,    compare_token_renders,)from templatestudio.tools import tool_schemarenderers = [    TransformersTokens(load_tokenizer(QWEN_TOKENIZER)),    LlamaCppTokens(QWEN_URL),]shapes = {    "plain": {},    "thinking off": {"enable_thinking": False},    "tools": {"tools": [tool_schema(get_weather)]},}for label, kwargs in shapes.items():    divergences = compare_token_renders(renderers, QUESTION, **kwargs)    print(f"{label}: {'same' if not divergences else 'DIFFERS'}")    for d in divergences:        print(f"  {d.engine} leaves {d.reference} at token {d.position}")        print(f"  lengths: {d.reference_len} vs {d.engine_len}")

Run it:

python serve_compare.py

Expected output:

plain: samethinking off: sametools: DIFFERS  llama.cpp leaves transformers at token 106  lengths: 177 vs 168

What just happened: the plain request and the one with enable_thinking=False produce the same IDs on both engines, so llama-server passes chat_template_kwargs through to the template as Transformers does. The request with a tool does not. The two prompts are identical for 106 tokens, then part, and the server's is nine tokens shorter. Qwen3's template writes the tool list into the system turn, inside a <tools> block, and if you decode the tokens just before 106, they are the end of the tool's parameters, so the break is inside the tool's schema. Step 5 shows the same place as a string. Nothing about this request is unusual: one tool, built by the helper from Part 5. The next step finds which part of the schema the server lost.

Step 5: Find the field llama-server drops from a tool schema

Goal: compare the two rendered strings at the character where they part, and fix the request so every engine gets the same tool.

Why this step: a token index tells you where, not what. The strings say what. /apply-template returns the server's string, render from Part 1 returns Transformers', and first_divergence finds the first character that differs. This is the stage-1 difference from the diagram above. Transformers hands the template your tool dictionary as it is, while llama-server first parses each tool into its own structure, with a name, a description and parameters, and renders that.

direction: down
classes: {
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  tf: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  cpp: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  fix: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
schema: "tool_schema(get_weather)\n4 keys, with return" {class: data}
tf: "Transformers\nrenders 4 keys" {class: tf}
cpp: "llama-server\nkeeps 3" {class: cpp}
fix: "request_tool(schema)\n3 keys to every engine" {class: fix}
schema -> tf
schema -> cpp
cpp -> fix

The fix comes first, so the script can import it. Add request_tool to the end of templatestudio/serving.py:

# templatestudio/serving.py (add at the end)def request_tool(schema: dict) -> dict:    """A tool schema reduced to name, description and parameters, the keys    every engine here reads. Send every engine this, so none of them sees a    key another one drops."""    function = schema["function"]    keep = ("name", "description", "parameters")    reduced = {key: function[key] for key in keep if key in function}    return {"type": "function", "function": reduced}

Then create serve_tools.py. It compares the two strings with the full schema, then the token IDs with request_tool:

# serve_tools.py"""Why the tools request differs, and the request shape that fixes it."""from endpoints import QWEN_TOKENIZER, QWEN_URLfrom session import QUESTION, get_weatherfrom templatestudio.render import first_divergence, load_tokenizer, renderfrom templatestudio.serving import (    LlamaCppTokens,    TransformersTokens,    compare_token_renders,    http_post,    request_tool,)from templatestudio.tools import tool_schemaqwen = load_tokenizer(QWEN_TOKENIZER)schema = tool_schema(get_weather)print("schema keys:", list(schema["function"]))local = render(qwen, QUESTION, tools=[schema])body = {"messages": QUESTION, "tools": [schema]}served = http_post(f"{QWEN_URL}/apply-template", body)["prompt"]at = first_divergence(local, served)print(f"strings differ at character {at}")print("transformers:", repr(local[at - 12 : at + 30]))print("llama.cpp:   ", repr(served[at - 12 : at + 30]))renderers = [TransformersTokens(qwen), LlamaCppTokens(QWEN_URL)]tools = [request_tool(schema)]print("request_tool keys:", list(tools[0]["function"]))print("tokens now:", compare_token_renders(renderers, QUESTION, tools=tools))

Run it:

python serve_tools.py

Expected output:

schema keys: ['name', 'description', 'parameters', 'return']strings differ at character 436transformers: '": ["city"]}, "return": {"type": "string"}'llama.cpp:    '": ["city"]}}}\n</tools>\n\nFor each function'request_tool keys: ['name', 'description', 'parameters']tokens now: []

What just happened: Transformers' get_json_schema, which tool_schema calls, adds a return key describing what the function returns. The OpenAI function object that chat-completions requests use has no such key, and llama-server keeps only the keys it knows. So Transformers' prompt tells the model the tool returns a string, and the server's prompt does not. Which prompt is right depends on what the model was trained on. What cannot happen is for the two to differ, because if you test with one and serve the other, your tests are about a prompt your users never send. request_tool keeps the three keys every engine reads, and with it the token IDs match. Send tools through request_tool on both sides from now on. If your numbers in step 4 differ from this page, this step's string comparison tells you which field your build treats differently.

Step 6: Catch a doubled BOS token from a client that renders its own prompt

Goal: add a third engine, a client that renders Gemma 3's prompt with Transformers and posts the string, and compare its IDs with the other two.

This step changes model, to Gemma 3 on port 9931. Everything before used Qwen3, which has no <bos> token and so cannot show this failure.

Why this step: many clients do not send messages. They render the prompt themselves, with apply_chat_template(tokenize=False), and post the string to a completion endpoint, either because they want one template for several servers or because their framework was built that way. Such a client has taken over stage 2 and left stage 3 to a server that has no way to tell. /completion tokenizes a string prompt with add_special on, the same as /tokenize with add_special: true, so ClientCompletionTokens reproduces it with one call. Gemma 3 is the model to try it on. Its template starts with bos_token, and its GGUF also tells llama-server's tokenizer to add <bos> whenever it tokenizes with add_special on.

direction: down
classes: {
  tf: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  srv: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
render: "render(gemma, conversation)\n\"<bos><start_of_turn>user...\"" {class: tf}
server: "/completion tokenizes the string\nadd_special on: adds <bos>" {class: srv}
result: "[2, 2, 105, ...]\n<bos> twice" {class: bad}
chat: "messages to the chat path\n[2, 105, ...]" {class: good}
render -> server: "string"
server -> result
chat -> result: "differs at token 1" {style.stroke-dash: 3}

Add the client engine to the end of templatestudio/serving.py:

# templatestudio/serving.py (add at the end)class ClientCompletionTokens:    """The path of a client that renders the prompt itself and posts the    string to llama-server's /completion: the client's own render, then    /tokenize with add_special on, which is how /completion tokenizes."""    name = "client + /completion"    def __init__(        self,        render: Callable[..., str],        base_url: str,        transport: Transport = http_post,    ) -> None:        self.render = render  # the function your client really calls        self.base_url = base_url.rstrip("/")        self.transport = transport    def ids(        self,        conversation: Conversation,        tools: list[dict] | None = None,        **template_kwargs: Any,    ) -> list[int]:        prompt = self.render(conversation, tools=tools, **template_kwargs)        reply = self.transport(            f"{self.base_url}/tokenize",            {"content": prompt, "add_special": True, "parse_special": True},        )        return [int(t) for t in reply["tokens"]]

render is the function your client really calls. functools.partial(render, gemma) turns Part 1's render(tokenizer, conversation) into a function of the conversation alone.

Create serve_bos.py:

# serve_bos.py"""A client that renders Gemma 3's prompt itself and posts the string."""from functools import partialfrom endpoints import GEMMA_TOKENIZER, GEMMA_URLfrom session import QUESTIONfrom templatestudio.render import load_tokenizer, renderfrom templatestudio.serving import (    ClientCompletionTokens,    LlamaCppTokens,    TransformersTokens,    compare_token_renders,    http_post,)gemma = load_tokenizer(GEMMA_TOKENIZER)renderers = [    TransformersTokens(gemma),    LlamaCppTokens(GEMMA_URL),    ClientCompletionTokens(partial(render, gemma), GEMMA_URL),]served = http_post(f"{GEMMA_URL}/apply-template", {"messages": QUESTION})["prompt"]print("client's string starts:", repr(render(gemma, QUESTION)[:20]))print("server's prompt starts:", repr(served[:20]))for engine in renderers:    print(f"{engine.name}: {engine.ids(QUESTION)[:3]}")for d in compare_token_renders(renderers, QUESTION):    print(f"{d.engine} leaves {d.reference} at token {d.position}")    print(f"  lengths: {d.reference_len} vs {d.engine_len}")

Run it:

python serve_bos.py

Expected output:

client's string starts: '<bos><start_of_turn>'server's prompt starts: '<start_of_turn>user\n'transformers: [2, 105, 2364]llama.cpp: [2, 105, 2364]client + /completion: [2, 2, 105]client + /completion leaves transformers at token 1  lengths: 22 vs 23

What just happened: Transformers and llama-server's chat path agree, 22 tokens starting with one <bos>. The first two lines show how the chat path avoids a second one: the server's own render does not start with <bos>. When the tokenizer is going to add <bos> itself, llama-server leaves the template's leading <bos> out of the prompt it renders, so /apply-template returns a prompt that starts at <start_of_turn>, and tokenizing it adds exactly one. Nothing removes the <bos> at the start of the client's string, so the tokenizer adds a second one in front of it and the model gets 23 tokens. Every check you built in Parts 1 to 9 passes on this client, because its string is exactly the string Transformers renders. The template is right, and the tokenization after it is not.

Step 7: Confirm the doubled BOS on /completion and fix it

Goal: ask the server itself how many prompt tokens it evaluated: on /completion for the string and for the same prompt sent as token IDs, and on the real chat endpoint, /v1/chat/completions, for the messages.

Why this step: step 6 measured /tokenize, which is a stand-in for /completion, and LlamaCppTokens is a stand-in for the chat endpoint. A stand-in deserves one check against the real thing. /completion reports tokens_evaluated in its reply, and the chat endpoint reports usage.prompt_tokens, as an OpenAI-style API does. /completion also accepts the prompt as a list of token IDs instead of a string. It adds a <bos> to a string but none to a list of IDs (server README). That is the fix for a client that must render the prompt itself: tokenize on the client too, with TransformersTokens, and send the IDs. n_predict: 1 and max_tokens: 1 keep generation to a single token, and cache_prompt: false makes the server evaluate the whole prompt each time instead of reusing the previous call's.

direction: down
classes: {
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  srv: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
string: "prompt as a string\nserver adds <bos>" {class: bad}
ids: "prompt as token IDs\nserver adds nothing" {class: good}
completion: "POST /completion\nreply: tokens_evaluated" {class: srv}
msgs: "messages\nserver renders" {class: good}
chat: "POST /v1/chat/completions\nreply: usage.prompt_tokens" {class: srv}
string -> completion: "23 evaluated"
ids -> completion: "22 evaluated"
msgs -> chat: "22 evaluated"

Create serve_completion.py:

# serve_completion.py"""Ask the server itself how many prompt tokens it evaluated: /completion withthe rendered string, /completion with token IDs, and the chat endpoint."""from endpoints import GEMMA_TOKENIZER, GEMMA_URLfrom session import QUESTIONfrom templatestudio.render import load_tokenizer, renderfrom templatestudio.serving import TransformersTokens, http_postgemma = load_tokenizer(GEMMA_TOKENIZER)prompts = {    "string": render(gemma, QUESTION),    "token IDs": TransformersTokens(gemma).ids(QUESTION),}for label, prompt in prompts.items():    body = {"prompt": prompt, "n_predict": 1, "temperature": 0, "cache_prompt": False}    reply = http_post(f"{GEMMA_URL}/completion", body)    print(f"/completion, {label}: {reply['tokens_evaluated']} prompt tokens")body = {"messages": QUESTION, "max_tokens": 1, "temperature": 0, "cache_prompt": False}reply = http_post(f"{GEMMA_URL}/v1/chat/completions", body)print(f"/v1/chat/completions, messages: {reply['usage']['prompt_tokens']} prompt tokens")

Run it:

python serve_completion.py

Expected output:

/completion, string: 23 prompt tokens/completion, token IDs: 22 prompt tokens/v1/chat/completions, messages: 22 prompt tokens

What just happened: the server confirms both numbers from step 6. The string costs 23 prompt tokens and the ID list 22. The chat endpoint, given the messages, also evaluates 22, so the IDs LlamaCppTokens reported are the ones the real chat path sends. Now look at the terminal running the Gemma server. For the string request, llama.cpp logged a warning. You may find it more than once, because step 6's /tokenize call tokenized the same string the same way. The line starts with a timestamp and W, for warning, and yours has your own timestamp:

60.33.725.629 W llama_vocab::impl::tokenize: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want?

Most deployments never read that log, and the request succeeds.

Fix a doubled BOS: send messages or send token IDs

serve_completion.py has just run both fixes. In order of preference:

• Send messages. Use /v1/chat/completions and let the server render. The last line of the output above shows the chat path evaluates the reference's 22 tokens.
• Send token IDs. If the client must render, tokenize on the client too and send the list, as this step did.

A third option, not tested on this page, is to start the server with --override-kv tokenizer.ggml.add_bos_token=bool:false, which tells its tokenizer not to add <bos> (server README). It changes every request to that server, the chat path included. If you use it, re-run steps 6 and 7 and check that every line shows one <bos>.

Step 8: Test prefix-cache stability across chat requests

Goal: write templatestudio/prefix.py, and check a three-request Qwen3 session on both engines, with thinking on and off.

Why this step: a chat client sends the whole conversation on every request. An engine with a prefix cache, which llama-server, vLLM and SGLang all have, keeps the computed state of earlier prompts and reuses it for as many leading tokens as the next prompt shares. If request 2 starts with request 1 token for token, the engine only computes the new turns. If one early token changes, the engine computes everything after it again on that request, and the only symptom is latency. A client that changes it on every request pays that every time. Part 6's prefix_break checked one pair of requests through Transformers. find_prefix_breaks checks a whole session, through any engine's ids, and the test is mechanical: render the prompt at each point a client sends a request, after every user turn or tool result, and check that each prompt is a prefix of the next.

direction: down
classes: {
  req: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  cached: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
r1: "request 1: messages[:2]\n(system, user)" {class: req}
r2: "request 2: messages[:4]\nstarts with request 1?" {class: req}
r3: "request 3: messages[:6]\nstarts with request 2?" {class: req}
cache: "prefix cache\nreuses the shared tokens" {class: cached}
check: "find_prefix_breaks\nfirst token where it does not" {class: check}
r1 -> r2 -> r3
r2 -> cache {style.stroke-dash: 3}
r3 -> cache {style.stroke-dash: 3}
r2 -> check
r3 -> check

Create templatestudio/prefix.py:

# templatestudio/prefix.py"""The prefix contract: every prompt a conversation has already sent must bea token prefix of every later one, or the engine cannot reuse its cache."""from __future__ import annotationsfrom collections.abc import Callablefrom dataclasses import dataclassfrom templatestudio.messages import Conversationfrom templatestudio.render import first_divergence# conversation -> the token IDs the engine sees for that request# (a TokenRenderer's .ids method from serving.py).RenderIds = Callable[[Conversation], list[int]]@dataclass(frozen=True)class PrefixBreak:    """A later request that does not start with an earlier one."""    earlier: int  # messages in the earlier request    later: int  # messages in the later request    position: int  # first token index where the later request differsdef request_points(conversation: Conversation) -> list[int]:    """The lengths at which a client sends a request: after every message the    model must answer, a user turn or a tool result. A system message on its    own is never a request."""    return [        i + 1        for i, message in enumerate(conversation)        if message["role"] in ("user", "tool")    ]def find_prefix_breaks(    render_ids: RenderIds, conversation: Conversation, tail: int = 0) -> list[PrefixBreak]:    """Render the prompt at each request point and report every consecutive    pair where the earlier prompt is not a token prefix of the later one.    A break inside the last `tail` tokens of the earlier prompt is allowed."""    points = request_points(conversation)    prompts = [render_ids(conversation[:n]) for n in points]    breaks: list[PrefixBreak] = []    requests = list(zip(points, prompts))    for (n_a, a), (n_b, b) in zip(requests, requests[1:]):        position = first_divergence(a, b[: len(a)])        if position is None or position >= len(a) - tail:            continue        breaks.append(PrefixBreak(earlier=n_a, later=n_b, position=position))    return breaksdef assert_prefix_stable(    render_ids: RenderIds, conversation: Conversation, tail: int = 0) -> None:    """Raise if any request fails to extend the one before it token for    token, outside an allowed tail at the end of the earlier prompt."""    breaks = find_prefix_breaks(render_ids, conversation, tail)    if breaks:        first = breaks[0]        raise AssertionError(            f"{len(breaks)} prefix break(s), first at token {first.position} "            f"({first.earlier} -> {first.later} messages)"        )

Its render_ids parameter takes an engine's .ids method, a function of the conversation alone. Do not pass Part 1's render_ids function, which also needs a tokenizer.

request_points counts only user and tool turns. A system message alone is never a request, and an assistant turn is the reply, not a request. Leave the role filter out and the check compares a system-only prompt no client sends with the first real request, then reports a break that does not exist. b[: len(a)] compares the earlier prompt with the start of the later one, so a later prompt that only adds tokens passes.

Create serve_prefix.py. Its no_think function is the engine's ids with enable_thinking=False fixed, because find_prefix_breaks calls whatever function you give it with the conversation alone. The default argument engine=engine binds the loop's current engine when the function is defined; without it, both functions would call the last engine in the list:

# serve_prefix.py"""Check a three-request Qwen3 session for prefix breaks on both engines."""from endpoints import QWEN_TOKENIZER, QWEN_URLfrom session import SESSIONfrom templatestudio.prefix import PrefixBreak, find_prefix_breaksfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import LlamaCppTokens, TransformersTokensdef show(label: str, breaks: list[PrefixBreak]) -> None:    if not breaks:        print(f"  {label}: no breaks")    for b in breaks:        messages = f"{b.earlier} -> {b.later} messages"        print(f"  {label}: break at token {b.position} ({messages})")engines = [TransformersTokens(load_tokenizer(QWEN_TOKENIZER)), LlamaCppTokens(QWEN_URL)]for engine in engines:    print(engine.name)    show("thinking on ", find_prefix_breaks(engine.ids, SESSION))    def no_think(conversation, engine=engine):        return engine.ids(conversation, enable_thinking=False)    show("thinking off", find_prefix_breaks(no_think, SESSION))    show("off, tail=4 ", find_prefix_breaks(no_think, SESSION, tail=4))

Run it:

python serve_prefix.py

Expected output:

transformers  thinking on : no breaks  thinking off: break at token 25 (2 -> 4 messages)  thinking off: break at token 42 (4 -> 6 messages)  off, tail=4 : no breaksllama.cpp  thinking on : no breaks  thinking off: break at token 25 (2 -> 4 messages)  thinking off: break at token 42 (4 -> 6 messages)  off, tail=4 : no breaks

What just happened: with thinking on, every request starts with the one before it on both engines. This check compares prompts, not what the model generated: a real thinking-on reply carries reasoning that the next prompt drops, so the reply's cached tokens are recomputed anyway, and Part 6's prefix_break, which takes the generated IDs, is the check for that. With thinking off, each request breaks four tokens before the end of the one before it. That is by design. With enable_thinking=False, Qwen3's generation prompt ends in an empty reasoning block of four tokens: <think>, \n\n, </think>, \n\n. The next request renders the earlier reply without that block, as Part 6 showed, so the engine recomputes the last reply on every turn. Budget for that cost; it is not a bug. tail=4 tells the check to allow a break inside the last four tokens of the earlier prompt, and only there. A break anywhere before those four tokens is still reported, and the next step makes one.

Step 9: Find a prefix-cache miss caused by tool-list order

Goal: run assert_prefix_stable on a client that sends the same two tools in a different order after the first reply.

Why this step: the token that moves is usually in what the client sends, not in the template. Qwen3 writes the tool list into the system turn, before every message, so any change to the list moves an early token and every token after it. Reordering is a realistic way to change it. A client that builds the list from a set or a dictionary, or rediscovers tools from a tool server after a reconnect, can send the same tools in a new order, and the reordered function below is such a client. assert_prefix_stable is the form a test uses: it raises with the first break, so a CI job fails on it.

direction: down
classes: {
  ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  req: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
r1: "request 1\ntools: get_weather, get_time" {class: req}
r2: "request 2\ntools: get_time, get_weather" {class: req}
sys: "system turn: the tool list\nchanges at token 56" {class: bad}
rest: "every later token\nrecomputed" {class: bad}
same: "same order every request\nprefix stable" {class: ok}
r1 -> r2: "client reorders"
r2 -> sys -> rest
r1 -> same {style.stroke-dash: 3}

Create serve_cachebust.py:

# serve_cachebust.py"""A client that sends its tool list in a different order after turn one."""from endpoints import QWEN_TOKENIZER, QWEN_URLfrom session import SESSION, get_time, get_weatherfrom templatestudio.prefix import assert_prefix_stablefrom templatestudio.render import load_tokenizerfrom templatestudio.serving import LlamaCppTokens, TransformersTokens, request_toolfrom templatestudio.tools import tool_schematools = [request_tool(tool_schema(f)) for f in (get_weather, get_time)]engines = [TransformersTokens(load_tokenizer(QWEN_TOKENIZER)), LlamaCppTokens(QWEN_URL)]for engine in engines:    def stable(conversation, engine=engine):        return engine.ids(conversation, tools=tools)    def reordered(conversation, engine=engine):        order = tools if len(conversation) <= 2 else tools[::-1]        return engine.ids(conversation, tools=order)    for label, render_ids in (("same order", stable), ("reordered", reordered)):        try:            assert_prefix_stable(render_ids, SESSION)            print(f"{engine.name}, {label}: prefix stable")        except AssertionError as error:            print(f"{engine.name}, {label}: {error}")

Run it:

python serve_cachebust.py

Expected output:

transformers, same order: prefix stabletransformers, reordered: 1 prefix break(s), first at token 56 (2 -> 4 messages)llama.cpp, same order: prefix stablellama.cpp, reordered: 1 prefix break(s), first at token 56 (2 -> 4 messages)

What just happened: with the tools in the same order on every request, both engines keep the prefix. Reordered, both break at token 56, inside the system turn. That is a cache-buster: a change on the client's side that moves an early token, so the prefix cache stops matching. The engine can reuse only those first 56 tokens of its cache and computes every token after them again, and the longer the conversation at the point of the change, the more it recomputes. Request 3 extends request 2 cleanly, so the extra computation happens on one request only.

Nothing fails. The answers are correct, and the only symptom in production is a prefix-cache hit rate that drops. This check is the reliable way to see it, because a single timing depends on what the server ran before: llama-server can also restore prompts it saw in earlier requests, so the same request can look cheap on a second run.

The fix is on the client: sort the tool list by name, or keep it in the order it was first sent. Then run the same check on a session recorded from your own client rather than a made-up one, because the changes that break the prefix come from your data.

When it breaks: errors from checking a server's token IDs

The adapter has one known limit. /apply-template plus /tokenize with parse_special on reproduces today's chat path only because the server parses special-token text uniformly. An open llama.cpp issue, #28249, proposes tokenizing special-token text inside user messages as plain text. If a build ships that change, the adapter and the chat path will differ for messages that contain literal special-token strings, so re-run step 4 with such a message whenever you move to a newer build.

How the token-ID check fits together: your code, the boundary, and the server

The project now crosses a process boundary for the first time. Everything that decides what to test stays in Python, and the servers are measured, never trusted:

direction: down
classes: {
  mod: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  old: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  srv: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
earlier: "Earlier parts\nrender.py: render, render_ids, first_divergence\nmessages.py: assert_valid\ntools.py: tool_schema" {class: old}
data: "endpoints.py, session.py\nserve_*.py" {class: data}
serving: "templatestudio/serving.py" {
  label.near: top-center
  grid-rows: 2
  grid-columns: 3
  tf: "TransformersTokens" {class: mod}
  cpp: "LlamaCppTokens" {class: mod}
  client: "ClientCompletionTokens" {class: mod}
  cmp: "compare_token_renders\nTokenDivergence" {class: mod}
  tool: "request_tool" {class: mod}
  http: "http_post" {class: mod}
}
prefix: "templatestudio/prefix.py\nfind_prefix_breaks, assert_prefix_stable" {class: mod}
servers: "llama-server b12283" {
  label.near: top-center
  grid-rows: 1
  gemma: ":9931 Gemma 3 270M" {class: srv}
  qwen: ":9932 Qwen3-0.6B" {class: srv}
}
earlier -> serving: "imported by"
data -> serving: "drive"
data -> prefix
prefix -> serving: "takes any engine's ids"
serving -> servers: "JSON over HTTP"

compare_token_renders and find_prefix_breaks never know which engine they are talking to, so a new server is one class with a name and an ids method. vLLM is the obvious next one. Its /tokenize endpoint accepts messages and returns the token IDs of the full chat render, so a VllmTokens class is one http_post call.

Complete code for the token-ID check: serving.py and prefix.py

The project tree after this part, leaving out the earlier parts' scripts and the earlier modules this part does not import:

chat-templates-part1/├── templatestudio/│   ├── __init__.py│   ├── messages.py          (Part 2, last changed in Part 7)│   ├── prefix.py            (new)│   ├── render.py            (Part 1, last changed in Part 6)│   ├── serving.py           (new)│   └── tools.py             (Part 5)├── endpoints.py             (new)├── session.py               (new)├── serve_check.py           (new)├── serve_reference.py       (new)├── serve_llama.py           (new)├── serve_switches.py        (new)├── serve_compare.py         (new)├── serve_tools.py           (new)├── serve_bos.py             (new)├── serve_completion.py      (new)├── serve_prefix.py          (new)└── serve_cachebust.py       (new)

The complete templatestudio/serving.py:

# templatestudio/serving.py"""Token IDs from the engine itself: what the model receives after theserver's half of the render, which a template-string test never runs."""from __future__ import annotationsimport jsonimport urllib.requestfrom collections.abc import Callablefrom dataclasses import dataclassfrom typing import Any, Protocolfrom transformers import PreTrainedTokenizerBasefrom templatestudio.messages import Conversation, assert_validfrom templatestudio.render import first_divergence, render_ids# (url, JSON payload) -> parsed JSON reply. Tests can pass a fake.Transport = Callable[[str, dict], dict]def http_post(url: str, payload: dict, timeout: float = 30.0) -> dict:    """POST a JSON body and return the parsed JSON reply (stdlib only)."""    request = urllib.request.Request(        url,        data=json.dumps(payload).encode("utf-8"),        headers={"Content-Type": "application/json"},    )    with urllib.request.urlopen(request, timeout=timeout) as response:        return json.loads(response.read().decode("utf-8"))class TokenRenderer(Protocol):    """One engine's answer to: which token IDs does this request become?"""    name: str    def ids(        self,        conversation: Conversation,        tools: list[dict] | None = None,        **template_kwargs: Any,    ) -> list[int]: ...class TransformersTokens:    """The reference: apply_chat_template with tokenize=True, which does not    add special tokens a second time."""    name = "transformers"    def __init__(self, tokenizer: PreTrainedTokenizerBase) -> None:        self.tokenizer = tokenizer    def ids(        self,        conversation: Conversation,        tools: list[dict] | None = None,        **template_kwargs: Any,    ) -> list[int]:        return render_ids(            self.tokenizer, conversation, tools=tools, **template_kwargs        )class LlamaCppTokens:    """llama-server: /apply-template renders with the GGUF's own template,    then /tokenize turns the prompt into IDs with add_special and    parse_special on, the way the server's chat path tokenizes."""    name = "llama.cpp"    def __init__(self, base_url: str, transport: Transport = http_post) -> None:        self.base_url = base_url.rstrip("/")        self.transport = transport    def ids(        self,        conversation: Conversation,        tools: list[dict] | None = None,        **template_kwargs: Any,    ) -> list[int]:        body: dict[str, Any] = {"messages": conversation}        if tools:            body["tools"] = tools  # the request field, so the tool path runs        if template_kwargs:            body["chat_template_kwargs"] = template_kwargs        prompt = self.transport(f"{self.base_url}/apply-template", body)["prompt"]        reply = self.transport(            f"{self.base_url}/tokenize",            {"content": prompt, "add_special": True, "parse_special": True},        )        return [int(t) for t in reply["tokens"]]@dataclass(frozen=True)class TokenDivergence:    reference: str    engine: str    position: int  # first token index where the two differ    reference_len: int    engine_len: intdef compare_token_renders(    renderers: list[TokenRenderer],    conversation: Conversation,    tools: list[dict] | None = None,    **template_kwargs: Any,) -> list[TokenDivergence]:    """Send one conversation through every engine and report each engine    whose token IDs differ from the first one (the reference). IDs only    compare when every engine serves the same model."""    assert_valid(conversation)    reference, *others = renderers    expected = reference.ids(conversation, tools=tools, **template_kwargs)    divergences: list[TokenDivergence] = []    for engine in others:        actual = engine.ids(conversation, tools=tools, **template_kwargs)        position = first_divergence(expected, actual)        if position is None:            continue        divergences.append(            TokenDivergence(                reference=reference.name,                engine=engine.name,                position=position,                reference_len=len(expected),                engine_len=len(actual),            )        )    return divergencesdef request_tool(schema: dict) -> dict:    """A tool schema reduced to name, description and parameters, the keys    every engine here reads. Send every engine this, so none of them sees a    key another one drops."""    function = schema["function"]    keep = ("name", "description", "parameters")    reduced = {key: function[key] for key in keep if key in function}    return {"type": "function", "function": reduced}class ClientCompletionTokens:    """The path of a client that renders the prompt itself and posts the    string to llama-server's /completion: the client's own render, then    /tokenize with add_special on, which is how /completion tokenizes."""    name = "client + /completion"    def __init__(        self,        render: Callable[..., str],        base_url: str,        transport: Transport = http_post,    ) -> None:        self.render = render  # the function your client really calls        self.base_url = base_url.rstrip("/")        self.transport = transport    def ids(        self,        conversation: Conversation,        tools: list[dict] | None = None,        **template_kwargs: Any,    ) -> list[int]:        prompt = self.render(conversation, tools=tools, **template_kwargs)        reply = self.transport(            f"{self.base_url}/tokenize",            {"content": prompt, "add_special": True, "parse_special": True},        )        return [int(t) for t in reply["tokens"]]

The complete templatestudio/prefix.py:

# templatestudio/prefix.py"""The prefix contract: every prompt a conversation has already sent must bea token prefix of every later one, or the engine cannot reuse its cache."""from __future__ import annotationsfrom collections.abc import Callablefrom dataclasses import dataclassfrom templatestudio.messages import Conversationfrom templatestudio.render import first_divergence# conversation -> the token IDs the engine sees for that request# (a TokenRenderer's .ids method from serving.py).RenderIds = Callable[[Conversation], list[int]]@dataclass(frozen=True)class PrefixBreak:    """A later request that does not start with an earlier one."""    earlier: int  # messages in the earlier request    later: int  # messages in the later request    position: int  # first token index where the later request differsdef request_points(conversation: Conversation) -> list[int]:    """The lengths at which a client sends a request: after every message the    model must answer, a user turn or a tool result. A system message on its    own is never a request."""    return [        i + 1        for i, message in enumerate(conversation)        if message["role"] in ("user", "tool")    ]def find_prefix_breaks(    render_ids: RenderIds, conversation: Conversation, tail: int = 0) -> list[PrefixBreak]:    """Render the prompt at each request point and report every consecutive    pair where the earlier prompt is not a token prefix of the later one.    A break inside the last `tail` tokens of the earlier prompt is allowed."""    points = request_points(conversation)    prompts = [render_ids(conversation[:n]) for n in points]    breaks: list[PrefixBreak] = []    requests = list(zip(points, prompts))    for (n_a, a), (n_b, b) in zip(requests, requests[1:]):        position = first_divergence(a, b[: len(a)])        if position is None or position >= len(a) - tail:            continue        breaks.append(PrefixBreak(earlier=n_a, later=n_b, position=position))    return breaksdef assert_prefix_stable(    render_ids: RenderIds, conversation: Conversation, tail: int = 0) -> None:    """Raise if any request fails to extend the one before it token for    token, outside an allowed tail at the end of the earlier prompt."""    breaks = find_prefix_breaks(render_ids, conversation, tail)    if breaks:        first = breaks[0]        raise AssertionError(            f"{len(breaks)} prefix break(s), first at token {first.position} "            f"({first.earlier} -> {first.later} messages)"        )

The scripts are listed in the steps that create them.

Where to go next: add vLLM as a third engine and watch prefix-cache hits

• Add vLLM as a third engine. vLLM's POST /tokenize takes messages, tools and chat_template_kwargs and returns tokens for the whole chat render, with add_special_tokens defaulting to false because the template writes them (protocol). A VllmTokens class with one http_post call works with compare_token_renders unchanged. vLLM's GPU builds target Linux (GPU installation).
• The rest of the server's half. Earlier parts pointed four serving-side settings at this part, and it hands them on to the book. They are vLLM's --tool-call-parser (Part 5), which turns the model's output back into tool calls; --chat-template-content-format (Part 7), which decides how vLLM hands multimodal content to the template; the stop tokens a server is configured with (Part 8); and measuring cache hits on the server (Part 6), the next bullet. This part checks only the input side, the token IDs. Chapter 10 of the Handbook covers those settings, and Part 11 turns this part's comparisons into golden-token tests.
• Watch the hit rate in production. vLLM exports vllm:prefix_cache_queries and vllm:prefix_cache_hits, both counted in tokens (metrics). A ratio that drops after a deploy or a tool-server reconnect is a cache-buster, and find_prefix_breaks on a recorded session finds the token.
• Read the chapter. Chapter 10 of The Chat Templates Handbook, "The Template Behind the Endpoint", follows the request through vLLM, SGLang and Ollama as well, and the companion repository's v1.2/ch10-endpoint tag holds the chapter's serving.py and prefix.py with their tests.

References

• llama.cpp server README: /apply-template, /tokenize and its add_special, parse_special and with_pieces fields, /completion and its token-ID prompts, --port, -hf and --override-kv.
• llama.cpp release b12283: the build this page was verified on. Release v0.7.0: the default port change to 9931.
• llama.cpp llama-vocab.cpp: where the tokenizer adds <bos> and logs the doubled-BOS warning.
• llama.cpp cpu.Dockerfile: the Docker image's LLAMA_ARG_PORT=8080. llama.cpp PR #11489: the change that added /apply-template.
• llama.cpp issue #28249: marking template input so user text is not parsed as special tokens.
• Transformers chat templating documentation: why apply_chat_template tokenizes with add_special_tokens=False. Transformers 5.22.2 release notes: the Jinja sandbox hardening.
• OpenAI Chat Completions API reference: the tools[].function object, with name, description, parameters and strict, and no return key.
• Transformers get_json_schema: the schema builder behind Part 5's tool_schema, including its return key.
• Unsloth's Gemma 3 guide: its warning not to add a second <bos> with llama.cpp.
• vLLM tokenize protocol and vLLM metrics, with vLLM GPU installation for its platform requirements: the chat /tokenize request and the prefix-cache counters.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023, and Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, 2024: how engines share computed state across requests with a common prefix.
• template-studio v1.2/ch10-endpoint: the book's companion code for chapter 10.
• Models: ggml-org/gemma-3-270m-it-GGUF, Qwen/Qwen3-0.6B-GGUF, unsloth/gemma-3-270m-it (a copy of google/gemma-3-270m-it's tokenizer files), Qwen/Qwen3-0.6B.
• The Chat Templates Handbook, chapter 10, "The Template Behind the Endpoint": the serving boundary, the doubled BOS and the prefix contract.

Related Articles

Llms

• How to Catch Chat Template Regressions with Golden-Token Tests
• How to Test a Chat Template Across Jinja Engines
• How to Write a Chat Template for a Fine-Tuned Model and Test It
• How to Validate Chat Messages Before apply_chat_template