How to Test a Chat Template Across Jinja Engines

What you'll build: a cross-engine test for chat templates

A chat template that renders correctly in Python can crash, or render a different prompt, in Transformers.js, LM Studio or llama.cpp. This intermediate tutorial builds a test that renders a chat template on CPython Jinja2 and on @huggingface/jinja, diffs the output, and lints for constructs that do not port. It renders eight templates on two engines and prints where they disagree. Seven come from model repositories, and the last row is the template you wrote in Part 8. Each template gets three conversations, one per column: a plain chat, a question with tools, and a finished float call to a tool:

family    lint chat       tools      float call qwen3     0    same       same       @1073      llama3    0    same       same       js fails   gemma3    0    same       same       both fail  gemma4    0    same       same       @639       llama3.2  0    same       same       @1547      smollm3   2    same       js fails   js fails   gpt-oss   0    same       same       @864       part8     0    same       same       both fail  9 problem(s)

By the end you will have templatestudio/parity.py. It renders any chat template with CPython Jinja2, the engine Transformers uses, and with @huggingface/jinja, the JavaScript port that Transformers.js runs. It reports the first character where the two renders differ, and it lints a template for constructs the JavaScript port does not have. The tutorial is for engineers who develop against Transformers and serve the same template from somewhere else: a JavaScript stack, LM Studio, llama.cpp, or anything that does not embed Python.

Verified against Python 3.12.2, transformers==5.22.1, tokenizers==0.23.3, huggingface-hub==2.4.0, jinja2==3.1.6, Node.js 20.20.0 with npm 10.8.2, and @huggingface/jinja==0.5.12 on 2026-11-11.

Read the table before you build it. Every template renders the plain chat the same on both engines, which is why nobody notices the problem until production.

• tools column, the loud failure. SmolLM3's tool branch crashes the JavaScript port.
• float call column, the quiet one. When a tool call's arguments arrive as a dict, the argument 1.0 renders as 1.0 in Python and 1 in JavaScript, and nothing raises. @1073 means the first 1,073 characters match and the prompts differ from there.
• Llama 3 in the float call column crashes the JavaScript port for a third reason, measured in step 7 and explained in step 9.
• lint counts the lines step 8's lint flags, and js fails means only the JavaScript port raised.
• both fail means both engines raised on that conversation. The template itself refuses it, so this is a template rule, not a difference between engines.

The script also exits with status 1, so the same table can fail a CI job.

This happens in production. In January 2026, llama.cpp returned HTTP 500 on /v1/chat/completions for Unsloth's Qwen3-Coder template, with Value is not callable: null at row 62, column 115: (issue #18852). The same template rendered in Python. A test that renders on the engine you deploy catches that before a user does.

This is Part 9 of Chat Templates in Practice and continues the project from Part 8, How to Write a Chat Template for a Fine-Tuned Model and Test It. It takes about 25 minutes. It needs Node.js and downloads one npm package of about 450 KB and three new tokenizers of about 61 MB in total. It needs no GPU, no API key, no account and no Hugging Face token.

Prerequisites: install Node.js and @huggingface/jinja 0.5.12

You need the project from the earlier parts: the chat-templates-part1/ folder with its virtual environment, and these files from it:

• templatestudio/render.py (Part 1, last changed in Part 6)
• templatestudio/messages.py (Part 2, last changed in Part 7)
• templatestudio/jinja_env.py (Part 3)
• templatestudio/tools.py (Part 5)
• templatestudio/authoring.py (Part 8)
• naive.py (Part 1), rates.py (Part 5) and families.py (Part 4, last changed in Part 8)

If one is missing, copy it from the complete code of the part that last changed it: Part 1, Part 3, Part 5, Part 6, Part 7 and Part 8. This part adds one module, templatestudio/parity.py, one Node script, engines/hf_jinja.mjs, two data files, amounts.py and constructs.py, and ten scripts. The only earlier file it changes is families.py, which gets one constant at the end.

Go to the project folder, activate the environment, and turn off the Transformers advisory warnings. Run the cd from the folder that contains chat-templates-part1/:

cd chat-templates-part1source .venv/bin/activateexport TRANSFORMERS_NO_ADVISORY_WARNINGS=1

On Windows, activate with .venv\Scripts\activate in PowerShell or source .venv/Scripts/activate in Git Bash, and set the variable in PowerShell with $env:TRANSFORMERS_NO_ADVISORY_WARNINGS = "1". Every script in this part runs from the project root.

The Python pins from Parts 1 to 8 are unchanged: Python 3.12, transformers 5.22.1, tokenizers 0.23.3, huggingface-hub 2.4.0 and jinja2 3.1.6. Nothing in this part needs PyTorch. Transformers 5.22.2 (2026-11-06) hardens the Jinja sandbox, but its changes target inputs that are not plain JSON and namespace attributes that start with an underscore, and this part uses neither. The series moves to it in one step at the end, not part by part.

This part adds Node.js and one npm package. @huggingface/jinja needs Node.js 18 or later.

• Check: run node --version. If the command is not found, install Node.js, then open a new terminal before you check again.
• Install: use a current long-term support release, such as Node.js 26 from nodejs.org. The npm that ships with it is fine; any npm from version 7 on handles the two commands below the same way.
• Verified on: Node.js 20.20.0 with npm 10.8.2. That release still runs this tutorial, but it stopped receiving security fixes on 2026-04-30.

From the project root, create a package.json and install the package, pinned to an exact version:

npm init -ynpm install --save-exact @huggingface/jinja@0.5.12

Run npm init -y first even though it looks optional. Without a package.json in the project root, npm installs into the nearest parent folder that has a package.json or a node_modules/ folder, and the scripts below then cannot find the package. --save-exact writes "0.5.12" into package.json instead of "^0.5.12". That matters here. The package is still at 0.x and fixes Python differences in patch releases, so a newer version can change the output of step 7. The two commands create package.json, package-lock.json and a node_modules/ folder in the project root. Add node_modules/ to your .gitignore if the project is a Git repository.

A block whose first line ends in (add at the end) goes after the last line of that file, separated from it by two blank lines. In such a block the first line only says where the code goes: do not paste it. Add the templates this part tests to the end of families.py:

# families.py (add at the end)# Part 9: templates to render on more than one Jinja engine. Tokenizers only.PARITY_MODELS: dict[str, str] = {    **FAMILIES,    "gemma4": GEMMA4,    "llama3.2": "unsloth/Llama-3.2-1B-Instruct",    "smollm3": "HuggingFaceTB/SmolLM3-3B",    "gpt-oss": "openai/gpt-oss-20b",}

FAMILIES holds Qwen3, Llama 3 and Gemma 3 from Part 4, and GEMMA4 is Gemma 4 E2B. The three new repositories are ungated, and only their tokenizers download.

Create parity_versions.py. It runs node --version and reads the installed package's own package.json:

# parity_versions.pyimport jsonimport subprocessfrom pathlib import Pathimport jinja2import transformersnode = subprocess.run(["node", "--version"], capture_output=True, text=True, check=True)package = Path("node_modules/@huggingface/jinja/package.json")js_port = json.loads(package.read_text(encoding="utf-8"))["version"]print("node", node.stdout.strip())print("@huggingface/jinja", js_port)print("transformers", transformers.__version__, "jinja2", jinja2.__version__)

Check the install before you go on:

python parity_versions.py

Expected output, with your Node.js version on the first line:

node v20.20.0@huggingface/jinja 0.5.12transformers 5.22.1 jinja2 3.1.6

If the second line is not 0.5.12, run both npm commands again from the project root. If the script raises a FileNotFoundError, its message names what is missing: node itself, or the package's package.json. When it breaks has a row for each.

How a chat template renders differently on two Jinja engines: the parity gap

A chat template is a string. To run it, an inference stack needs a Jinja engine. Transformers, vLLM (renderers/hf.py) and SGLang run CPython Jinja2, the reference implementation, through the tokenizer's apply_chat_template. Transformers.js (tokenization_utils.js) runs @huggingface/jinja, a JavaScript port written for chat templates. LM Studio uses the same port, according to the error messages its users report (LM Studio #626). llama.cpp runs its own C++ engine, which its authors say was partly translated from the JavaScript port (PR #18462). These are separate programs that read the same language. The engine belongs to whoever runs the stack, not to whoever wrote the template, which is part of why the chat template is the last unowned layer in your LLM stack.

The Chat Templates Handbook, the book this series accompanies, names the difference between what one template renders on two engines the parity gap. The gap has two faces. The loud face is a crash: the template uses a filter or a test the engine does not have, and the request fails. You find it on the first request. The silent face is worse. Both engines render, and the two prompts differ by a character, a number format or a key order. Nothing fails. This is the silent format penalty from Part 1 again, caused this time by the engine.

You cannot find the gap by reading two engines' source code. You find it by experiment, the way Part 4 found a template's behaviour with differential probing. Give both engines the same template and the same variables, then compare the strings they return. The diagram shows the test you are about to build:

direction: down
classes: {
  input: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  engine: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  loud: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  quiet: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
inputs: "The same inputs" {
  label.near: top-center
  grid-rows: 1
  template: "chat template\n(tokenizer.chat_template)" {class: input}
  variables: "variables\n(messages, tools, tokens)" {class: input}
}
engines: "Two engines" {
  label.near: top-center
  grid-rows: 1
  py: "CPython Jinja2\n(the reference)" {class: engine}
  js: "@huggingface/jinja\n(the JavaScript port)" {class: engine}
}
diff: "first_divergence(expected, actual)" {class: check}
faces: "The parity gap" {
  label.near: top-center
  grid-rows: 1
  loud: "Loud face:\none engine raises" {class: loud}
  silent: "Silent face:\nboth render, strings differ" {class: quiet}
}
inputs -> engines: "1. render on each"
engines -> diff: "2. compare"
diff -> faces: "3. classify"

The reference engine's output is the expected string. That is the engine the template's author tested with, and usually the one the model's training data went through. Every other engine is compared against it. In this tutorial the second engine is the JavaScript port, because npm installs it in seconds. The same harness takes any engine you can call from Python, and Where to go next shows how a llama.cpp adapter fits.

Step 1: How to run @huggingface/jinja from Python

Goal: write a Node script that renders one template with the JavaScript port, and call it from Python.

Why this step: your harness is Python and the engine is JavaScript, so something has to cross between the two processes. The simplest bridge is JSON, because both languages read and write it the same way. Python writes the template and its variables to the script's standard input, and the script writes back the rendered text or the error. An error is a normal reply here, not a crash, because a template that fails on this engine is exactly what you are testing for.

direction: down
classes: {
  py: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  node: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
python: "Python process\n(your harness)" {class: py}
node: "node engines/hf_jinja.mjs\n(new Template(...).render(...))" {class: node}
python -> node: "1. stdin: {template, variables}"
node -> python: "2. stdout: {text} or {error}"

Create the folder engines/ and the file engines/hf_jinja.mjs. The .mjs extension tells Node to load it as a JavaScript module, so it can import the package. Node finds the package in the node_modules/ folder you installed in the project root:

// engines/hf_jinja.mjs// Render one chat template with @huggingface/jinja, the JavaScript Jinja port.// Reads {"template": ..., "variables": {...}} as JSON on stdin and writes// {"text": ...} or {"error": ...} as JSON on stdout.import { readFileSync } from "node:fs";import { Template } from "@huggingface/jinja";const job = JSON.parse(readFileSync(0, "utf8"));let reply;try {  reply = { text: new Template(job.template).render(job.variables) };} catch (error) {  reply = { error: `${error.name}: ${error.message}` };}process.stdout.write(JSON.stringify(reply));

Create parity_js_engine.py. It sends two jobs: one that renders, and one that uses a filter the port does not have:

# parity_js_engine.pyimport jsonimport subprocessjobs = [    {        "template": "{{ greeting | upper }}, {{ name }}!",        "variables": {"greeting": "hi", "name": "Ada"},    },    {"template": "{{ 2.5 | round }}", "variables": {}},]for job in jobs:    result = subprocess.run(        ["node", "engines/hf_jinja.mjs"],        input=json.dumps(job),        capture_output=True,        text=True,        encoding="utf-8",        check=True,    )    print(result.stdout)

Run it:

python parity_js_engine.py

Expected output:

{"text":"HI, Ada!"}{"error":"Error: Unknown NumericValue filter: round"}

What just happened: the first job came back as {"text": ...} and the second as {"error": ...}. CPython Jinja2 renders {{ 2.5 | round }} as 2.0, but the port has no round filter, so the second line is the loud face in miniature. The engine on the JavaScript side works, and the script did not crash on the error, so the Python side can read both outcomes the same way. The next two steps give each engine a Python adapter with the same interface.

Step 2: Build the variables apply_chat_template passes to a template

Goal: collect the variables a template sees into one dictionary, and render it with CPython Jinja2 exactly as apply_chat_template does.

Why this step: a diff between two engines only means something if the engines get identical input. If the JavaScript side receives different variables, every difference you find could come from your harness instead of the engine. So you build the variables once, in Python, and hand the same dictionary to every engine. Part 3 worked out which variables Transformers 5.22.1 passes: the messages with their reasoning fields aliased, tools, documents, add_generation_prompt and the tokenizer's special tokens. chat_variables is that list as one function.

direction: down
classes: {
  src: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  fn: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  out: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
sources: "What a template sees" {
  label.near: top-center
  grid-rows: 1
  tokens: "special_tokens_map\n(eos_token, pad_token)" {class: src}
  conv: "conversation\n(assert_valid, alias_reasoning)" {class: src}
  opts: "tools, add_generation_prompt" {class: src}
}
build: "chat_variables(...)" {class: fn}
vdict: "one variables dict" {class: out}
cpy: "CPythonJinja().render(...)" {class: fn}
same: "== apply_chat_template" {class: ok}
sources -> build
build -> vdict
vdict -> cpy: "used by every engine"
cpy -> same: "checked in this step"

Create templatestudio/parity.py. This first block writes the module's whole import block, so later steps only add code at the end. json, re, subprocess, dataclass, field and Path stay unused until steps 3, 4 and 8, and a linter that flags unused imports will complain until then. That is expected:

# templatestudio/parity.pyfrom __future__ import annotationsimport jsonimport reimport subprocessfrom dataclasses import dataclass, fieldfrom pathlib import Pathfrom typing import Protocolfrom jinja2 import TemplateErrorfrom transformers import PreTrainedTokenizerBasefrom templatestudio.jinja_env import alias_reasoning, build_template_envfrom templatestudio.messages import Conversation, assert_validfrom templatestudio.render import first_divergenceclass EngineError(RuntimeError):    """An engine could not render the template."""class EngineRenderer(Protocol):    """One adapter per Jinja engine: same template, same variables, a string back."""    name: str    def render(self, template_source: str, variables: dict[str, object]) -> str: ...def chat_variables(    tokenizer: PreTrainedTokenizerBase,    conversation: Conversation,    *,    add_generation_prompt: bool = True,    tools: list[dict] | None = None,    **template_kwargs: object,) -> dict[str, object]:    """Build the variables apply_chat_template passes a template, once for all engines."""    assert_valid(conversation)    return {        **tokenizer.special_tokens_map,        "messages": alias_reasoning(conversation),        "tools": tools,        "documents": None,        "add_generation_prompt": add_generation_prompt,        **template_kwargs,    }class CPythonJinja:    """The reference: CPython Jinja2, configured like Transformers 5.22.1."""    name = "cpython-jinja2"    def render(self, template_source: str, variables: dict[str, object]) -> str:        try:            template = build_template_env().from_string(template_source)            return template.render(**variables)        except TemplateError as error:            raise EngineError(f"{type(error).__name__}: {error}") from error

EngineRenderer is a Protocol. Any class with a name and a render(template_source, variables) method counts as an engine, with no base class to inherit. CPythonJinja uses Part 3's build_template_env, the environment configured like Transformers'. chat_variables also runs Part 2's assert_valid first, so a malformed conversation fails before any engine sees it. CPythonJinja.render turns a Jinja TemplateError into EngineError, so a template that refuses its input looks the same from every engine.

Create parity_reference.py, which renders Part 1's CONVERSATION with Qwen3's template and compares the result with Part 1's render:

# parity_reference.pyfrom families import FAMILIESfrom naive import CONVERSATIONfrom templatestudio.parity import CPythonJinja, chat_variablesfrom templatestudio.render import load_tokenizer, rendertokenizer = load_tokenizer(FAMILIES["qwen3"])variables = chat_variables(tokenizer, CONVERSATION)text = CPythonJinja().render(tokenizer.chat_template, variables)print(sorted(variables))print(len(text), "characters")print("same as apply_chat_template:", text == render(tokenizer, CONVERSATION))

Run it:

python parity_reference.py

Expected output:

['add_generation_prompt', 'documents', 'eos_token', 'messages', 'pad_token', 'tools']292 characterssame as apply_chat_template: True

What just happened: the dictionary holds the six variables a Qwen3 template sees. Rendered with your reference engine, it gives the same 292 characters as apply_chat_template, and that equality is what makes CPython Jinja2 a reference you can trust. From here on, CPythonJinja stands in for Transformers. Had the last line printed False, every later diff would be measuring your harness.

Step 3: Render the same chat template in JavaScript

Goal: add a NodeJinja engine that sends the same variables to the Node script.

Why this step: the second engine sits behind the same interface, so the only new thing to understand is the boundary between the processes. NodeJinja turns the variables into JSON text with json.dumps, and Node turns that text back into JavaScript values with JSON.parse. Most values cross unchanged. Numbers do not. JavaScript has a single number type, so the text 1.0 and the text 1 both become the number 1, and the engine never sees a difference. A JavaScript server receives exactly the same numbers from a request body. So the adapter is faithful, and step 5 shows what that costs.

direction: right
classes: {
  py: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  wire: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
  js: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
pyval: "Python value\n{'amount': 1.0}" {class: py}
text: "JSON text\n{\"amount\": 1.0}" {class: wire}
jsval: "JavaScript value\n{amount: 1}" {class: js}
pyval -> text: "json.dumps"
text -> jsval: "JSON.parse"

Add NodeJinja to the end of templatestudio/parity.py:

# templatestudio/parity.py (add at the end)class NodeJinja:    """@huggingface/jinja, the JavaScript port, run by Node in a subprocess."""    name = "huggingface-jinja"    def __init__(self, script: Path = Path("engines/hf_jinja.mjs")) -> None:        self.script = script    def render(self, template_source: str, variables: dict[str, object]) -> str:        job = json.dumps({"template": template_source, "variables": variables})        result = subprocess.run(            ["node", str(self.script)],            input=job,            capture_output=True,            text=True,            encoding="utf-8",            check=True,        )        reply = json.loads(result.stdout)        if "error" in reply:            raise EngineError(reply["error"])        return reply["text"]

check=True makes Python raise if Node itself crashes, which is a broken setup rather than a template failure. A template error comes back as {"error": ...} and becomes EngineError, the same exception CPythonJinja raises. The adapter starts one Node process per render, so every render pays Node's start-up time. A long-lived Node process would be faster. One process per render keeps every render independent, though, and the 24 comparisons in step 9 still finish in seconds.

Create parity_node.py. It renders the conversation from step 2 on the JavaScript port and prints the result. Its sys.stdout.reconfigure(encoding="utf-8") line keeps Windows from failing on non-ASCII output when stdout is redirected; parity_float.py and parity_battery.py keep it too:

# parity_node.pyimport sysfrom families import FAMILIESfrom naive import CONVERSATIONfrom templatestudio.parity import NodeJinja, chat_variablesfrom templatestudio.render import load_tokenizer, rendersys.stdout.reconfigure(encoding="utf-8")tokenizer = load_tokenizer(FAMILIES["qwen3"])variables = chat_variables(tokenizer, CONVERSATION)text = NodeJinja().render(tokenizer.chat_template, variables)print(text, end="")print("---")print("template:", len(tokenizer.chat_template), "characters")print("render:", len(text), "characters")print("same as apply_chat_template:", text == render(tokenizer, CONVERSATION))

Run it:

python parity_node.py

Expected output:

<|im_start|>systemYou are a terse support agent. Answer in one sentence.<|im_end|><|im_start|>userWhere is my order #4471?<|im_end|><|im_start|>assistantIt shipped Monday and arrives Thursday.<|im_end|><|im_start|>userCan I change the delivery address?<|im_end|><|im_start|>assistant---template: 4168 charactersrender: 292 characterssame as apply_chat_template: True

What just happened: a second engine, in another language and another process, rendered Qwen3's 4,168-character template to the same 292 characters as Transformers. Both adapters now share one interface, so the comparison in the next step does not need to know which engine is which. The two engines agree here because this conversation holds no numbers, booleans or tool calls, and nothing crossed the JSON boundary in a form JavaScript changes. Step 5 sends one that does.

Step 4: Diff a chat template across engines

Goal: compare every engine against the reference and record where each one diverges.

Why this step: step 3 compared by hand with ==. A test needs a report: which engine, identical or not, the first character where it differs, and the error if it crashed. The report records the error instead of raising it, because the crash is one of the two results you are looking for, and one engine's crash must not hide another engine's result. The index comes from first_divergence, the helper from Part 1 that already works on strings.

direction: down
classes: {
  fn: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  engine: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  report: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
ref: "renderers[0]: the reference\nrenders once -> expected" {class: engine}
other: "each other engine\nrenders -> actual, or EngineError" {class: engine}
diff: "first_divergence(expected, actual)" {class: fn}
report: "ParityReport\nidentical, first_divergence, error" {class: report}
ref -> diff
other -> diff
diff -> report
other -> report: "error"

Add the report and the comparison to the end of templatestudio/parity.py:

# templatestudio/parity.py (add at the end)@dataclassclass ParityReport:    """How one engine's render compares with the reference engine's."""    reference: str    other: str    identical: bool    first_divergence: int | None  # character index, or None when identical    error: str | None = None  # set when the other engine could not render    expected: str = field(default="", repr=False)    actual: str = field(default="", repr=False)def compare_engines(    template_source: str,    variables: dict[str, object],    renderers: list[EngineRenderer],) -> list[ParityReport]:    """Render once per engine and diff every engine against the first one."""    reference, *others = renderers    expected = reference.render(template_source, variables)    reports: list[ParityReport] = []    for engine in others:        try:            actual = engine.render(template_source, variables)        except EngineError as error:            reports.append(                ParityReport(                    reference.name, engine.name, False, None, str(error), expected                )            )            continue        index = first_divergence(expected, actual)        reports.append(            ParityReport(                reference.name, engine.name, index is None, index, None, expected, actual            )        )    return reports

expected and actual keep both renders on the report with repr=False, so printing a report stays short while a script can still show the text around a divergence. If the reference engine itself fails, compare_engines lets that EngineError propagate. With no expected string, there is nothing to compare against.

Create parity_families.py. It runs the comparison on every template in PARITY_MODELS:

# parity_families.pyfrom families import PARITY_MODELSfrom naive import CONVERSATIONfrom templatestudio.parity import CPythonJinja, NodeJinja, chat_variables, compare_enginesfrom templatestudio.render import load_tokenizerengines = [CPythonJinja(), NodeJinja()]for family, model_id in PARITY_MODELS.items():    tokenizer = load_tokenizer(model_id)    variables = chat_variables(tokenizer, CONVERSATION)    [report] = compare_engines(tokenizer.chat_template, variables, engines)    print(f"{family:9} {report.identical!s:6} {len(report.expected)} characters")

Run it:

python parity_families.py

The first run downloads the Llama 3.2, SmolLM3 and gpt-oss tokenizers. It can print the Hub's unauthenticated-requests warning and, on Windows, a symlinks warning. Both are harmless.

Expected output:

qwen3     True   292 charactersllama3    True   430 charactersgemma3    True   282 charactersgemma4    True   252 charactersllama3.2  True   493 characterssmollm3   True   406 charactersgpt-oss   True   590 characters

What just happened: seven templates from five model families, written by different teams, render the plain chat identically on both engines. That is the normal result, and it is why the parity gap survives: a team that checks one ordinary conversation on a new engine sees True and moves on. The next two steps send conversations that reach other branches of the same templates.

Step 5: Find the silent face: a tool-call argument that renders differently

Goal: render a tool call whose argument is a float, and find where the two engines disagree.

Why this step: a template prints a tool call's arguments into the prompt, usually with tojson, and arguments are where numbers live. How the number reaches the template depends on its form. An OpenAI-style API sends arguments as a JSON string, and Qwen3's template copies a string as it is. Your own agent code, or a server that parses the string first, passes a dict instead, and then the template serializes the number itself. Each engine does that its own way.

grid-rows: 1
grid-gap: 40
classes: {
  step: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  same: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
string: "arguments as a JSON string" {
  label.near: top-center
  grid-columns: 1
  a: "'{\"amount\": 1.0, ...}'" {class: step}
  b: "template copies the text" {class: step}
  c: "both engines: 1.0" {class: same}
}
dict: "arguments as a dict" {
  label.near: top-center
  grid-columns: 1
  a: "{'amount': 1.0, ...}" {class: step}
  b: "template applies tojson" {class: step}
  c: "Python: 1.0\nJavaScript: 1" {class: bad}
}

Create amounts.py with a tool that takes a float and one finished call to it. ARGUMENTS is the string exactly as the API sends it:

# amounts.py# A tool that takes a float, and one finished call to it.from rates import QUESTION# The arguments exactly as an OpenAI-style API sends them: a JSON string.ARGUMENTS = '{"amount": 1.0, "base": "EUR", "quote": "JPY"}'def convert(amount: float, base: str, quote: str) -> float:    """Convert an amount of one currency into another.    Args:        amount: How much of the base currency, such as 1.0.        base: ISO 4217 code of the currency you hold, such as "EUR".        quote: ISO 4217 code of the currency you want, such as "JPY".    """    return 162.4 * amountdef convert_loop(arguments: dict | str) -> list[dict]:    """The question, the assistant's call to convert, and the tool's answer."""    call = {"type": "function", "function": {"name": "convert", "arguments": arguments}}    return [        *QUESTION,        {"role": "assistant", "tool_calls": [call]},        {"role": "tool", "name": "convert", "content": "162.4"},    ]

Create parity_float.py. It renders the same tool loop twice: once with the string, and once with the dict that Part 5's normalize_arguments makes from it. Around a divergence it prints 22 characters before and 10 after, through Part 1's visible_whitespace:

# parity_float.pyimport sysfrom amounts import ARGUMENTS, convert, convert_loopfrom families import FAMILIESfrom templatestudio.parity import CPythonJinja, NodeJinja, chat_variables, compare_enginesfrom templatestudio.render import load_tokenizer, visible_whitespacefrom templatestudio.tools import normalize_arguments, tool_schemasys.stdout.reconfigure(encoding="utf-8")tokenizer = load_tokenizer(FAMILIES["qwen3"])engines = [CPythonJinja(), NodeJinja()]tools = [tool_schema(convert)]forms = {"JSON string": ARGUMENTS, "dict": normalize_arguments(ARGUMENTS)}for label, arguments in forms.items():    variables = chat_variables(tokenizer, convert_loop(arguments), tools=tools)    [report] = compare_engines(tokenizer.chat_template, variables, engines)    if report.identical:        print(f"arguments as a {label}: identical")        continue    at = report.first_divergence    print(f"arguments as a {label}: differ at character {at}")    print("  cpython-jinja2   ", visible_whitespace(report.expected[at - 22 : at + 10]))    print("  huggingface-jinja", visible_whitespace(report.actual[at - 22 : at + 10]))

Run it:

python parity_float.py

Expected output:

arguments as a JSON string: identicalarguments as a dict: differ at character 1073  cpython-jinja2    guments":·{"amount":·1.0,·"base"  huggingface-jinja guments":·{"amount":·1,·"base":·

What just happened: with the string, both prompts are the same. With the dict, they are identical for 1,073 characters, and then one engine writes 1.0 where the other writes 1. Neither engine raised, and each prompt looks correct on its own. That is what the silent face looks like in a real template.

The dict path is not an edge case. SGLang parses string arguments into dicts before it renders the template (serving_chat.py), and your own agent loop does the same if it uses Part 5's tool_call_message, which stores arguments as a dict. Which number format is right depends on what the model was trained on. Either way, the two engines are not interchangeable, and only a diff shows it. If you serve on a JavaScript engine, keep arguments as the JSON string the API sent. The template copies it, and both engines write the same text. The matrix at the top of this page found the same divergence in the Gemma 4, Llama 3.2 and gpt-oss templates, at different positions.

Step 6: Find the loud face: a filter the JavaScript port does not have

Goal: render a template whose tool branch uses a filter the JavaScript port lacks, and read the error from the report.

Why this step: SmolLM3's template builds its tool list by writing each tool schema with (tool | string). In CPython Jinja2, the string filter turns a dict into its Python representation, and the JavaScript port has no string filter for dicts. The plain chat never reaches that line, which is why step 4 printed True for SmolLM3. A request with tools does.

direction: down
classes: {
  tpl: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  rep: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
line: "SmolLM3 tool branch: ns.xml_tool_string ~ (tool | string)" {class: tpl}
results: "The two engines" {
  label.near: top-center
  grid-rows: 1
  py: "CPython Jinja2:\nwrites the dict's Python repr" {class: ok}
  js: "@huggingface/jinja:\nUnknown ObjectValue filter: string" {class: bad}
}
report: "ParityReport.error" {class: rep}
line -> results: "tools passed"
results -> report: "EngineError"

Create parity_loud.py. It renders Part 5's exchange-rate question without and with the tool, then prints the tool line that the reference engine wrote:

# parity_loud.pyimport textwrapfrom families import PARITY_MODELSfrom rates import QUESTION, get_exchange_ratefrom templatestudio.parity import CPythonJinja, NodeJinja, chat_variables, compare_enginesfrom templatestudio.render import load_tokenizerfrom templatestudio.tools import tool_schematokenizer = load_tokenizer(PARITY_MODELS["smollm3"])template = tokenizer.chat_templateengines = [CPythonJinja(), NodeJinja()]tools = [tool_schema(get_exchange_rate)][plain] = compare_engines(template, chat_variables(tokenizer, QUESTION), engines)print(f"without tools: identical={plain.identical} error={plain.error}")with_tools = chat_variables(tokenizer, QUESTION, tools=tools)[loud] = compare_engines(template, with_tools, engines)print(f"with tools: identical={loud.identical} error={loud.error}")line = next(row for row in loud.expected.splitlines() if "get_exchange_rate" in row)print("cpython-jinja2 wrote this tool line:")print(textwrap.fill(line, 72))

Run it:

python parity_loud.py

Expected output:

without tools: identical=True error=Nonewith tools: identical=False error=Error: Unknown ObjectValue filter: stringcpython-jinja2 wrote this tool line:{'type': 'function', 'function': {'name': 'get_exchange_rate','description': 'Look up how many units of one currency one unit ofanother buys.', 'parameters': {'type': 'object', 'properties': {'base':{'type': 'string', 'description': 'ISO 4217 code of the currency youhold, such as "EUR".'}, 'quote': {'type': 'string', 'description': 'ISO4217 code of the currency you want, such as "JPY".'}}, 'required':['base', 'quote']}, 'return': {'type': 'number'}}}

What just happened: without tools, both engines agree. With tools, the JavaScript port fails, and the report carries its exact error instead of raising it. The last lines show what CPython Jinja2 put in the prompt: a Python repr with single quotes, which is what SmolLM3 sees in Transformers. The crash is the loud face, and it does not stay hidden. A server running the JavaScript port returns an error on the first request with tools, so you find it at once. Running this harness in CI finds it before that first request, and step 9 gives the harness the exit status a CI job needs.

Step 7: Measure which Jinja constructs work on both engines

Goal: run one-line snippets through both engines and sort each construct into same, differs or fails.

Why this step: steps 5 and 6 each found one difference inside a long template. To know what to look for in the next template, you need the differences one construct at a time. A battery of small snippets gives you that. Each line tests one thing a real template does, against variables chosen to expose a difference: a float, a boolean, None, an emoji, and a dict with a key that looks like a number.

grid-rows: 1
grid-gap: 24
classes: {
  same: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  diff: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  fail: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
same: "same (6)" {
  label.near: top-center
  grid-columns: 1
  a: "namespace, selectattr,\nmap attribute, loop.last" {class: same}
  b: "string methods,\ntojson on a string" {class: same}
}
differs: "differs (8)" {
  label.near: top-center
  grid-columns: 1
  a: "floats: 1.0 -> 1" {class: diff}
  b: "True -> true, None -> empty" {class: diff}
  c: "dicts and lists print as JSON,\nnumeric keys move first" {class: diff}
  d: "emoji length 2" {class: diff}
}
fails: "fails (9)" {
  label.near: top-center
  grid-columns: 1
  a: "string on a dict,\ntrim on undefined" {class: fail}
  b: "reject, select, groupby,\nformat, %, round, wordcount" {class: fail}
}

Create constructs.py with the variables and the snippets:

# constructs.py# One-line Jinja snippets, each testing one construct real chat templates use.VARIABLES = {    "tool": {"name": "convert", "strict": True},    "args": {"amount": 1.0, "base": "EUR"},    "codes": {"b": "JPY", "10": "EUR"},    "words": ["user", "tool"],    "messages": [{"role": "user"}, {"role": "tool"}, {"role": "user"}],    "text": "café 😀",    "flag": True,    "nothing": None,}CONSTRUCTS = {    "namespace": (        "{% set ns = namespace(n=0) %}"        "{% for m in messages %}{% set ns.n = ns.n + 1 %}{% endfor %}{{ ns.n }}"    ),    "selectattr": (        "{{ messages | selectattr('role', 'equalto', 'tool') | list | length }}"    ),    "map attribute": "{{ messages | map(attribute='role') | join(',') }}",    "loop.last": (        "{% for w in words %}{{ w }}{% if not loop.last %}+{% endif %}{% endfor %}"    ),    "string method": "{{ text.upper() }}|{{ 'a b'.split(' ') | join('-') }}",    "tojson string": "{{ text | tojson }}",    "tojson float": "{{ args | tojson }}",    "print float": "{{ args.amount }}",    "print bool": "{{ flag }}",    "print none": "[{{ nothing }}]",    "print dict": "{{ tool }}",    "dict order": "{% for k in codes %}{{ k }},{% endfor %}",    "length": "{{ text | length }}",    "string on list": "{{ words | string }}",    "string on dict": "{{ tool | string }}",    "trim undefined": "[{{ missing | trim }}]",    "reject": "{{ words | reject('equalto', 'tool') | list }}",    "groupby": "{% for g in messages | groupby('role') %}{{ g.grouper }}{% endfor %}",    "format": "{{ '%s=%s' | format('base', args.base) }}",    "% operator": "{{ '%s!' % args.base }}",    "round": "{{ args.amount | round }}",    "select": "{{ words | select('equalto', 'tool') | list }}",    "wordcount": "{{ text | wordcount }}",}

Add the battery runner to the end of templatestudio/parity.py. It reuses compare_engines with each snippet as the template:

# templatestudio/parity.py (add at the end)def probe_constructs(    constructs: dict[str, str],    variables: dict[str, object],    renderers: list[EngineRenderer],) -> dict[str, list[ParityReport]]:    """Run each one-line snippet on every engine; compare each with the first."""    results: dict[str, list[ParityReport]] = {}    for name, snippet in constructs.items():        try:            results[name] = compare_engines(snippet, variables, renderers)        except EngineError as error:  # the reference engine itself refused            results[name] = [                ParityReport(renderers[0].name, other.name, False, None, f"ref: {error}")                for other in renderers[1:]            ]    return results

Create parity_battery.py:

# parity_battery.pyimport sysfrom constructs import CONSTRUCTS, VARIABLESfrom templatestudio.parity import CPythonJinja, NodeJinja, probe_constructssys.stdout.reconfigure(encoding="utf-8")results = probe_constructs(CONSTRUCTS, VARIABLES, [CPythonJinja(), NodeJinja()])for name, [report] in results.items():  # one report: two engines    if report.identical:        print(f"{name:15} same     {report.expected}")    elif report.error:        print(f"{name:15} FAILS")        print(f"    {report.error}")    else:        print(f"{name:15} DIFFERS  {report.expected}")        print(f"{'':21}vs  {report.actual}")

Run it:

python parity_battery.py

Expected output:

namespace       same     3selectattr      same     1map attribute   same     user,tool,userloop.last       same     user+toolstring method   same     CAFÉ 😀|a-btojson string   same     "café 😀"tojson float    DIFFERS  {"amount": 1.0, "base": "EUR"}                     vs  {"amount": 1, "base": "EUR"}print float     DIFFERS  1.0                     vs  1print bool      DIFFERS  True                     vs  trueprint none      DIFFERS  [None]                     vs  []print dict      DIFFERS  {'name': 'convert', 'strict': True}                     vs  {"name": "convert", "strict": true}dict order      DIFFERS  b,10,                     vs  10,b,length          DIFFERS  6                     vs  7string on list  DIFFERS  ['user', 'tool']                     vs  ["user", "tool"]string on dict  FAILS    Error: Unknown ObjectValue filter: stringtrim undefined  FAILS    Error: Cannot apply filter "trim" to type: UndefinedValuereject          FAILS    Error: Unknown ArrayValue filter: rejectgroupby         FAILS    Error: Unknown ArrayValue filter: groupbyformat          FAILS    Error: Unknown StringValue filter: format% operator      FAILS    SyntaxError: Unknown operator "%" between StringValue and StringValueround           FAILS    Error: Unknown NumericValue filter: roundselect          FAILS    Error: Unknown ArrayValue filter: selectwordcount       FAILS    Error: Unknown StringValue filter: wordcount

What just happened: you have a measured table for this pair of engines at these versions. The six constructs at the top are the ones that make up most real templates, and they agree.

The eight DIFFERS rows are the silent face. Each engine prints a value the way its language does. True becomes true, None becomes nothing, 1.0 becomes 1, and the key "10" jumps in front of "b" because JavaScript orders integer-like keys first. The length row differs for a related reason: JavaScript strings count UTF-16 code units, and an emoji takes two of them. The nine FAILS rows are the loud face. One of them is the trim on a missing value that Llama 3's template does when an assistant turn has no content; CPython Jinja2 renders that row as [].

This table is true for @huggingface/jinja 0.5.12. Run it again whenever you change either engine's version.

Step 8: Lint a chat template for constructs that do not port

Goal: scan a template's source, line by line, for the constructs the battery measured as failing.

Why this step: the diff needs every engine installed and a conversation that reaches the risky line. A lint needs neither. It reads the template as text, so it runs in any CI job before anything renders, and it flags lines that no test conversation reaches yet. Its rules come from step 7's measurements, not from a list of engine features in someone's documentation. Three facts decide which rows become rules:

• Eight of the nine FAILS rows become seven rules. reject and select share one.
• The ninth, trim on a missing value, cannot become a rule. Whether the value is missing depends on the conversation, not on the text of the line.
• No DIFFERS row becomes a rule except string on list, which the string rule already covers. The others depend on the data too.

direction: down
classes: {
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  fn: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  out: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}
battery: "step 7: measured rows" {class: data}
rules: "PORTABILITY_RULES\n(one regex per construct)" {class: data}
blank: "each line, with quoted text\nreplaced by \"\"" {class: fn}
lint: "lint_portability(template)" {class: fn}
hits: "line 53: string filter ..." {class: out}
battery -> rules: "1. written from"
rules -> lint: "2. applied by"
blank -> lint: "3. matched against"
lint -> hits: "4. reports"

Add the rules and the lint to the end of templatestudio/parity.py:

# templatestudio/parity.py (add at the end)# Constructs that step 7 measured as failing on @huggingface/jinja 0.5.12, or# printing differently there. Measure again when you change either engine.PORTABILITY_RULES: dict[str, str] = {    r"\|\s*string\b": "string filter: fails on a dict, prints a list as JSON",    r"\|\s*(?:reject|select)\s*\(": "reject/select: missing (use a for loop)",    r"\|\s*groupby\b": "groupby: missing",    r"\|\s*format\b": "format filter: missing",    r"['\"]\s+%\s+\S": "string % formatting: missing",    r"\|\s*round\b": "round: missing",    r"\|\s*wordcount\b": "wordcount: missing",}STRING_LITERAL = re.compile(r"'[^']*'|\"[^\"]*\"")def lint_portability(template_source: str) -> list[str]:    """Flag each line that uses a construct the JS port lacks or renders differently."""    problems: list[str] = []    for number, line in enumerate(template_source.splitlines(), start=1):        code = STRING_LITERAL.sub('""', line)  # text inside quotes is not code        for pattern, reason in PORTABILITY_RULES.items():            if re.search(pattern, code):                problems.append(f"line {number}: {reason}")    return problems

STRING_LITERAL replaces every quoted string with "" before the rules run. Without it, gpt-oss's template trips the string rule on line 173, a line of documentation text inside quotes that reads number | string. Blanking literals keeps the quotes, so a real '...' % value still matches. The % rule also needs a space on both sides of %, which is how Python's string formatting is written in templates. A format string such as strftime_now("%d %b %Y") puts % inside the quotes and never matches.

Create parity_lint.py. It lints SmolLM3, Qwen3, your Part 8 template, and BOOK_EXAMPLE, a filter line that the book's chapter 9 describes crashing an older llama.cpp build. It is in the test because the JavaScript port has no reject filter either:

# parity_lint.pyfrom families import PARITY_MODELSfrom templatestudio.authoring import SUPPORT_TEMPLATEfrom templatestudio.parity import lint_portabilityfrom templatestudio.render import load_tokenizer# A filter line from the book's chapter 9: an older llama.cpp build answered it# with HTTP 500, and the JavaScript port has no reject filter either.BOOK_EXAMPLE = '{%- for key in param_fields.keys() | reject("in", handled_keys) %}'sources = {    family: load_tokenizer(PARITY_MODELS[family]).chat_template    for family in ("smollm3", "qwen3")}sources["part8"] = SUPPORT_TEMPLATEsources["book"] = BOOK_EXAMPLEfor name, source in sources.items():    problems = lint_portability(source)    print(f"{name}: {len(problems)} problem(s)")    for problem in problems:        print("  ", problem)

Run it:

python parity_lint.py

Expected output:

smollm3: 2 problem(s)   line 53: string filter: fails on a dict, prints a list as JSON   line 61: string filter: fails on a dict, prints a list as JSONqwen3: 0 problem(s)part8: 0 problem(s)book: 1 problem(s)   line 1: reject/select: missing (use a for loop)

What just happened: the lint found SmolLM3's two (tool | string) lines without rendering anything. It passed Qwen3's template and your Part 8 template clean, and it flagged the book's reject("in", ...). Each hit names a line number, so the fix is a targeted edit, usually replacing the risky filter with an explicit for loop that every engine supports. Check any fix with the diff from step 4. The reference engine's output must not change, or you have changed what the model sees.

Step 9: Test every template you ship on every engine

Goal: run the lint and the diff together across every template and three conversations, and read what each one catches.

Why this step: neither check is enough on its own, and the matrix shows why. The lint is cheap and finds the crash before any conversation reaches it. The diff finds what the lint cannot see: the silent face, and crashes that depend on the data. So run both, and run the diff with conversations that reach the template's tool branch, not only its plain chat. The three columns are chosen for that. chat exercises the message loop every template has, tools reaches the branch that writes tool schemas into the prompt, and float call reaches the branch that writes a past tool call with a number in it. A both fail cell is not counted as a problem, because both engines agree that the template refuses that input.

grid-rows: 3
grid-columns: 2
grid-gap: 26
classes: {
  q: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  a: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
q1: "1. Lint every template in CI\n(no engine needed)" {class: q}
a1: "finds: SmolLM3 (tool | string)" {class: a}
q2: "2. Diff on each engine you ship\n(chat, tools, tool calls)" {class: q}
a2: "finds: Llama 3 trim, every 1.0 vs 1" {class: a}
q3: "3. Re-run when an engine\nor a template changes" {class: q}
a3: "finds: what a version bump moved" {class: a}
q1 -> q2
q2 -> q3

Create parity_matrix.py. verdict turns each comparison into one short cell: same, @ plus the first divergence, js fails, both fail, or ref fails. When the reference engine refuses a conversation, compare_engines raises before the JavaScript port runs. In that case verdict renders on the port alone. If the port raises too, the cell is both fail; if it renders, the cell is ref fails. The cell names assume exactly two engines. The last row is your Part 8 template, loaded with Part 8's authored_tokenizer. The script counts lint hits plus every cell that is not same or both fail, and exits with status 1 when the count is not zero, so CI fails on it:

# parity_matrix.pyimport sysfrom amounts import ARGUMENTS, convert, convert_loopfrom families import FAMILIES, PARITY_MODELSfrom naive import CONVERSATIONfrom rates import QUESTION, get_exchange_ratefrom templatestudio.authoring import authored_tokenizerfrom templatestudio.parity import (    CPythonJinja,    EngineError,    NodeJinja,    chat_variables,    compare_engines,    lint_portability,)from templatestudio.render import load_tokenizerfrom templatestudio.tools import normalize_arguments, tool_schemaengines = [CPythonJinja(), NodeJinja()]CASES = {    "chat": (CONVERSATION, None),    "tools": (QUESTION, [tool_schema(get_exchange_rate)]),    "float call": (convert_loop(normalize_arguments(ARGUMENTS)), [tool_schema(convert)]),}tokenizers = {    family: load_tokenizer(model_id) for family, model_id in PARITY_MODELS.items()}tokenizers["part8"] = authored_tokenizer(FAMILIES["qwen3"])  # your Part 8 templatedef verdict(template: str, variables: dict[str, object]) -> str:    """One short cell for a matrix of exactly two engines: reference, then JS."""    try:        [report] = compare_engines(template, variables, engines)    except EngineError:  # the reference refused, so render on the JS port alone        try:            engines[1].render(template, variables)        except EngineError:            return "both fail"        return "ref fails"    if report.error:        return "js fails"    return "same" if report.identical else f"@{report.first_divergence}"problems = 0print(f"{'family':9} {'lint':5}" + "".join(f"{case:11}" for case in CASES))for family, tokenizer in tokenizers.items():    template = tokenizer.chat_template    cells = [        verdict(template, chat_variables(tokenizer, conversation, tools=tools))        for conversation, tools in CASES.values()    ]    lint = len(lint_portability(template))    problems += lint + sum(cell not in ("same", "both fail") for cell in cells)    print(f"{family:9} {lint:<5}" + "".join(f"{cell:11}" for cell in cells))print(f"{problems} problem(s)")sys.exit(1 if problems else 0)

Run it:

python parity_matrix.py

Expected output:

family    lint chat       tools      float call qwen3     0    same       same       @1073      llama3    0    same       same       js fails   gemma3    0    same       same       both fail  gemma4    0    same       same       @639       llama3.2  0    same       same       @1547      smollm3   2    same       js fails   js fails   gpt-oss   0    same       same       @864       part8     0    same       same       both fail  9 problem(s)

Check the exit status straight after it. In bash:

echo $?

Expected output, which is what makes a CI job fail on this table:

1

In PowerShell, the same check is $LASTEXITCODE.

What just happened: this is the table from the top of the page, and now you can read every cell. The lint column finds SmolLM3 and nothing else. The diff finds four silent divergences at four positions, plus a Llama 3 crash that has no line to lint: its template applies trim to content, and an assistant turn that only calls a tool has none.

Gemma 3 reads both fail because its template raises its own error on both engines. The template refuses the conversation, and that is not a parity gap. Your Part 8 template reads both fail for a similar reason. It has no tool-call branch, so an assistant turn with no content fails on both engines, and Part 8 left the tool path as a next step; this row is what that gap looks like on two engines. On chat and tools it ports cleanly, {% generation %} tag included. The count on the last line is the 2 lint hits plus the 7 problem cells. Only chat is clean everywhere, which is why testing only the plain chat proves very little.

When it breaks: chat template errors on Transformers.js, LM Studio and llama.cpp

Two older errors from the port are fixed in 0.5.12, so you will not see them: Unknown StringValue filter: safe and Unknown test: sequence. If a battery row in step 7 differs from this page, run python parity_versions.py first. A different package version is the usual reason.

How the cross-engine test fits together: Python harness and Node engine

The project now has two processes. Python owns everything that decides what to test and what the result means. Node owns exactly one thing: running the JavaScript port. The only line between them is JSON over a pipe:

direction: down
classes: {
  mod: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  old: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  node: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
earlier: "Earlier parts\nrender.py: first_divergence (Part 1)\njinja_env.py: build_template_env (Part 3)\ntools.py: normalize_arguments (Part 5)" {class: old}
data: "Data and scripts\nfamilies.PARITY_MODELS\namounts.py, constructs.py\nparity_*.py" {class: data}
parity: "templatestudio/parity.py" {
  label.near: top-center
  grid-rows: 2
  grid-columns: 3
  cvars: "chat_variables" {class: mod}
  cpy: "CPythonJinja" {class: mod}
  njs: "NodeJinja" {class: mod}
  cmp: "compare_engines\nParityReport" {class: mod}
  bat: "probe_constructs" {class: mod}
  lint: "lint_portability\nPORTABILITY_RULES" {class: mod}
}
nodeproc: "Node process" {
  label.near: top-center
  grid-rows: 1
  mjs: "engines/hf_jinja.mjs" {class: node}
  pkg: "node_modules/@huggingface/jinja\n0.5.12" {class: node}
}
earlier -> parity: "imported by"
data -> parity: "drive"
parity -> nodeproc: "JSON on stdin and stdout"

compare_engines never knows which engine it is talking to, so a third engine is one more class with a name and a render. The step 7 table and the lint rules are measurements of one engine at one version. That is why they live in data and constants you can re-measure, not in logic.

The complete code: parity.py and the Node engine script

The project tree after this part, with the earlier parts' scripts left out:

chat-templates-part1/├── engines/│   └── hf_jinja.mjs├── node_modules/            (created by npm install)├── package.json             (created by npm init -y)├── package-lock.json        (created by npm install)├── templatestudio/│   ├── __init__.py│   ├── authoring.py         (Part 8)│   ├── jinja_env.py         (Part 3)│   ├── messages.py          (Part 2)│   ├── parity.py            (new)│   ├── render.py            (Part 1)│   └── tools.py             (Part 5)├── amounts.py               (new)├── constructs.py            (new)├── families.py              (PARITY_MODELS added)├── naive.py                 (Part 1)├── rates.py                 (Part 5)├── parity_versions.py       (new)├── parity_js_engine.py      (new)├── parity_reference.py      (new)├── parity_node.py           (new)├── parity_families.py       (new)├── parity_float.py          (new)├── parity_loud.py           (new)├── parity_battery.py        (new)├── parity_lint.py           (new)└── parity_matrix.py         (new)

package.json holds the defaults npm init -y wrote, plus the one line this part depends on:

"dependencies": {  "@huggingface/jinja": "0.5.12"}

engines/hf_jinja.mjs:

// engines/hf_jinja.mjs// Render one chat template with @huggingface/jinja, the JavaScript Jinja port.// Reads {"template": ..., "variables": {...}} as JSON on stdin and writes// {"text": ...} or {"error": ...} as JSON on stdout.import { readFileSync } from "node:fs";import { Template } from "@huggingface/jinja";const job = JSON.parse(readFileSync(0, "utf8"));let reply;try {  reply = { text: new Template(job.template).render(job.variables) };} catch (error) {  reply = { error: `${error.name}: ${error.message}` };}process.stdout.write(JSON.stringify(reply));

The complete templatestudio/parity.py:

# templatestudio/parity.pyfrom __future__ import annotationsimport jsonimport reimport subprocessfrom dataclasses import dataclass, fieldfrom pathlib import Pathfrom typing import Protocolfrom jinja2 import TemplateErrorfrom transformers import PreTrainedTokenizerBasefrom templatestudio.jinja_env import alias_reasoning, build_template_envfrom templatestudio.messages import Conversation, assert_validfrom templatestudio.render import first_divergenceclass EngineError(RuntimeError):    """An engine could not render the template."""class EngineRenderer(Protocol):    """One adapter per Jinja engine: same template, same variables, a string back."""    name: str    def render(self, template_source: str, variables: dict[str, object]) -> str: ...def chat_variables(    tokenizer: PreTrainedTokenizerBase,    conversation: Conversation,    *,    add_generation_prompt: bool = True,    tools: list[dict] | None = None,    **template_kwargs: object,) -> dict[str, object]:    """Build the variables apply_chat_template passes a template, once for all engines."""    assert_valid(conversation)    return {        **tokenizer.special_tokens_map,        "messages": alias_reasoning(conversation),        "tools": tools,        "documents": None,        "add_generation_prompt": add_generation_prompt,        **template_kwargs,    }class CPythonJinja:    """The reference: CPython Jinja2, configured like Transformers 5.22.1."""    name = "cpython-jinja2"    def render(self, template_source: str, variables: dict[str, object]) -> str:        try:            template = build_template_env().from_string(template_source)            return template.render(**variables)        except TemplateError as error:            raise EngineError(f"{type(error).__name__}: {error}") from errorclass NodeJinja:    """@huggingface/jinja, the JavaScript port, run by Node in a subprocess."""    name = "huggingface-jinja"    def __init__(self, script: Path = Path("engines/hf_jinja.mjs")) -> None:        self.script = script    def render(self, template_source: str, variables: dict[str, object]) -> str:        job = json.dumps({"template": template_source, "variables": variables})        result = subprocess.run(            ["node", str(self.script)],            input=job,            capture_output=True,            text=True,            encoding="utf-8",            check=True,        )        reply = json.loads(result.stdout)        if "error" in reply:            raise EngineError(reply["error"])        return reply["text"]@dataclassclass ParityReport:    """How one engine's render compares with the reference engine's."""    reference: str    other: str    identical: bool    first_divergence: int | None  # character index, or None when identical    error: str | None = None  # set when the other engine could not render    expected: str = field(default="", repr=False)    actual: str = field(default="", repr=False)def compare_engines(    template_source: str,    variables: dict[str, object],    renderers: list[EngineRenderer],) -> list[ParityReport]:    """Render once per engine and diff every engine against the first one."""    reference, *others = renderers    expected = reference.render(template_source, variables)    reports: list[ParityReport] = []    for engine in others:        try:            actual = engine.render(template_source, variables)        except EngineError as error:            reports.append(                ParityReport(                    reference.name, engine.name, False, None, str(error), expected                )            )            continue        index = first_divergence(expected, actual)        reports.append(            ParityReport(                reference.name, engine.name, index is None, index, None, expected, actual            )        )    return reportsdef probe_constructs(    constructs: dict[str, str],    variables: dict[str, object],    renderers: list[EngineRenderer],) -> dict[str, list[ParityReport]]:    """Run each one-line snippet on every engine; compare each with the first."""    results: dict[str, list[ParityReport]] = {}    for name, snippet in constructs.items():        try:            results[name] = compare_engines(snippet, variables, renderers)        except EngineError as error:  # the reference engine itself refused            results[name] = [                ParityReport(renderers[0].name, other.name, False, None, f"ref: {error}")                for other in renderers[1:]            ]    return results# Constructs that step 7 measured as failing on @huggingface/jinja 0.5.12, or# printing differently there. Measure again when you change either engine.PORTABILITY_RULES: dict[str, str] = {    r"\|\s*string\b": "string filter: fails on a dict, prints a list as JSON",    r"\|\s*(?:reject|select)\s*\(": "reject/select: missing (use a for loop)",    r"\|\s*groupby\b": "groupby: missing",    r"\|\s*format\b": "format filter: missing",    r"['\"]\s+%\s+\S": "string % formatting: missing",    r"\|\s*round\b": "round: missing",    r"\|\s*wordcount\b": "wordcount: missing",}STRING_LITERAL = re.compile(r"'[^']*'|\"[^\"]*\"")def lint_portability(template_source: str) -> list[str]:    """Flag each line that uses a construct the JS port lacks or renders differently."""    problems: list[str] = []    for number, line in enumerate(template_source.splitlines(), start=1):        code = STRING_LITERAL.sub('""', line)  # text inside quotes is not code        for pattern, reason in PORTABILITY_RULES.items():            if re.search(pattern, code):                problems.append(f"line {number}: {reason}")    return problems

amounts.py and constructs.py are listed in full in steps 5 and 7, and the scripts in the steps that create them.

Where to go next: add llama.cpp or MiniJinja as a third engine

• Add a llama.cpp engine. llama-server's POST /apply-template returns the rendered prompt in a prompt field without running inference (server README). It needs a model file loaded, because llama-server reads the template from the GGUF or from --chat-template-file. A LlamaServerJinja class that posts messages there and returns prompt works with compare_engines without changes. Be aware that the server applies its own changes to a request before the template sees it. A later part of this series follows the request into the server.
• Add a third engine from pip. minijinja 3.1.0 is a Python binding of the Rust MiniJinja engine (PyPI). Wrap it in an EngineRenderer, add it to the engines list, and run steps 7 and 9 again. Its battery will not match this one, which is the point.
• Lock the reference output. Save each report.expected as a file, and fail CI when any engine's output stops matching it byte for byte. That is the golden-file method minja's maintainers use. Chapter 9 of The Chat Templates Handbook builds it out and adds the output side, because tool-call parsers and reasoning fields differ across vLLM, SGLang and llama.cpp too. The companion repository's v1.2/ch09-cross-engine tag holds the chapter's own parity.py. A later part of this series turns golden files into golden-token tests.

References

• @huggingface/jinja 0.5.12 on npm and its source, packages/jinja in huggingface.js: Template, the filters and tests used in step 7.
• Transformers.js apply_chat_template: how Transformers.js renders a template with @huggingface/jinja.
• Transformers 5.22.1 chat_template_utils.py: the variables and the environment the reference engine mirrors. Transformers 5.22.2 release notes: the Jinja sandbox hardening (PR #50431).
• Jinja 3.1.6 documentation: the built-in filters and tests of CPython Jinja2.
• Node.js release schedule: Node.js 20 end of life on 2026-04-30; Node.js 26 long-term support.
• MDN: JSON.parse and MDN: Number: one number type, so 1.0 parses to 1.
• npm folders and npm install --save-exact: why npm walks up to a parent folder, and the exact pin.
• MDN: Object.keys: why integer-like keys come first in JavaScript.
• llama.cpp PR #18462: llama.cpp's own Jinja engine, inspired by the JavaScript port; issue #18852 and issue #30781: the errors in When it breaks.
• LM Studio issue #626: Unknown ArrayValue filter: trim on the JavaScript port.
• vLLM renderers/hf.py and SGLang serving_chat.py: how two Python servers call apply_chat_template, and SGLang parsing string arguments into dicts.
• llama.cpp server README: POST /apply-template.
• template-studio v1.2/ch09-cross-engine: the book's companion code for chapter 9.
• minja: the C++ engine whose golden-file tests the book describes.
• minijinja on PyPI: a Python binding of MiniJinja.
• Models: Qwen/Qwen3-0.6B, NousResearch/Meta-Llama-3-8B-Instruct, unsloth/gemma-3-1b-it, google/gemma-4-E2B-it, unsloth/Llama-3.2-1B-Instruct, HuggingFaceTB/SmolLM3-3B, openai/gpt-oss-20b.
• The Chat Templates Handbook, chapter 9, "The Cross-Engine Problem": the parity gap, the golden-file method and output-path parity.

Related Articles

AI Engineering

• How to Detect Chat Template Backdoors and Forged Tokens

Llms

• How to Catch Chat Template Regressions with Golden-Token Tests
• How to Check the Token IDs llama-server Sends the Model
• How to Gate Chat Template Changes in CI with One Audit