What you'll build: an image-aware chat template renderer
To pass images to apply_chat_template, you render the conversation through a Hugging Face processor, and a vision-language model reads each image only where the chat template put a placeholder for it. Given five image conversations on Qwen3-VL, the function you build, safe_expand, expands the two good ones into image tokens and refuses the three broken ones before the processor runs:
typed, attached image: ok, 196 image tokensOpenAI image_url: ok, 196 image tokensmisspelled type: InvalidConversation: message 0: content[0]: unknown part type 'Image'bare-string image_url: InvalidConversation: message 0: content[0]: image_url part needs image_url.urlbare part, no image: MediaMismatch: 1 media part(s), 0 media item(s) suppliedcaller's request unchanged: image_urlBy the end of this intermediate tutorial, safe_expand will turn an image-and-text conversation into the token IDs and pixel values a model receives. It refuses the conversation instead when an image part is malformed or when the placeholders and images do not pair up. The function lives in a new module, templatestudio/multimodal.py, which also renders typed content through a Hugging Face processor, rewrites OpenAI-style image_url parts on a copy, and counts what each image costs in tokens. Part 2's validate_conversation learns to reject image parts that would otherwise be dropped. The tutorial is for engineers who send images to a vision-language model, such as Qwen2.5-VL, Qwen3-VL or Gemma, through Transformers or through a server that uses its processor.
Verified against Python 3.12.2, transformers==5.22.1, tokenizers==0.23.3, huggingface-hub==2.4.0, jinja2==3.1.6, pillow==13.0.0, torch==2.15.0+cpu and torchvision==0.29.2+cpu on 2026-11-10.
You need the check because processors do not agree on it. This table, from step 8, covers four model families. Each column gives the number of placeholders in the prompt and the number of images handed to the processor:
family 1 ph / 1 img 1 ph / 2 img 0 ph / 1 img 1 ph / 0 img 2 ph / 1 imgqwen2.5-vl no error no error no error no error StopIterationqwen3-vl no error no error no error no error StopIterationgemma3 no error ValueError ValueError no error ValueErrorgemma4 no error ValueError ValueError ValueError ValueErrorThe Qwen-VL processors raise nothing for three of the four mismatches, while Gemma 3 and Gemma 4 raise a ValueError for most of them. A silent mismatch shows up only later, in the model's answer or in a check inside the model's forward pass, after your request handler has returned.
Here is the failure, as a composite. No public post-mortem matches it exactly. A support assistant takes a photo of a damaged product and a question. The client builds the user message with a plain string as its content and sends the photo beside it. The model answers about the question alone, as if no image was sent. Nothing failed on the way in: the upload worked and the processor accepted the photo. But the template had no image part to write a placeholder for, so the processor had nowhere to put the pixels. Step 2 reproduces the cause, without model weights.
This is Part 7 of a series and continues the project from Part 6, How to Pass reasoning_content Back Through a Chat Template. It takes about 30 minutes. It downloads about 55 MB of processor files for three new models, measured in the Hugging Face cache, and no model weights. If you have no PyTorch yet, the CPU wheel adds about 127 MB on Windows or 210 MB on Linux. It needs no API key, no account and no Hugging Face token.
Prerequisites: transformers 5.22.1, Pillow, torchvision and the Part 6 project
You need the project from Parts 1 to 6: the chat-templates-part1/ folder with its virtual environment, the templatestudio/ package with messages.py and render.py, and Part 6's families.py and samples.py. This part adds one module, templatestudio/multimodal.py, and fourteen scripts. It replaces one branch of validate_conversation in messages.py, adds one function to the end of that file, and appends a few lines to families.py and samples.py. If you do not have the project, create the folder and the environment with Part 1's Prerequisites, add an empty templatestudio/__init__.py, and copy four files: templatestudio/messages.py, templatestudio/render.py, families.py and samples.py from Part 6's complete code.
Go to the project folder, activate the environment, and turn off the Transformers advisory warnings. Run the cd from the folder that contains chat-templates-part1/:
cd chat-templates-part1source .venv/bin/activateexport TRANSFORMERS_NO_ADVISORY_WARNINGS=1On Windows, activate with .venv\Scripts\activate in PowerShell or source .venv/Scripts/activate in Git Bash, and set the variable in PowerShell with $env:TRANSFORMERS_NO_ADVISORY_WARNINGS = "1". Every script in this part runs from the project root.
The text pins are unchanged from Parts 1 to 6. This part adds three packages: Pillow, which opens the images, and PyTorch and torchvision, which were optional in earlier parts and are now required. The reason is the video processor that the processors of Qwen2.5-VL, Qwen3-VL and Gemma 4 each bundle. In Transformers 5.22.1 those video processors require torchvision, and AutoProcessor cannot load them without it (the troubleshooting table quotes the error). Gemma 3's processor has no video processor and loads without torchvision. Install the new packages inside the activated environment, with PyTorch and torchvision from the CPU wheel index. Run both lines even if you installed PyTorch for Part 1's step 8 or Part 5's step 13: pip keeps an existing torch 2.15.0+cpu and adds torchvision beside it.
pip install "pillow==13.0.0"pip install "torch==2.15.0" "torchvision==0.29.2" \ --index-url https://download.pytorch.org/whl/cpuIn PowerShell, continue the line with a backtick instead of a backslash:
pip install "pillow==13.0.0"pip install "torch==2.15.0" "torchvision==0.29.2" ` --index-url https://download.pytorch.org/whl/cpuThe series stays on transformers 5.22.1. Version 5.22.2 (2026-11-06) hardens the Jinja sandbox against a CPython bug that is fixed only in Python 3.13 and later (PR #50431). This part renders templates written by model publishers, so do not put the 5.22.1 and Python 3.12 combination behind an endpoint that renders templates from untrusted repositories. Part 12 covers templates as an attack surface.
This part compares four vision-language models. Qwen2.5-VL-3B, Qwen3-VL-2B and the 4B Gemma 3 are new. The Gemma 4 E2B model is the one from Part 4, which already handles images. The gemma3 entry in Part 1's FAMILIES is the 1B model, which is text-only, so the vision list names the 4B model instead. Only the processor files download, never the weights.
A block whose first line ends in (add at the end) goes after the last line of that file, separated from it by two blank lines. In those blocks, and in blocks whose first line ends in (replace ...), the first line only says where the code goes: do not paste it. Add the list to the end of families.py:
# families.py (add at the end)# Part 7: vision-language models. Only their processors download, never weights.VISION_MODELS: dict[str, str] = { "qwen2.5-vl": "Qwen/Qwen2.5-VL-3B-Instruct", "qwen3-vl": "Qwen/Qwen3-VL-2B-Instruct", "gemma3": "unsloth/gemma-3-4b-it", "gemma4": GEMMA4,}Create make_images.py, which writes three test images of fixed sizes. The token cost of an image depends on its size, so the sizes are part of what this tutorial measures:
# make_images.pyfrom pathlib import Pathfrom PIL import ImageSIZES = {"small.png": (448, 448), "wide.png": (1920, 1080), "photo.png": (3000, 4000)}Path("images").mkdir(exist_ok=True)for name, (width, height) in SIZES.items(): Image.new("RGB", (width, height), (180, 70, 60)).save(f"images/{name}") print(f"images/{name:10} {width} x {height}")Create check_versions.py, which prints the four versions that matter in this part:
# check_versions.pyimport PILimport torchimport torchvisionimport transformersprint( transformers.__version__, PIL.__version__, torch.__version__, torchvision.__version__,)Create fetch_vision.py, which downloads each processor once and prints the placeholder its template writes for an image:
# fetch_vision.pyfrom transformers import AutoProcessorfrom families import VISION_MODELSfor model_id in VISION_MODELS.values(): processor = AutoProcessor.from_pretrained(model_id) name = type(processor).__name__ print(f"{model_id:29} {name:20} placeholder {processor.image_token}")Check the install, the images and the processors before you go on:
python check_versions.pypython make_images.pypython fetch_vision.pyExpected output:
5.22.1 13.0.0 2.15.0+cpu 0.29.2+cpuimages/small.png 448 x 448images/wide.png 1920 x 1080images/photo.png 3000 x 4000Qwen/Qwen2.5-VL-3B-Instruct Qwen2_5_VLProcessor placeholder <|image_pad|>Qwen/Qwen3-VL-2B-Instruct Qwen3VLProcessor placeholder <|image_pad|>unsloth/gemma-3-4b-it Gemma3Processor placeholder <start_of_image>google/gemma-4-E2B-it Gemma4Processor placeholder <|image|>The first run can print a warning starting Warning: You are sending unauthenticated requests to the HF Hub. and, on Windows, one starting UserWarning: `huggingface_hub` cache-system uses symlinks by default. Both are harmless.
How processor.apply_chat_template renders an image: one placeholder, then many tokens
A text model's chat template lives on its tokenizer. A vision-language model ships a processor: one object that holds the tokenizer, an image processor and, for most models, a video processor. The Chat Templates Handbook, the book this series accompanies, calls the move from tokenizer to processor the processor boundary. This tutorial uses the term for what 5.22.1 actually does. In Transformers 5.22.1 the boundary falls here: the tokenizer and the processor carry the same template and render the same string, but only the processor can load an image and replace each placeholder with the tokens that hold it. The number of placeholders the template writes must therefore equal the number of images the processor is given. Step 1 checks the first claim, that the tokenizer and the processor render the same string, on four models.
The processor renders a multimodal prompt in two stages:
- Stage 1, the template. Each message's content is an ordered list of typed parts, such as
{"type": "text", ...}and{"type": "image", ...}. The template walks the list and writes one placeholder token for each image part, at the place the part sits. Qwen-VL writes<|image_pad|>, Gemma 3 writes<start_of_image>, and Gemma 4 writes<|image|>. The output is still a string you can read. - Stage 2, the processor. It loads each image, resizes it, and cuts it into patches for the vision encoder. Then it replaces each placeholder with a run of image tokens, one slot for each piece of the image's embedding. The length of the run depends on the model and, for most models, on the image size.
The model sees stage 2. A string comparison of the prompt only ever sees stage 1.
grid-rows: 1
grid-gap: 40
s1: "Stage 1: the template" {
label.near: top-center
style: {fill: "#EAF3FC"; stroke: "#2C6FB0"; font-color: "#2C2C2A"}
grid-columns: 1
parts: "content parts\ntext | image | text" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
tpl: "chat template\none placeholder per image" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
str: "prompt string\n...with<|image_pad|>this..." {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
s2: "Stage 2: the processor" {
label.near: top-center
style: {fill: "#EAF7EE"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
grid-columns: 1
img: "image processor\nresize, patches -> pixel_values" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
exp: "expand each placeholder\ninto N image tokens" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
ids: "input_ids + pixel_values\nwhat the model receives" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
Read each column from top to bottom. The left column is what earlier parts of this series called rendering. The right column is new for multimodal models. The pixels never pass through the template, and its placeholders are only slots. Every check in this tutorial asks whether the two columns agree.
Step 1: Render an image-and-text message with processor.apply_chat_template
Goal: build a user message whose content is a list of typed parts, and render it with the processor's chat template.
Why this step: Part 2's Message type already allowed content to be a list of parts, and its validator checked each part for a type key. But Part 2 rendered list content only once, to show that Qwen3-0.6B's text-only template turns it into an empty string. An image is not an attachment to a string. It is one part in an ordered list, and the order carries meaning: the content list [text, image, text] puts the image between the two text spans, where "this product" refers to it. render_multimodal is stage 1 only, so it never opens the file the part points to.
direction: right
t1: "text\nWhat is wrong with" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
im: "image\nimages/small.png" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
t2: "text\nthis product?" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
t1 -> im -> t2: "order is kept"
The three parts render in this order, and the image part becomes a placeholder between the two text spans.
Create templatestudio/multimodal.py with its first block. load_processor loads the processor for a model. text_part, image_part and user_message build the typed content list. image_part takes a path or URL string, a PIL image, or nothing for a bare placeholder part. render_multimodal calls the processor's apply_chat_template with tokenize=False. The import block is the module's final one, so it also imports names the later steps use, such as BatchFeature and assert_valid:
# templatestudio/multimodal.pyfrom __future__ import annotationsfrom collections.abc import Sequencefrom typing import castfrom PIL import Imagefrom transformers import AutoProcessor, BatchFeature, ProcessorMixinfrom templatestudio.messages import Conversation, Message, assert_validdef load_processor(model_id: str) -> ProcessorMixin: """Load the processor: the tokenizer plus the image and video processors.""" return AutoProcessor.from_pretrained(model_id)def text_part(text: str) -> dict: """A text part of a multimodal content list.""" return {"type": "text", "text": text}def image_part(source: str | Image.Image | None = None) -> dict: """An image part. A string is a URL, a local path or a base64 data URI; a PIL image is used as it is; None leaves a bare placeholder part.""" if source is None: return {"type": "image"} if isinstance(source, Image.Image): return {"type": "image", "image": source} return {"type": "image", "url": source}def user_message(parts: list[dict]) -> Message: """A user turn whose content is an ordered list of typed parts.""" return {"role": "user", "content": parts}def render_multimodal( processor: ProcessorMixin, conversation: Conversation, *, add_generation_prompt: bool = True,) -> str: """Stage 1: render through the processor's template. Images stay placeholders.""" return processor.apply_chat_template( conversation, tokenize=False, add_generation_prompt=add_generation_prompt, )Create show_typed.py. It prints Qwen2.5-VL's render, then compares, on all four families, the processor's render with the render of the processor's own tokenizer through Part 1's render:
# show_typed.pyfrom families import VISION_MODELSfrom templatestudio.multimodal import ( image_part, load_processor, render_multimodal, text_part, user_message,)from templatestudio.render import renderconversation = [ user_message( [ text_part("What is wrong with"), image_part("images/small.png"), text_part("this product?"), ] )]processor = load_processor(VISION_MODELS["qwen2.5-vl"])print(render_multimodal(processor, conversation), end="")print("---")for family, model_id in VISION_MODELS.items(): processor = load_processor(model_id) same = render(processor.tokenizer, conversation) == render_multimodal( processor, conversation ) print(f"{family:12} tokenizer renders the same string: {same}")Run it:
python show_typed.pyExpected output:
<|im_start|>systemYou are a helpful assistant.<|im_end|><|im_start|>userWhat is wrong with<|vision_start|><|image_pad|><|vision_end|>this product?<|im_end|><|im_start|>assistant---qwen2.5-vl tokenizer renders the same string: Trueqwen3-vl tokenizer renders the same string: Truegemma3 tokenizer renders the same string: Truegemma4 tokenizer renders the same string: TrueWhat just happened: the placeholder sits exactly where the image part sat, wrapped in Qwen's <|vision_start|> and <|vision_end|> markers. Qwen2.5-VL also adds a default system prompt when the conversation has none. The last four lines test a claim from the book's chapter 7, that a multimodal template lives only on the processor. In Transformers 5.22.1 that is not true at stage 1, because all four models' tokenizers carry the same template and render the same string. The processor boundary sits at stage 2, where the tokenizer has nothing to load the image with. The rest of the tutorial is about what can go wrong between this string and the images.
Step 2: Reproduce the image the model never sees
Goal: run the opening failure, with a string content and the photo handed to the processor beside it, and count what reaches the model.
Why this step: a processor call takes the text and the images as two separate arguments, and nothing forces them to agree. This step makes them disagree in the most common way and shows what each side of the processor boundary does with that. Once you have seen these numbers, you will recognize the failure later.
grid-rows: 2
grid-columns: 2
grid-gap: 30
p: "prompt string\n0 placeholders" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
i: "photo\nhanded to the processor" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
ids: "input_ids\n0 image tokens" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
pv: "pixel_values\n1,024 rows, no slot" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
p -> ids
i -> pv
The two columns never meet: the text produces no slot, and the pixels have nowhere to go.
Create show_string_content.py:
# show_string_content.pyfrom PIL import Imagefrom families import VISION_MODELSfrom templatestudio.multimodal import load_processor, render_multimodalprocessor = load_processor(VISION_MODELS["qwen2.5-vl"])photo = Image.open("images/small.png")# The bug: the text is a plain string, and the photo travels beside it.conversation = [{"role": "user", "content": "What is wrong with this product?"}]prompt = render_multimodal(processor, conversation)encoded = processor(text=[prompt], images=[photo], return_tensors="pt")print("placeholders in the prompt:", prompt.count(processor.image_token))print("images handed to the processor: 1")print("pixel_values rows:", encoded["pixel_values"].shape[0])ids = encoded["input_ids"][0].tolist()print("image tokens the model receives:", ids.count(processor.image_token_id))Run it:
python show_string_content.pyExpected output:
placeholders in the prompt: 0images handed to the processor: 1pixel_values rows: 1024image tokens the model receives: 0What just happened: the processor accepted the photo and produced 1,024 rows of pixel values for it, which are 1,024 image patches. The prompt had no placeholder, so the model receives zero image tokens and no slot to place those patches in. Nothing raised. The fix is to put the image in the content list, as step 1 did, and step 9 adds the check that catches the mistake when someone forgets. One note on the script: on Qwen, processor.image_token_id is the ID of <|image_pad|>. Step 6 shows a family where that attribute holds a different token.
Step 3: Find which image parts a template drops without an error
Goal: render four spellings of an image part with a url source, plus one misspelled part with an image key, on all four families, and count the placeholders each one gets.
Why this step: an image part that the template does not recognize gives no error. The template walks the content list and writes a placeholder only for the type names it tests for, and it skips every other part. To find out which names those are, use Part 4's differential probing on content parts: change one thing in the input, render, and compare. That shows you the template's real behaviour without reading its Jinja.
grid-rows: 2
grid-columns: 4
grid-gap: 20
a: "type image" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
b: "type Image" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
c: "type picture" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
d: "no type" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
a1: "1 placeholder" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
b1: "0, no error" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
c1: "0, no error" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
d1: "0, no error" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
Each column is one spelling of the same image part, with its source under a url key: the top row is the part, and the bottom row is what one render of it produces.
Create show_dropped_parts.py:
# show_dropped_parts.pyfrom PIL import Imagefrom families import VISION_MODELSfrom templatestudio.multimodal import load_processor, render_multimodal, text_partPARTS = { "type image": {"type": "image", "url": "images/small.png"}, "type Image": {"type": "Image", "url": "images/small.png"}, "Image, image=": {"type": "Image", "image": "images/small.png"}, "type picture": {"type": "picture", "url": "images/small.png"}, "no type": {"url": "images/small.png"},}print(f"{'family':12}" + "".join(f"{label:>14}" for label in PARTS))for family, model_id in VISION_MODELS.items(): processor = load_processor(model_id) row = f"{family:12}" for part in PARTS.values(): conversation = [{"role": "user", "content": [part, text_part("Describe it.")]}] prompt = render_multimodal(processor, conversation) row += f"{prompt.count(processor.image_token):>14}" print(row)# Stage 2 for the part Qwen gave a placeholder: does the processor fill it?processor = load_processor(VISION_MODELS["qwen2.5-vl"])part = {"type": "Image", "image": Image.open("images/small.png")}conversation = [{"role": "user", "content": [part, text_part("Describe it.")]}]encoded = processor.apply_chat_template(conversation, tokenize=True, return_dict=True)ids = encoded["input_ids"][0]print("qwen2.5-vl, type Image with an image key:")print(" image tokens:", ids.count(processor.image_token_id))print(" pixel_values:", "pixel_values" in encoded)Run it:
python show_dropped_parts.pyExpected output:
family type image type Image Image, image= type picture no typeqwen2.5-vl 1 0 1 0 0qwen3-vl 1 0 1 0 0gemma3 1 0 0 0 0gemma4 1 0 0 0 0qwen2.5-vl, type Image with an image key: image tokens: 1 pixel_values: FalseWhat just happened: with the source under url, only the exact type name image produces a placeholder, on every family. A capital letter, a synonym or a missing type drops the image and keeps the text, so the model answers "Describe it." about nothing. The middle column shows that the families disagree on why. Gemma's templates test the part's type. Qwen's templates also write a placeholder for any part that has an image key, whatever its type. That is not a rescue, because the processor loads images only from parts typed image: the last three lines show one unfilled <|image_pad|> and no pixel values. On every family the image is lost without an error, so switching models does not help. The fix belongs in your code, before the render.
Step 4: Validate image content parts before apply_chat_template
Goal: make Part 2's validate_conversation check every part of a list content, so assert_valid and safe_render reject the dropped parts from step 3.
Why this step: Part 2 built the input contract, the rules a conversation must meet before any template renders it. For list content it checked only that each part is a dict with a type key. That check already catches the "no type" part. The type Image and type picture parts pass it, and the template then drops them. A part needs a known type and the fields that type cannot work without. The check goes in messages.py, not in the new module, so that the input contract stays free of Transformers and image libraries.
direction: down
conv: "conversation" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
vc: "validate_conversation\nroles, tool_calls, content" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
vp: "validate_content_parts\nknown type + its fields" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
out: "problems list\nempty = well-formed" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
conv -> vc
vc -> vp: "list content"
vp -> out
validate_conversation hands each list content to validate_content_parts and adds the message number to every problem it returns.
In templatestudio/messages.py, find the last branch of the content check inside validate_conversation. It starts elif isinstance(content, list): and loops over the parts. Replace that branch, its four lines, with this:
# templatestudio/messages.py (replace the list branch in validate_conversation) elif isinstance(content, list): problems.extend( f"message {i}: {p}" for p in validate_content_parts(content) )Then add the new function at the end of templatestudio/messages.py. It keeps Part 2's "is not a typed part" problem word for word, and adds checks for a part's type and fields. It accepts the OpenAI image_url shape, because step 5 rewrites that shape before rendering:
# templatestudio/messages.py (add at the end)MEDIA_TYPES: frozenset[str] = frozenset({"image", "video", "audio"})def validate_content_parts(content: list[object]) -> list[str]: """Check each part of a list content: a dict with a known type and its fields.""" problems: list[str] = [] for j, part in enumerate(content): if not isinstance(part, dict) or "type" not in part: problems.append(f"content[{j}] is not a typed part") continue kind = part["type"] if kind == "text": if not isinstance(part.get("text"), str): problems.append(f"content[{j}]: text part needs a 'text' string") elif kind == "image_url": source = part.get("image_url") url = source.get("url") if isinstance(source, dict) else None if not isinstance(url, str) or not url: problems.append(f"content[{j}]: image_url part needs image_url.url") elif kind not in MEDIA_TYPES: problems.append(f"content[{j}]: unknown part type {kind!r}") return problemsCreate validate_parts.py. It runs the four url spellings from step 3 and three more through the contract:
# validate_parts.pyfrom templatestudio.messages import validate_conversationPARTS = [ {"type": "image", "url": "images/small.png"}, {"type": "Image", "url": "images/small.png"}, {"type": "picture", "url": "images/small.png"}, {"url": "images/small.png"}, {"type": "text"}, {"type": "image_url", "image_url": {"url": "images/small.png"}}, {"type": "image_url", "image_url": "images/small.png"},]for part in PARTS: conversation = [{"role": "user", "content": [part, {"type": "text", "text": "Hi"}]}] problems = validate_conversation(conversation) print(problems or "ok")Run it:
python validate_parts.pyExpected output:
ok["message 0: content[0]: unknown part type 'Image'"]["message 0: content[0]: unknown part type 'picture'"]['message 0: content[0] is not a typed part']["message 0: content[0]: text part needs a 'text' string"]ok['message 0: content[0]: image_url part needs image_url.url']What just happened: every part that step 3 showed being dropped now produces a problem string, and so does a text part with no text. A correct image_url part passes. One whose image_url is a bare string fails, because the OpenAI specification makes image_url an object with a url field. safe_render and assert_valid both call validate_conversation, so both now reject these conversations with InvalidConversation. MEDIA_TYPES lists the three media types the processor's apply_chat_template collects from content parts in 5.22.1: image, video and audio.
Step 5: Convert OpenAI image_url parts without changing the caller's messages
Goal: see what the processor does to an OpenAI-style image_url part, and write a normalizer that makes the same rewrite on a copy.
Why this step: clients that speak the OpenAI API send images as {"type": "image_url", "image_url": {"url": ...}}, and the templates do not agree on that shape. Gemma's templates write a placeholder only for type image. Qwen's write one for type image or for any part with an image or image_url key.
The processor's apply_chat_template hides the difference by rewriting image_url parts for you, and it does the rewrite by assigning new content into your message dicts. The tokenizer path does no rewrite, and Gemma 3's template ignores image_url. One request therefore renders differently depending on which object renders it, and a render call you expected only to read your request changes it.
grid-rows: 1
grid-gap: 40
proc: "processor.apply_chat_template" {
label.near: top-center
style: {fill: "#FDEDEC"; stroke: "#B03A2E"; font-color: "#2C2C2A"}
grid-columns: 1
a: "caller's message\nimage_url part" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
b: "same message dict,\ncontent replaced" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
a -> b: "in place"
}
norm: "normalize_conversation" {
label.near: top-center
style: {fill: "#EAF7EE"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
grid-columns: 1
c: "caller's message\nimage_url part" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
d: "new copy\ntype image" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
c -> d: "copy"
}
Both sides produce the same rewritten part. Only the right side leaves the caller's message as it was.
First add a request builder to the end of samples.py. It is a function, not a constant, so every call gets a fresh request. Then create show_image_url.py:
# samples.py (add at the end)def openai_image_request() -> list[dict]: """A user turn in the shape an OpenAI-compatible client sends. A function, not a constant, so every caller gets a fresh copy.""" image = {"type": "image_url", "image_url": {"url": "images/small.png"}} text = {"type": "text", "text": "Describe it."} return [{"role": "user", "content": [image, text]}]# show_image_url.pyfrom families import VISION_MODELSfrom samples import openai_image_requestfrom templatestudio.multimodal import load_processor, render_multimodalfrom templatestudio.render import renderprocessor = load_processor(VISION_MODELS["gemma3"])conversation = openai_image_request()print("before:", conversation[0]["content"][0])prompt = render_multimodal(processor, conversation)print("after: ", conversation[0]["content"][0])print("processor placeholders:", prompt.count(processor.image_token))tokenizer_prompt = render(processor.tokenizer, openai_image_request())print("tokenizer placeholders:", tokenizer_prompt.count(processor.image_token))Run it:
python show_image_url.pyExpected output:
before: {'type': 'image_url', 'image_url': {'url': 'images/small.png'}}after: {'type': 'image', 'url': 'images/small.png'}processor placeholders: 1tokenizer placeholders: 0The processor rendered the image, and the after: line shows that your own message changed. Gemma 3's tokenizer, given a fresh copy of the same request, rendered no placeholder at all. Now add the normalizer to the end of templatestudio/multimodal.py:
# templatestudio/multimodal.py (add at the end)def normalize_content_part(part: dict) -> dict: """Rewrite an OpenAI image_url part as {"type": "image", "url": ...}.""" if part.get("type") != "image_url": return part return {"type": "image", "url": part["image_url"]["url"]}def normalize_conversation(conversation: Conversation) -> Conversation: """Return a copy with every image_url part rewritten. The input is not changed.""" normalized: Conversation = [] for message in conversation: content = message.get("content") if isinstance(content, list): parts = [normalize_content_part(part) for part in content] message = cast(Message, {**message, "content": parts}) normalized.append(message) return normalizedCreate show_normalized_image_url.py:
# show_normalized_image_url.pyfrom families import VISION_MODELSfrom samples import openai_image_requestfrom templatestudio.multimodal import load_processor, normalize_conversationfrom templatestudio.render import renderprocessor = load_processor(VISION_MODELS["gemma3"])conversation = openai_image_request()normalized = normalize_conversation(conversation)print("caller's part:", conversation[0]["content"][0])print("normalized: ", normalized[0]["content"][0])prompt = render(processor.tokenizer, normalized)print("tokenizer placeholders:", prompt.count(processor.image_token))Run it:
python show_normalized_image_url.pyExpected output:
caller's part: {'type': 'image_url', 'image_url': {'url': 'images/small.png'}}normalized: {'type': 'image', 'url': 'images/small.png'}tokenizer placeholders: 1What just happened: normalize_conversation built new message dicts and new content lists, so the caller's part is still image_url. The rewritten copy renders a placeholder even through Gemma 3's tokenizer, and the result no longer depends on which object renders it. Keep the order fixed: validate first, then normalize and render. Step 10 builds the production path in that order and shows why.
Step 6: Expand placeholders into image tokens and count them
Goal: run stage 2 with apply_chat_template(..., tokenize=True, return_dict=True) and count how many tokens one image becomes on each family, using the token ID that is right on all four.
Why this step: the string from stage 1 hides an image's real size. The processor replaces one placeholder with a run of image tokens, and that run is what fills the context window, costs prefill compute and takes prefix-cache space. To count it you need the ID of the token that fills the run. On Gemma 3 that is not the placeholder token, and counting the wrong ID returns 1.
direction: right
ph: "1 placeholder\n<|image_pad|>" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
proc: "processor\nimage 448 x 448" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
run: "256 x <|image_pad|>\nin input_ids" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
ph -> proc -> run
On Qwen2.5-VL a 448-pixel square becomes 256 copies of the same token. On Gemma 3 the placeholder is replaced by a different token, repeated 256 times between two markers.
Add the stage 2 helpers to the end of templatestudio/multimodal.py. image_token_id reads processor.image_token_ids[0]. On all four families that attribute is a one-item list holding the token the image run is made of: <image_soft_token> on Gemma 3, and the placeholder token itself on the other three. expand returns a BatchFeature, a dict of tensors holding input_ids, pixel_values and, for Qwen, the patch grid:
# templatestudio/multimodal.py (add at the end)def image_token_id(processor: ProcessorMixin) -> int: """The token each image expands into. On Gemma 3 this is not the placeholder.""" return processor.image_token_ids[0]def expand( processor: ProcessorMixin, conversation: Conversation, *, add_generation_prompt: bool = True,) -> BatchFeature: """Stage 2: load the images and expand every placeholder into image tokens.""" return processor.apply_chat_template( conversation, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=add_generation_prompt, )def count_expanded_placeholders(input_ids: Sequence[int], token_id: int) -> int: """Count one token in a list of IDs: after stage 2, what the images cost.""" return sum(1 for token in input_ids if token == token_id)Create show_expansion.py. It passes the image as a PIL object under the part's image key, which skips the file decoding step:
# show_expansion.pyfrom PIL import Imagefrom families import VISION_MODELSfrom templatestudio.multimodal import ( count_expanded_placeholders, expand, image_part, image_token_id, load_processor, render_multimodal, text_part, user_message,)photo = Image.open("images/small.png")conversation = [user_message([image_part(photo), text_part("Describe it.")])]print(f"{'family':12} {'placeholder':20} {'in string':>9} {'expanded':>9} {'ids':>5}")for family, model_id in VISION_MODELS.items(): processor = load_processor(model_id) placeholder = processor.image_token in_string = render_multimodal(processor, conversation).count(placeholder) ids = expand(processor, conversation)["input_ids"][0].tolist() expanded = count_expanded_placeholders(ids, image_token_id(processor)) print(f"{family:12} {placeholder:20} {in_string:>9} {expanded:>9} {len(ids):>5}")gemma3 = load_processor(VISION_MODELS["gemma3"])ids = expand(gemma3, conversation)["input_ids"][0].tolist()print("gemma3 counted with processor.image_token_id:", count_expanded_placeholders(ids, gemma3.image_token_id))Run it:
python show_expansion.pyExpected output:
family placeholder in string expanded idsqwen2.5-vl <|image_pad|> 1 256 280qwen3-vl <|image_pad|> 1 196 209gemma3 <start_of_image> 1 256 271gemma4 <|image|> 1 256 270gemma3 counted with processor.image_token_id: 1What just happened: every family wrote exactly one placeholder at stage 1, and stage 2 turned it into 196 to 256 tokens. Qwen2.5-VL's 256 is step 2's 1,024 patches after merging: the 448-pixel square is 32 by 32 patches of 14 pixels, and each 2 by 2 block of patches becomes one token. The ids column is the whole prompt, text and image tokens together. The last line is the Gemma 3 trap. There, processor.image_token_id holds the ID of <start_of_image>, the placeholder, while the expanded run is made of <image_soft_token>. Count with image_token_id(processor) instead and the number is right on every family.
Step 7: Count how many tokens an image costs on Qwen-VL and Gemma
Goal: expand the three test images on all four families and compare the token counts.
Why this step: step 6 used one small image. An uploaded phone photo is about 12 megapixels, and whether the processor shrinks it depends on the model's pixel bounds, which each model ships in its preprocessor_config.json. A string test of the prompt cannot see any of this, because the string has one placeholder whatever the image size.
Create compare_image_cost.py:
# compare_image_cost.pyfrom PIL import Imagefrom families import VISION_MODELSfrom templatestudio.multimodal import ( count_expanded_placeholders, expand, image_part, image_token_id, load_processor, text_part, user_message,)IMAGES = ["small.png", "wide.png", "photo.png"]print(f"{'image':20}" + "".join(f"{family:>12}" for family in VISION_MODELS))processors = {family: load_processor(m) for family, m in VISION_MODELS.items()}for name in IMAGES: photo = Image.open(f"images/{name}") conversation = [user_message([image_part(photo), text_part("Describe it.")])] row = f"{name} {photo.width}x{photo.height}" row = f"{row:20}" for processor in processors.values(): ids = expand(processor, conversation)["input_ids"][0].tolist() row += f"{count_expanded_placeholders(ids, image_token_id(processor)):>12,}" print(row)Run it:
python compare_image_cost.pyExpected output:
image qwen2.5-vl qwen3-vl gemma3 gemma4small.png 448x448 256 196 256 256wide.png 1920x1080 2,691 2,040 256 264photo.png 3000x4000 15,301 11,750 256 266What just happened: Gemma 3 resizes every image to 896 by 896 (its preprocessor_config.json sets the size), which is why its column never leaves 256. Gemma 4 E2B keeps the aspect ratio inside a budget of at most 280 tokens, and its cost moves only a little, from 256 to 264 to 266. The Qwen models keep most of the image's resolution, up to their pixel bounds. Qwen2.5-VL-3B ships an upper bound of 12,845,056 pixels, so the phone photo costs 15,301 tokens. Qwen3-VL-2B uses larger 16-pixel patches and needs 11,750 for the same photo. Two requests whose prompt strings are byte-identical can differ by 15,000 tokens of context. A test of a multimodal prompt should therefore assert on expanded token IDs, with pinned image sizes, and not only on the rendered string.
Step 8: Compare what each processor does when placeholders and images do not match
Goal: hand each processor a prompt and an image list whose counts disagree, in four ways, and record whether it raises.
Why this step: before you write a check, find out which processors already check for you and which cases they miss. Step 2 tested one mismatch on one model, which is not enough to know. The answer decides whether your own check is a backup or the only one.
Create compare_mismatch.py. Each case writes some number of bare image parts and passes some number of images. The first case is the matched one:
# compare_mismatch.pyimport textwrapfrom PIL import Imagefrom families import VISION_MODELSfrom templatestudio.multimodal import image_part, load_processor, render_multimodalphoto = Image.open("images/small.png")# (placeholders in the prompt, images handed to the processor)CASES = [(1, 1), (1, 2), (0, 1), (1, 0), (2, 1)]errors: dict[str, str] = {}print(f"{'family':12}" + "".join(f"{f'{p} ph / {n} img':>15}" for p, n in CASES))for family, model_id in VISION_MODELS.items(): processor = load_processor(model_id) row = f"{family:12}" for placeholders, count in CASES: parts = [image_part() for _ in range(placeholders)] parts.append({"type": "text", "text": "Describe it."}) prompt = render_multimodal(processor, [{"role": "user", "content": parts}]) try: processor(text=[prompt], images=[photo] * count or None, return_tensors="pt") outcome = "no error" except (ValueError, StopIteration) as error: outcome = type(error).__name__ errors.setdefault(f"{family}: {str(error) or '(no message)'}", "") row += f"{outcome:>15}" print(row)print()for message in errors: print(textwrap.fill(message, 72, subsequent_indent=" "))Run it:
python compare_mismatch.pyExpected output:
family 1 ph / 1 img 1 ph / 2 img 0 ph / 1 img 1 ph / 0 img 2 ph / 1 imgqwen2.5-vl no error no error no error no error StopIterationqwen3-vl no error no error no error no error StopIterationgemma3 no error ValueError ValueError no error ValueErrorgemma4 no error ValueError ValueError ValueError ValueErrorqwen2.5-vl: (no message)qwen3-vl: (no message)gemma3: Prompt contained 1 image tokens but received 2 images.gemma3: Prompt contained 0 image tokens but received 1 images.gemma3: Prompt contained 2 image tokens but received 1 images.gemma4: The total number of <|image|> tokens in the prompts should be the same as the number of images passed. Found [1] <|image|> tokens and [2] images per sample.gemma4: The total number of <|image|> tokens in the prompts should be the same as the number of images passed. Found [0] <|image|> tokens and [1] images per sample.gemma4: Found 1 <|image|> tokens in the text but no images were passed.gemma4: The total number of <|image|> tokens in the prompts should be the same as the number of images passed. Found [2] <|image|> tokens and [1] images per sample.What just happened: both Qwen-VL processors accepted every mismatch except two placeholders with one image. That case raised a bare StopIteration with no message, which is hard to diagnose from a log. It comes from the placeholder replacement loop in the 5.22.1 processing_utils.py, line 901, which asks for one more image than it was given. Gemma 3 raised a clear ValueError when images were passed and the counts differed, but stayed silent when the prompt had a placeholder and no image was passed. Gemma 4 raised in every mismatched case.
The silent Qwen cases still go wrong later. Step 2 showed one of them: pixel values for an image with no slot in input_ids. Step 3 showed the reverse, one <|image_pad|> placeholder sent unexpanded with no pixel values. Transformers' Qwen-VL model code checks the two counts against each other in its forward pass, and other code paths may not. A comment in the same processing_utils.py notes that vLLM encodes text and media separately and errors when a placeholder has no media. A check before the processor is therefore required for Qwen. It is still useful for Gemma, because it gives one error type and one message for every model.
Step 9: Check that placeholders and images pair up before the processor
Goal: add check_media_alignment, which counts the media parts in a conversation and the media items supplied for them, and raises MediaMismatch when the two differ.
Why this step: step 8 showed that you cannot rely on the processor to catch a mismatch, so the check has to be yours. Two earlier results keep it small. After step 4's contract, every media part has a known type. And step 3's first column showed that all four families write exactly one placeholder for an image part, so for images, counting parts is counting placeholders.
Each placeholder needs exactly one image. There are two ways to supply it:
- From the parts.
apply_chat_template(..., tokenize=True)loads each image from its part. Every part must carry a source. The check takesimages=Nonefor this path. - From a list.
processor(text=..., images=...)fills the placeholders from its list. It ignores sources in the parts. The parts must be bare, and the list must match them. The check takes the list for this path.
The two paths never mix, so a part with a source plus a list is refused. It runs before rendering, works the same on every model, and fails with a message you wrote. It has one limit: it counts parts, not rendered placeholders, so video and audio parts sent to a model that cannot take them are outside what it checks.
direction: down
parts: "count_media_parts\nslots: one per placeholder" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
sup: "count_attached_media (images=None)\nor len(images) (bare parts)" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
cmp: "check_media_alignment\ncompare the two numbers" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
ok: "equal: render" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
bad: "different: MediaMismatch" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
parts -> cmp
sup -> cmp
cmp -> ok
cmp -> bad: "or a source plus images="
The purple box counts slots and the blue box counts what fills them. The check compares the two numbers before anything is rendered.
Add the check to the end of templatestudio/multimodal.py. SOURCE_KEYS are the keys Transformers reads a part's media from: a PIL image, a URL, a path or base64 data. A part with none of them is a bare placeholder, which only the list path can fill:
# templatestudio/multimodal.py (add at the end)# The keys Transformers reads a media part's source from.SOURCE_KEYS: tuple[str, ...] = ("image", "video", "audio", "url", "path", "base64")def count_media_parts(conversation: Conversation) -> int: """Media parts across the conversation. The template writes one placeholder each.""" total = 0 for message in conversation: content = message.get("content") if isinstance(content, list): total += sum(1 for part in content if part.get("type") != "text") return totaldef count_attached_media(conversation: Conversation) -> int: """Media parts that carry their own source, which the processor loads itself.""" total = 0 for message in conversation: content = message.get("content") if isinstance(content, list): total += sum( 1 for part in content if part.get("type") != "text" and any(k in part for k in SOURCE_KEYS) ) return totalclass MediaMismatch(ValueError): """Raised when placeholders and media items do not pair up one to one."""def check_media_alignment( conversation: Conversation, images: Sequence[object] | None = None) -> None: """Raise unless every media part has exactly one media item to fill it. images=None: apply_chat_template(tokenize=True) loads the media from the parts, so every media part must carry its source. A list: you call processor(text=..., images=images) yourself, which ignores sources in the parts, so every media part must be bare and the list must match them. Call it after assert_valid: it expects every part to be a dict. """ placeholders = count_media_parts(conversation) attached = count_attached_media(conversation) if images is None: supplied = attached elif attached: raise MediaMismatch( f"{attached} part(s) have a source, but images= was passed" ) else: supplied = len(images) if placeholders != supplied: raise MediaMismatch( f"{placeholders} media part(s), {supplied} media item(s) supplied" )Create show_alignment.py. It runs step 2's bug, the mismatches from step 8, an attached image sent down the list path, and the two correct cases:
# show_alignment.pyfrom PIL import Imagefrom templatestudio.multimodal import ( MediaMismatch, check_media_alignment, image_part, text_part, user_message,)photo = Image.open("images/small.png")bare = [user_message([image_part(), text_part("Hi")])]attached = [user_message([image_part(photo), text_part("Hi")])]# (conversation, images): None means the processor loads media from the parts.CASES = { "string content, 1 image": ( [{"role": "user", "content": "What is wrong with this product?"}], [photo], ), "1 bare part, 2 images": (bare, [photo] * 2), "1 bare part, no list": (bare, None), "2 bare parts, 1 image": ( [user_message([image_part(), image_part(), text_part("Hi")])], [photo], ), "1 attached, 1 image": (attached, [photo]), "1 attached, no list": (attached, None), "1 bare part, 1 image": (bare, [photo]),}for label, (conversation, images) in CASES.items(): try: check_media_alignment(conversation, images) print(f"{label:24} ok") except MediaMismatch as error: print(f"{label:24} MediaMismatch: {error}")Run it:
python show_alignment.pyExpected output:
string content, 1 image MediaMismatch: 0 media part(s), 1 media item(s) supplied1 bare part, 2 images MediaMismatch: 1 media part(s), 2 media item(s) supplied1 bare part, no list MediaMismatch: 1 media part(s), 0 media item(s) supplied2 bare parts, 1 image MediaMismatch: 2 media part(s), 1 media item(s) supplied1 attached, 1 image MediaMismatch: 1 part(s) have a source, but images= was passed1 attached, no list ok1 bare part, 1 image okWhat just happened: every case that a Qwen processor accepted in silence, including step 2's string content, now fails before rendering, with the two counts in the message. "1 attached, 1 image" fails too. The caller supplied the image on both paths, and each path ignores the other's source, so the request is ambiguous. If your code hands the processor a separate image list, this is the call to make before processor(text=..., images=...), with bare image parts, as the last case shows. The two correct shapes pass: an image attached to its part with no list, and a bare part with one image in the list. The check does not need a processor, so it runs the same way in a request handler, in a queue worker and in a test.
Step 10: Expand a conversation safely in one call
Goal: add safe_expand, the production path, which checks the input contract, normalizes a copy, checks the media count, and only then runs stage 2.
Why this step: the checks from steps 4, 5 and 9 protect a request only if every call runs them, and runs them in the right order. The contract comes first. normalize_conversation reads part["image_url"]["url"], so a malformed part, such as step 4's bare-string image_url, crashes it with TypeError: string indices must be integers, not 'str' before the validator can name the problem. Normalizing comes second, because the count works on the rewritten parts. The count comes third, and the processor last.
safe_expand plays the role for images that Part 2's safe_render plays for text. It handles media attached to their parts, which is what an OpenAI-style request carries; for a separate image list, use step 9's call directly.
direction: right
v: "assert_valid\ninput contract" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
n: "normalize_conversation\ncopy, image_url -> image" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
c: "check_media_alignment\nparts = media" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
e: "expand\nstage 2" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
v -> n: "1"
n -> c: "2"
c -> e: "3"
A conversation that fails at the first or the third box never reaches the processor.
Add it to the end of templatestudio/multimodal.py:
# templatestudio/multimodal.py (add at the end)def safe_expand( processor: ProcessorMixin, conversation: Conversation, *, add_generation_prompt: bool = True,) -> BatchFeature: """Check the input contract, normalize a copy, check the media count, expand.""" assert_valid(conversation) conversation = normalize_conversation(conversation) check_media_alignment(conversation) return expand(processor, conversation, add_generation_prompt=add_generation_prompt)Create show_safe_expand.py. It runs a good conversation, an OpenAI request, a misspelled part, a bare-string image_url and a bare part with no image through Qwen3-VL:
# show_safe_expand.pyimport textwrapfrom PIL import Imagefrom families import VISION_MODELSfrom samples import openai_image_requestfrom templatestudio.messages import InvalidConversationfrom templatestudio.multimodal import ( MediaMismatch, count_expanded_placeholders, image_part, image_token_id, load_processor, safe_expand, text_part, user_message,)processor = load_processor(VISION_MODELS["qwen3-vl"])photo = Image.open("images/small.png")request = openai_image_request()CASES = { "typed, attached image": [user_message([image_part(photo), text_part("Hi")])], "OpenAI image_url": request, "misspelled type": [user_message([{"type": "Image", "image": photo}])], "bare-string image_url": [ user_message([{"type": "image_url", "image_url": "images/small.png"}]) ], "bare part, no image": [user_message([image_part(), text_part("Hi")])],}for label, conversation in CASES.items(): try: ids = safe_expand(processor, conversation)["input_ids"][0].tolist() tokens = count_expanded_placeholders(ids, image_token_id(processor)) print(f"{label}: ok, {tokens} image tokens") except (InvalidConversation, MediaMismatch) as error: line = f"{label}: {type(error).__name__}: {error}" print(textwrap.fill(line, 72, subsequent_indent=" "))print("caller's request unchanged:", request[0]["content"][0]["type"])Run it:
python show_safe_expand.pyExpected output:
typed, attached image: ok, 196 image tokensOpenAI image_url: ok, 196 image tokensmisspelled type: InvalidConversation: message 0: content[0]: unknown part type 'Image'bare-string image_url: InvalidConversation: message 0: content[0]: image_url part needs image_url.urlbare part, no image: MediaMismatch: 1 media part(s), 0 media item(s) suppliedcaller's request unchanged: image_urlThe OpenAI request carries a path, not a PIL image, so the processor decodes the file. Before the output above, it prints one warning on stderr: [transformers] Image decoding with `torchvision` is deprecated and will be removed in future versions .Please install `torchcodec>=0.16.0` instead. The warning is harmless. Installing torchcodec removes it, and passing PIL images, as steps 6 and 7 do, avoids the decode entirely.
What just happened: the two good conversations expanded to 196 image tokens each. The misspelled part and the bare-string image_url failed the input contract, and the bare part failed the media count, all three before the processor ran. The bare-string case is the reason the contract runs first: it gets a named problem from the validator instead of a crash in the normalizer. The OpenAI request rendered, and the last line shows the caller's copy still says image_url, because safe_expand worked on a copy. You now have one call that either returns token IDs whose placeholders and images match, or raises an error that names the problem.
Fix "Image features and image tokens do not match" and other multimodal errors
Image features and image tokens do not match is the error the model itself raises at generation time when the expanded image tokens and the image features disagree; the processor errors below come earlier, or not at all. Rows that cite a step reproduce in that step. The others name where the string comes from: the Transformers 5.22.1 source, or a public report.
| Error or symptom | Cause | Fix |
|---|---|---|
ValueError: Image features and image tokens do not match, tokens: N, features: M at generation time, where N and M are the two counts (the 5.22.1 format string in modeling_qwen2_5_vl.py; this tutorial loads no weights) | The prompt's image tokens and the images' features disagree, often string content with a separate image (step 2). A Qwen2.5-VL-72B report on Transformers 4.x shows the older wording, Image features and image tokens do not match: tokens: 0, features 5040 (discussion #13) | Build typed content and run check_media_alignment before the processor (step 9) |
ValueError: Prompt contained 1 image tokens but received 2 images. | Gemma 3: more images than <start_of_image> placeholders (step 8) | Make the image parts and the image list the same length, or attach each image to its part |
ValueError: Prompt contained 0 image tokens but received 1 images. on Gemma 3 | You wrote the expanded token <image_soft_token> into the prompt by hand, or the content is a string | Let the template write the placeholder from an image part |
StopIteration with no message from a Qwen-VL processor call | More placeholders than images (step 8) | Run check_media_alignment; it raises MediaMismatch with both counts |
KeyError: 'type' from apply_chat_template(..., tokenize=True) | A content part with no type key: step 3's "no type" part, sent through stage 2. Stage 1 drops it in silence, and stage 2 reads part["type"] while collecting images (5.22.1 processing_utils.py; reproduced while preparing this tutorial, not in a step) | Run assert_valid; step 4's check reports is not a typed part |
| The model answers as if no image was sent, with no error | Qwen-VL with string content, a misspelled part type, or a bare part with no image; Gemma 3 with a bare part and no image (steps 2, 3, 8) | safe_expand (step 10) |
MediaMismatch: 1 part(s) have a source, but images= was passed | An image attached to its part and also passed in an images= list (step 9) | Use bare parts with the list, or drop the list and let safe_expand load the parts |
TypeError: string indices must be integers, not 'str' from normalize_conversation | An image_url part whose image_url is a string, normalized before validation (step 10) | Run assert_valid first, or call safe_expand, which does |
| No placeholder when a client's OpenAI request renders through a tokenizer | The tokenizer path does not rewrite image_url, and Gemma 3's template ignores it (step 5) | normalize_conversation before any render |
An image_url part changed to type: image after a render call | processor.apply_chat_template rewrites image_url parts in place (step 5) | Pass a copy from normalize_conversation |
Image decoding with `torchvision` is deprecated and will be removed in future versions .Please install `torchcodec>=0.16.0` instead. | An image part holds a path or URL, and torchcodec is not installed (step 10) | Harmless; install torchcodec, or pass PIL images |
Kwargs passed to `processor.__call__` have to be in `processor_kwargs` dict, not in `**kwargs` on every call | A processor option such as num_frames=8 passed straight to apply_chat_template; since 5.22.0 (PR #49812) it is merged into processor_kwargs instead of replacing it, but the warning still fires | Pass processor_kwargs={"num_frames": 8} |
FutureWarning: Passing `min_pixels` and `max_pixels` to a processor call is deprecated | Per-call pixel bounds on a Qwen-VL processor; the warning says the arguments "will be removed in v5.23" (image_processing_qwen2_vl.py at v5.22.1) | Pass size={"shortest_edge": ..., "longest_edge": ...} |
ImportError: Qwen2VLVideoProcessor requires the Torchvision library but it was not found in your environment. when loading Qwen2.5-VL without torchvision; Qwen3-VL and Gemma 4 raise the same message naming their own video processor class | torchvision is not installed; their processors include a video processor | Install torchvision==0.29.2 from the CPU index (Prerequisites) |
Gemma 4 raises "The total number of <|image|> tokens in the prompts should be the same"
Gemma 4's processor checks the count in every direction. Step 8 prints its two messages wrapped at 72 characters. Unwrapped, for the one-placeholder, two-image case and for the no-image case, they read:
The total number of <|image|> tokens in the prompts should be the same as the number of images passed. Found [1] <|image|> tokens and [2] images per sample.Found 1 <|image|> tokens in the text but no images were passed.
The fix is the same as for the other families: give each image part exactly one image, and run check_media_alignment so the error is the same on every model.
How the multimodal chat template module fits the processor render loop
multimodal.py is the only new module. It sits beside Part 1's render.py, uses Part 2's messages.py for the contract, and calls the processor for both stages. The diagram uses the names from your code.
direction: down
app: "your request handler\nOpenAI request or typed parts" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
mm: "multimodal.py: safe_expand" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
checks: "before the processor" {
label.near: top-center
style: {fill: "#F4F1FD"; stroke: "#5A4BC4"; font-color: "#2C2C2A"}
grid-rows: 1
grid-gap: 20
msg: "2. assert_valid\nmessages.py (Part 2)" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
norm: "3. normalize_conversation\nmultimodal.py" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
align: "4. check_media_alignment\nmultimodal.py" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
proc: "processor.apply_chat_template\nstage 1 + stage 2" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
out: "input_ids + pixel_values\ncount_expanded_placeholders" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
app -> mm: "1. conversation"
mm -> checks
checks -> proc: "5. expand"
proc -> out: "6. tokens"
Follow the numbers. Your handler calls safe_expand once. The purple contract check runs first, then the blue copy step, then the purple count check, and none of the three needs a processor. Only a conversation that passes both checks reaches the green processor. render_multimodal is not on this path: it is the stage 1 render, for reading and debugging prompts. multimodal.py depends on messages.py and on Transformers, while messages.py still imports neither multimodal.py nor Transformers.
Complete code: render images through a chat template processor in Python
The project after this part, with the files this part added or changed marked. In the listings below, a step number inside a code comment refers to the part that wrote that line, because the listings match the files you already have. messages.py has no file-name comment on its first line, because Part 2 did not give it one:
chat-templates-part1/├── images/ (new: small.png, wide.png, photo.png)├── templatestudio/│ ├── __init__.py│ ├── messages.py (list branch replaced; validate_content_parts added)│ ├── render.py (unchanged since Part 6)│ ├── multimodal.py (new)│ └── ... tools.py, reasoning.py (Parts 5, 6); jinja_env.py, anatomy.py if built├── families.py (VISION_MODELS appended)├── samples.py (openai_image_request appended)├── check_versions.py (new)├── make_images.py (new)├── fetch_vision.py (new)├── show_typed.py (new)├── show_string_content.py (new)├── show_dropped_parts.py (new)├── validate_parts.py (new)├── show_image_url.py (new)├── show_normalized_image_url.py (new)├── show_expansion.py (new)├── compare_image_cost.py (new)├── compare_mismatch.py (new)├── show_alignment.py (new)├── show_safe_expand.py (new)└── ... Parts 1 to 6 scripts (fetch_models.py, rates.py, show_*.py and the rest)templatestudio/multimodal.py:
# templatestudio/multimodal.pyfrom __future__ import annotationsfrom collections.abc import Sequencefrom typing import castfrom PIL import Imagefrom transformers import AutoProcessor, BatchFeature, ProcessorMixinfrom templatestudio.messages import Conversation, Message, assert_validdef load_processor(model_id: str) -> ProcessorMixin: """Load the processor: the tokenizer plus the image and video processors.""" return AutoProcessor.from_pretrained(model_id)def text_part(text: str) -> dict: """A text part of a multimodal content list.""" return {"type": "text", "text": text}def image_part(source: str | Image.Image | None = None) -> dict: """An image part. A string is a URL, a local path or a base64 data URI; a PIL image is used as it is; None leaves a bare placeholder part.""" if source is None: return {"type": "image"} if isinstance(source, Image.Image): return {"type": "image", "image": source} return {"type": "image", "url": source}def user_message(parts: list[dict]) -> Message: """A user turn whose content is an ordered list of typed parts.""" return {"role": "user", "content": parts}def render_multimodal( processor: ProcessorMixin, conversation: Conversation, *, add_generation_prompt: bool = True,) -> str: """Stage 1: render through the processor's template. Images stay placeholders.""" return processor.apply_chat_template( conversation, tokenize=False, add_generation_prompt=add_generation_prompt, )def normalize_content_part(part: dict) -> dict: """Rewrite an OpenAI image_url part as {"type": "image", "url": ...}.""" if part.get("type") != "image_url": return part return {"type": "image", "url": part["image_url"]["url"]}def normalize_conversation(conversation: Conversation) -> Conversation: """Return a copy with every image_url part rewritten. The input is not changed.""" normalized: Conversation = [] for message in conversation: content = message.get("content") if isinstance(content, list): parts = [normalize_content_part(part) for part in content] message = cast(Message, {**message, "content": parts}) normalized.append(message) return normalizeddef image_token_id(processor: ProcessorMixin) -> int: """The token each image expands into. On Gemma 3 this is not the placeholder.""" return processor.image_token_ids[0]def expand( processor: ProcessorMixin, conversation: Conversation, *, add_generation_prompt: bool = True,) -> BatchFeature: """Stage 2: load the images and expand every placeholder into image tokens.""" return processor.apply_chat_template( conversation, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=add_generation_prompt, )def count_expanded_placeholders(input_ids: Sequence[int], token_id: int) -> int: """Count one token in a list of IDs: after stage 2, what the images cost.""" return sum(1 for token in input_ids if token == token_id)# The keys Transformers reads a media part's source from.SOURCE_KEYS: tuple[str, ...] = ("image", "video", "audio", "url", "path", "base64")def count_media_parts(conversation: Conversation) -> int: """Media parts across the conversation. The template writes one placeholder each.""" total = 0 for message in conversation: content = message.get("content") if isinstance(content, list): total += sum(1 for part in content if part.get("type") != "text") return totaldef count_attached_media(conversation: Conversation) -> int: """Media parts that carry their own source, which the processor loads itself.""" total = 0 for message in conversation: content = message.get("content") if isinstance(content, list): total += sum( 1 for part in content if part.get("type") != "text" and any(k in part for k in SOURCE_KEYS) ) return totalclass MediaMismatch(ValueError): """Raised when placeholders and media items do not pair up one to one."""def check_media_alignment( conversation: Conversation, images: Sequence[object] | None = None) -> None: """Raise unless every media part has exactly one media item to fill it. images=None: apply_chat_template(tokenize=True) loads the media from the parts, so every media part must carry its source. A list: you call processor(text=..., images=images) yourself, which ignores sources in the parts, so every media part must be bare and the list must match them. Call it after assert_valid: it expects every part to be a dict. """ placeholders = count_media_parts(conversation) attached = count_attached_media(conversation) if images is None: supplied = attached elif attached: raise MediaMismatch( f"{attached} part(s) have a source, but images= was passed" ) else: supplied = len(images) if placeholders != supplied: raise MediaMismatch( f"{placeholders} media part(s), {supplied} media item(s) supplied" )def safe_expand( processor: ProcessorMixin, conversation: Conversation, *, add_generation_prompt: bool = True,) -> BatchFeature: """Check the input contract, normalize a copy, check the media count, expand.""" assert_valid(conversation) conversation = normalize_conversation(conversation) check_media_alignment(conversation) return expand(processor, conversation, add_generation_prompt=add_generation_prompt)templatestudio/messages.py:
from __future__ import annotationsfrom typing import Literal, NotRequired, TypedDict, UnionRole = Literal["system", "developer", "user", "assistant", "tool"]VALID_ROLES: frozenset[str] = frozenset( {"system", "developer", "user", "assistant", "tool"})# A content part is one item in a multimodal or tool message.ContentPart = dict[str, object]Content = Union[str, list[ContentPart]]# One entry in an assistant message's tool_calls list (step 6).ToolCall = dict[str, object]class Message(TypedDict): role: Role content: NotRequired[Content | None] tool_calls: NotRequired[list[ToolCall]] reasoning_content: NotRequired[str]Conversation = list[Message]def validate_conversation(conversation: Conversation) -> list[str]: """Return every input-contract violation. An empty list means well-formed.""" if not conversation: return ["conversation is empty"] problems: list[str] = [] for i, message in enumerate(conversation): if not isinstance(message, dict): kind = type(message).__name__ problems.append(f"message {i}: must be a dict, got {kind}") continue role = message.get("role") if role not in VALID_ROLES: problems.append(f"message {i}: unknown role {role!r}") tool_calls = message.get("tool_calls") if tool_calls is not None: problems.extend( f"message {i}: {p}" for p in _validate_tool_calls(tool_calls) ) reasoning = message.get("reasoning_content") if reasoning is not None and not isinstance(reasoning, str): kind = type(reasoning).__name__ problems.append(f"message {i}: reasoning_content must be str, got {kind}") content = message.get("content") # An assistant turn that calls a tool may carry no content at all. # An empty tool_calls list calls nothing, so it gets no exception. calls_only = role == "assistant" and bool(tool_calls) if content is None: if not calls_only: problems.append(f"message {i}: content is None") elif not isinstance(content, (str, list)): kind = type(content).__name__ problems.append(f"message {i}: content must be str or list, got {kind}") elif isinstance(content, str) and not content.strip(): # A tool that returned nothing still answered its call. if not calls_only and role != "tool": problems.append(f"message {i}: content is empty or whitespace") elif isinstance(content, list): problems.extend( f"message {i}: {p}" for p in validate_content_parts(content) ) for i, message in enumerate(conversation): if isinstance(message, dict) and message.get("role") == "system" and i != 0: problems.append(f"message {i}: system message must be first") return problemsdef _validate_tool_calls(tool_calls: object) -> list[str]: """Check the shape of an assistant message's tool_calls.""" if not isinstance(tool_calls, list): return ["tool_calls must be a list"] problems: list[str] = [] for j, call in enumerate(tool_calls): function = call.get("function") if isinstance(call, dict) else None if not isinstance(function, dict): problems.append(f"tool_calls[{j}]: missing function object") elif "name" not in function or "arguments" not in function: problems.append( f"tool_calls[{j}]: function needs name and arguments" ) return problemsclass InvalidConversation(ValueError): """Raised when a conversation violates the input contract."""def assert_valid(conversation: Conversation) -> None: """Raise InvalidConversation listing every violation, or return cleanly.""" problems = validate_conversation(conversation) if problems: raise InvalidConversation("; ".join(problems))MEDIA_TYPES: frozenset[str] = frozenset({"image", "video", "audio"})def validate_content_parts(content: list[object]) -> list[str]: """Check each part of a list content: a dict with a known type and its fields.""" problems: list[str] = [] for j, part in enumerate(content): if not isinstance(part, dict) or "type" not in part: problems.append(f"content[{j}] is not a typed part") continue kind = part["type"] if kind == "text": if not isinstance(part.get("text"), str): problems.append(f"content[{j}]: text part needs a 'text' string") elif kind == "image_url": source = part.get("image_url") url = source.get("url") if isinstance(source, dict) else None if not isinstance(url, str) or not url: problems.append(f"content[{j}]: image_url part needs image_url.url") elif kind not in MEDIA_TYPES: problems.append(f"content[{j}]: unknown part type {kind!r}") return problemsfamilies.py:
# families.py# Three families, three wire formats. The Llama 3 and Gemma 3 repos are ungated# copies whose chat templates match the official, gated repos exactly.FAMILIES: dict[str, str] = { "qwen3": "Qwen/Qwen3-0.6B", "llama3": "NousResearch/Meta-Llama-3-8B-Instruct", "gemma3": "unsloth/gemma-3-1b-it",}# Gemma 4 is ungated. It joins in step 6, where its turn markers matter.GEMMA4 = "google/gemma-4-E2B-it"# Part 6: templates that handle reasoning, plus one family that has none.THINKING_MODELS: dict[str, str] = { "qwen3": "Qwen/Qwen3-0.6B", "qwen3.5": "Qwen/Qwen3.5-0.8B", "qwen3-instruct-2507": "Qwen/Qwen3-4B-Instruct-2507", "qwen3-thinking-2507": "Qwen/Qwen3-4B-Thinking-2507", "gemma4": GEMMA4, "llama3": FAMILIES["llama3"],}# Part 7: vision-language models. Only their processors download, never weights.VISION_MODELS: dict[str, str] = { "qwen2.5-vl": "Qwen/Qwen2.5-VL-3B-Instruct", "qwen3-vl": "Qwen/Qwen3-VL-2B-Instruct", "gemma3": "unsloth/gemma-3-4b-it", "gemma4": GEMMA4,}samples.py:
# samples.py# Raw text in each family's tool-call format, as a model writes it after its# generation prompt. Step 13 has Qwen3 write a real call instead.QWEN3_CALL = ( "<tool_call>\n" '{"name": "get_exchange_rate", "arguments": {"base": "EUR", "quote": "JPY"}}\n' "</tool_call>")FENCED_CALL = ( "```json\n" '{"name": "get_exchange_rate", "arguments": {"base": "EUR", "quote": "JPY"}}\n' "```")GEMMA4_CALL = ( '<|tool_call>call:get_exchange_rate{base:<|"|>EUR<|"|>,quote:<|"|>JPY<|"|>}' "<tool_call|>")PLAIN_ANSWER = "One euro buys about 162.4 yen."# Part 6: one reasoned tool step, as Qwen3 writes it with thinking on.QWEN3_REASONING = "The user wants EUR to JPY. I should call get_exchange_rate."QWEN3_REASONED_CALL = ( "<think>\n" + QWEN3_REASONING + "\n</think>\n\n" + QWEN3_CALL + "<|im_end|>")def openai_image_request() -> list[dict]: """A user turn in the shape an OpenAI-compatible client sends. A function, not a constant, so every caller gets a fresh copy.""" image = {"type": "image_url", "image_url": {"url": "images/small.png"}} text = {"type": "text", "text": "Describe it."} return [{"role": "user", "content": [image, text]}]render.py, tools.py and reasoning.py are unchanged; their listings are in Part 6's complete code. The scripts are listed in full in the steps that create them.
Next steps: extend the multimodal module to video, client limits and cost tests
- Video and frame sampling. Add a video part to a Qwen3-VL conversation and pass
processor_kwargs={"num_frames": 8}. Then measure how the expanded token count grows withnum_frames, as step 7 did for image size. Video decoding needstorchcodec, which in turn needs FFmpeg installed on your machine. - Per-request image limits. Use
count_expanded_placeholdersonexpand's output to reject a request whose images cost more than a token budget, before generation. Run step 7's table for your own models, and pin the image sizes you test with. - A strictness switch.
validate_content_partsrejects any type it does not know. A new model may accept a type it does not list, such as an audio part with a different name. Make the rule configurable: reject in CI, and warn or pass through in production.
The Chat Templates Handbook, the book this series accompanies, covers multimodal templates in chapter 7. That includes the processor boundary, a function that reproduces the Qwen-VL resize arithmetic, video timestamps, and how servers such as vLLM place media. The companion repository's v1.2/ch07-multimodal tag holds the chapter's own multimodal.py. Part 8 moves from using templates to writing one: authoring a chat template for a model you fine-tuned.
References
- Hugging Face Transformers v5.22.1: multimodal chat templates, typed content and the processor's
apply_chat_template. processing_utils.pyat transformers v5.22.1: theimage_urlrewrite, media loading and placeholder expansion.- Transformers v5.22.1 release, v5.22.2 release and PR #50431, the Jinja sandbox hardening.
- PR #49812, which in 5.22.0 stopped stray processor kwargs from replacing an explicit
processor_kwargs. - Qwen/Qwen2.5-VL-3B-Instruct and its
preprocessor_config.json, the pixel bounds behind step 7. - Qwen/Qwen3-VL-2B-Instruct and its
preprocessor_config.json. - unsloth/gemma-3-4b-it, its
chat_template.json, which places images only for typeimage, and itspreprocessor_config.json, the fixed 896 by 896 size. - google/gemma-4-E2B-it and its
processor_config.json, the image token budgets. - OpenAI Chat Completions API reference, the
image_urlcontent part. - vLLM: multimodal inputs, and vLLM v0.16.0: chat template content format.
- Qwen2.5-VL-72B-Instruct discussion #13 and Hugging Face forum: Image features and image tokens do not match, the model-side mismatch error in practice.
- Hugging Face forum: image inference with Gemma 3, the Gemma 3
Prompt contained 0 image tokens but received 1 imageserror in practice. - The pinned packages: transformers 5.22.1, tokenizers 0.23.3, huggingface-hub 2.4.0, Jinja2 3.1.6 and Pillow 13.0.0.
- PyTorch 2.15.0 release, torchvision 0.29.2 and the CPU wheel index.
modeling_qwen2_5_vl.pyat transformers v5.22.1, the model-sideImage features and image tokens do not matchcheck.- torchcodec, the video and image decoder Transformers prefers over torchvision; it needs FFmpeg.
image_processing_qwen2_vl.pyat transformers v5.22.1, the Qwen-VL resize and themin_pixelsdeprecation.image_utils.pyat transformers v5.22.1, the torchvision image-decoding deprecation warning.- Bai et al., Qwen2.5-VL Technical Report, 2025, dynamic resolution in the Qwen-VL vision encoder.
Related Articles
- How to Write a Chat Template for a Fine-Tuned Model and Test It
- How to Validate Chat Messages Before apply_chat_template
- How to Render a Chat Template with Jinja Like Transformers
- How to See What apply_chat_template Sends Your Model
