LLM Chatbot Intent Classification: The Label Isn't Enough

A banking assistant receives: "block my card, someone is using it." The classifier returns Card_Block with a confidence of 0.83. The router looks up the handler and runs it. The card is blocked. Good outcome, until you notice the user has two cards. The message never said which one, and the handler blocked the default. The compromised card stays live. The classifier was right about the intent and the system still failed, because it acted on a label without checking what the action actually needed.

This is a practical guide to LLM chatbot intent classification and intent handling for regulated systems, and to the failure mode that intent-accuracy leaderboards cannot see. The label was correct. The routing was correct. The bot still did the wrong thing.

The Thesis: The Intent Label Is the Least Important Output

Here is the claim this article exists to prove: in a regulated, high-stakes chatbot, the predicted intent label is the least important output of your classifier. What decides whether the bot is safe is three other things - the calibrated confidence of the prediction, the validated entities around it, and a per-handler contract that refuses to act on a label it cannot trust.

Teams spend their effort pushing intent-classification accuracy from 94 to 96 percent. That is real work with small returns. The accuracy that actually protects users is different: it is whether the system knows when not to act - when to ask, when to confirm, when to hand off to a human. A bot that classifies perfectly and acts on every label is more dangerous than a bot that classifies adequately and refuses to act when it is unsure.

The consensus this challenges is the one every tutorial teaches: build a great intent classifier, map the intent to a handler, done. That pipeline is not wrong. It is incomplete in exactly the place where money moves. The label tells you where the user probably wants to go. It does not tell you whether it is safe to take them there.

Why This Matters: The Bot's Mistake Is Your Liability

This is more than a design preference. Regulation, model behavior, and data-residency rules each push the same way.

The mistake is legally yours. In Moffatt v. Air Canada (British Columbia Civil Resolution Tribunal, 2024), an airline chatbot gave a customer wrong information about bereavement fares. The airline argued the bot was a separate entity responsible for its own words. The tribunal rejected that and held the company liable for negligent misrepresentation, awarding damages of $812.02. The number is small; the precedent is not. An $812 airline error set the rule; a bank's blast radius is orders of magnitude larger. Your bot's confident wrong answer is your company's statement.

The confidence number is probably lying to you. The instinct is to trust the score: if confidence is high, act. But modern neural networks are systematically overconfident. Guo et al. (2017) showed that the softmax value does not reflect the true likelihood of correctness; it is miscalibrated and skews high. For Large Language Models (LLMs) it is worse. When you ask an LLM how sure it is, the self-reported confidence is poorly calibrated, with average calibration error above 37 percent across eight tasks in one evaluation (Xiong et al., 2024). The threshold you gate on means nothing until you know what the number behind it represents.

The data cannot always leave the building. For a bank operating in India, the Reserve Bank of India (RBI) payment-data localization directive (2018) requires end-to-end payment data to be stored on servers inside the country. The European Union's General Data Protection Regulation (GDPR) adds data-residency and minimization pressure. Sending every customer message to a cloud LLM Application Programming Interface (API) is not a neutral engineering choice; it is a compliance decision. This quietly shapes which classifier you are even allowed to use.

The Wrong Way: Route on the Argmax Label

Here is the pattern almost every guide shows, including the earlier version of this one. Classify, take the top label, look up the handler, call it.

# The naive router: the label goes straight to an action.intent_router = {    "Check_Balance": handle_check_balance,    "Fund_Transfer": handle_fund_transfer,    "Card_Block": handle_card_block,    "Branch_Location": handle_branch_location,}prediction = classifier(user_message)      # {"intent": "Fund_Transfer", "confidence": 0.88}handler = intent_router[prediction["intent"]]return handler(user_message)               # money moves here

This is the same mistake as trusting a tool call because the model proposed it - an intent label alone is not verification. Teams that sense the risk bolt on a single confidence check:

if prediction["confidence"] < 0.60:    return ask_clarification("Can you confirm what you want to do?")handler = intent_router[prediction["intent"]]return handler(user_message)

This is better, and still wrong in three ways.

First, one global threshold treats a balance query and a fund transfer as equally risky. They are not. Reading a balance at 0.65 confidence is fine; moving money at 0.65 confidence is not.

Second, the threshold assumes the confidence is a probability. For a zero-shot or LLM classifier it is not (more on this below), so < 0.60 is gating on a number whose meaning you have not established.

Third, and most important, confidence is not the only thing the action needs. handle_fund_transfer needs a validated amount, a resolved target account, and step-up authentication. A correct label with missing slots is still an unsafe action. The naive router has no place to express any of that. It knows the intent and nothing about what the intent requires.

The Right Way: Intent Handling via a Handler Contract

Invert the control. Instead of the classifier pushing a label into an action, each action declares up front the conditions under which it is willing to run. The classifier's label becomes a request against those conditions.

I call this the Handler Contract. Every intent's handler publishes three requirements: the minimum calibrated confidence it needs, the slots (entities) that must be present and valid, and the authentication level it demands. A single gate enforces every contract before any handler runs.

from dataclasses import dataclassfrom typing import Callable@dataclass(frozen=True)class HandlerContract:    handler: Callable    min_confidence: float           # calibrated threshold THIS action requires    required_slots: tuple[str, ...] # entities the extractor must have produced    required_auth: str              # "none" | "session" | "otp"    confirm: bool = False           # high blast radius: confirm the parsed action even when confidence is high# The risk of the action sets the bar, not one global number.CONTRACTS = {    "Branch_Location": HandlerContract(handle_branch_location, 0.55, ("pincode",),                "none"),    "Check_Balance":   HandlerContract(handle_check_balance,   0.70, ("account_id",),             "session"),    "Card_Block":      HandlerContract(handle_card_block,      0.75, ("card_id",),                "session", confirm=True),    "Fund_Transfer":   HandlerContract(handle_fund_transfer,   0.90, ("amount", "target_account"), "otp",    confirm=True),}

The gate is small, deterministic, and the only path to an action. It runs its checks in order, and any failure routes to an abstain branch instead of the handler.

def route(prediction, entities, ctx):    """prediction: {"intent", "confidence"}; entities: validated dict; ctx: authenticated context."""    contract = CONTRACTS.get(prediction["intent"])    if contract is None:        return handoff("I could not understand that safely. Connecting you to an agent.")    # 1. Confidence Gate - a low-confidence label is a question, not an order.    if prediction["confidence"] < contract.min_confidence:        return clarify(prediction["intent"])       # "Did you want to transfer money?"    # 2. Slot Gate - the required entities must be present (each validated upstream in extraction).    missing = [s for s in contract.required_slots if s not in entities or entities[s] is None]    if missing:        return slot_fill(missing)                  # ask only for what is missing    # 3. Confirmation Gate - a HIGH-confidence label can still be the WRONG label.    #    For high blast radius, show the parsed action and make the user say yes.    if contract.confirm and not ctx.confirmed:        return confirm_action(prediction["intent"], entities)   # "Transfer 5000 to savings - confirm?"    # 4. Auth Gate - high-stakes actions demand step-up authentication.    if not auth_satisfies(ctx.auth_level, contract.required_auth):        return step_up(contract.required_auth)     # trigger OTP or login, then resume    # Only now is the intent allowed to become an action.    slots = {s: entities[s] for s in contract.required_slots}    return contract.handler(ctx, **slots)

The handler receives an authenticated context plus exactly the slots the contract promised were validated. It never has to re-check the gate's work.

def handle_fund_transfer(ctx, amount, target_account):    # Confidence, slots, confirmation, and auth are already guaranteed by the gate.    if amount > get_available_balance(ctx.user_id):        return "Insufficient funds for this transfer."    tx = call_core_banking_transfer(        ctx.user_id, amount, target_account,        idempotency_key=ctx.request_id,            # same key on retry, never a double transfer    )    if tx.success:        audit_log("fund_transfer", ctx.user_id, amount, target_account, tx.id)        return f"Transferred to {mask(target_account)}. Reference {tx.id}."    return "The transfer could not be completed. No money has left your account."

Look at what that buys you.

The confidence bar is now per-action. Branch_Location runs at 0.55 because the worst case is a wrong map pin. Fund_Transfer demands 0.90 because the worst case is a wrong transfer. The risk of the action sets the threshold, not a global constant.

The gate refuses to act on missing slots. The card-block failure from the opening cannot happen here: Card_Block requires a validated card_id, so an ambiguous "block my card" with two cards on file hits the Slot Gate and asks which one, instead of blocking a default.

Authentication is part of the contract, not an afterthought buried in the handler. Moving money requires a One-Time Password (OTP); reading a balance requires only a live session. The gate enforces the step-up before the handler is even reached. This is the same instinct as scoped, single-use authorization for agent actions: the permission is attached to the specific action, not granted globally.

High blast radius triggers an explicit confirmation, even when confidence is high. This is the check that catches a confidently wrong label. Fund_Transfer at 0.95 is still shown back to the user as "transfer 5000 to savings - confirm?" before any money moves, so a mislabelled request dies at the confirmation, not at the bank. More on that failure mode below.

So the system has only a few ways to behave: run the action, clarify, ask for a slot, confirm the parsed action, or step up auth, plus the handoff for anything unrecognized. Every non-run branch is the system choosing not to act. That choice is the safety mechanism.

The Named Concept: Handler Contract

The Handler Contract is the rule that an intent handler publishes its own preconditions - minimum calibrated confidence, required validated slots, required authentication - and that a single gate enforces those preconditions before the handler can run. The classifier proposes an intent; the contract disposes.

The one-line version, the part worth remembering in a code review: intent is a routing hint, not a verdict. The label tells the router where to look. It does not grant permission to act. Practitioners keep rediscovering this the hard way, in write-ups of production intent pipelines that reach the same conclusion: the label tells you where to route, not whether it is safe to act. The Handler Contract is that lesson written down as code you can enforce, instead of a lesson you relearn after an incident.

This is not a new idea in disguise. The Confidence Gate is a plain application of what the machine-learning literature calls selective classification, or prediction with a reject option - a classifier allowed to abstain when the cost of a wrong decision exceeds the cost of not deciding (Chow, 1970; El-Yaniv and Wiener, 2010; Geifman and El-Yaniv, 2019). The slot check is standard task-oriented dialog: the action depends on the tracked, validated slots, not the intent label alone (Louvan and Magnini, 2020). The contribution here is not any single piece. It is binding them into one gate that every action must pass, so "when not to act" stops being scattered defensive code and becomes a declared, reviewable property of each handler.

What the Handler Contract Does Not Catch

Be precise about the thesis, because there is one failure it does not fix. The gate selects the contract from the label: contract = CONTRACTS[prediction["intent"]]. So the label is still load-bearing in exactly one way - it decides which contract applies. A confidently wrong label (high confidence, wrong intent) sails past the Confidence Gate and gets measured against the wrong handler's contract. "Least important" does not mean the label can be garbage; it means that once routing is correct, the label does the least safety work of the four outputs. The label still has to be right enough to route.

Two things bound this. First, the confidence check catches the uncertain wrong label, which is the common case; a wrong label is usually also a low-confidence one. Second, and this is why Fund_Transfer and Card_Block set confirm=True, the Confirmation Gate catches the confident wrong label: the parsed action is shown back to the user before anything irreversible happens, so a mislabelled "check my balance" that the model confidently tagged as Fund_Transfer dies at "transfer 5000 to savings - confirm?" The residual risk is a confident wrong label that maps to a handler with no confirmation and slots the wrong input happens to satisfy. Keep that set small: every handler that can take an action a user would not want reversed sets confirm=True. The gate does not make a bad classifier safe. It makes a good-enough classifier safe to act on.

How the Gated Intent Classification Pipeline Fits Together

The pipeline is the familiar one, with the gate inserted where the label used to flow straight into an action.

flowchart TD
    A[User message]:::blue --> B[Preprocess<br/>normalize, extract entities]:::teal
    B --> C[Classify<br/>returns intent + confidence + entities]:::blue
    C --> D{Confidence Gate<br/>conf >= contract.min?}:::purple
    D -->|No| E[Clarify<br/>confirm the intent]:::yellow
    D -->|Yes| F{Slot Gate<br/>required slots present?}:::purple
    F -->|No| G[Slot fill<br/>ask for missing entity]:::yellow
    F -->|Yes| P{Confirm Gate<br/>high blast radius?}:::purple
    P -->|Yes, unconfirmed| Q[Confirm action<br/>show parsed action, get yes]:::yellow
    P -->|No / confirmed| H{Auth Gate<br/>auth level sufficient?}:::purple
    H -->|No| I[Step up<br/>OTP or login]:::orange
    H -->|Yes| J[Run handler<br/>the action executes]:::green
    E --> K[Handoff to human<br/>on repeated failure]:::red
    G --> K
    Q --> K
    I --> K

    classDef blue fill:#4A90E2,stroke:#3A7BC8,color:#FFFFFF
    classDef teal fill:#98D8C8,stroke:#6FBCA9,color:#2C2C2A
    classDef purple fill:#7B68EE,stroke:#5A4FCF,color:#FFFFFF
    classDef yellow fill:#FFD93D,stroke:#D4B02A,color:#2C2C2A
    classDef orange fill:#FFA07A,stroke:#E07850,color:#2C2C2A
    classDef green fill:#6BCF7F,stroke:#4CAF64,color:#2C2C2A
    classDef red fill:#E74C3C,stroke:#B03A2E,color:#FFFFFF

Everything above the gate is probabilistic - the model's best guess. Everything from the gate down is deterministic policy you wrote and can audit. The label crosses that line only when the contract lets it.

The Deep Dive: Your Confidence Number Is Not a Probability

The Confidence Gate is only as good as the number it reads. This is where the choice of classifier stops being about accuracy and starts being about whether the confidence means anything. The three common approaches differ less in how well they label and more in what kind of confidence they hand you.

Zero-Shot vs. Few-Shot vs. Fine-Tuned: Which Confidence Can You Gate On?

Fine-tuned encoder - a real, calibratable confidence. Train a small encoder on your labeled intents and its softmax output is a genuine distribution over your fixed intent set. It is still overconfident out of the box, but you can fix that: fit temperature scaling on a held-out set (Guo et al., 2017) and the number becomes a usable calibrated probability. This is the only one of the three approaches where min_confidence = 0.90 means something close to "90 percent likely correct." For a fixed set of banking intents, on-premises, this is the strongest default. distilbert-base-uncased still works as a light baseline, but in 2026 reach for microsoft/deberta-v3-base for accuracy per parameter, or SetFit when you have only a handful of labeled examples per intent and want to avoid a large model entirely.

# 2026-correct fine-tune. Note eval_strategy and processing_class - the old# evaluation_strategy and tokenizer= arguments are deprecated (renamed on the way to v5).from transformers import (    AutoTokenizer, AutoModelForSequenceClassification,    Trainer, TrainingArguments,)model_id = "microsoft/deberta-v3-base"   # DistilBERT works too; this labels bettertokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForSequenceClassification.from_pretrained(    model_id, num_labels=len(label_list), id2label=id2label, label2id=label2id,)args = TrainingArguments(    output_dir="./intent_classifier",    eval_strategy="epoch",               # was evaluation_strategy (deprecated)    save_strategy="epoch",    learning_rate=2e-5,    per_device_train_batch_size=16,    num_train_epochs=5,    load_best_model_at_end=True,)trainer = Trainer(    model=model,    args=args,    train_dataset=train_dataset,    eval_dataset=val_dataset,    processing_class=tokenizer,          # was tokenizer= (deprecated)    compute_metrics=compute_metrics,)trainer.train()# After training, fit temperature scaling on the validation logits so that# a reported 0.90 actually behaves like 0.90 before you gate on it.

Zero-shot Natural Language Inference (NLI) - a confidence that is not a posterior. A zero-shot classifier needs no training data, which is why it is the fastest way to bootstrap. But understand what its "confidence" is. In its default single-label mode, it reframes each candidate label as a hypothesis, scores entailment against the message, and softmaxes those entailment scores across the labels you supplied.

from transformers import pipeline# facebook/bart-large-mnli is the classic; MoritzLaurer/deberta-v3-base-zeroshot-v2.0# is a stronger current choice. Either way, read the caveat below.classifier = pipeline("zero-shot-classification",                      model="MoritzLaurer/deberta-v3-base-zeroshot-v2.0")intents = ["Check_Balance", "Fund_Transfer", "Card_Block", "Branch_Location"]result = classifier("Transfer 5000 to my savings account", candidate_labels=intents)# result["scores"][0] looks like a probability. It is not a calibrated posterior:# it is relative to THIS label set. Add or drop a candidate label and it shifts.

That score is relative to the label set you passed. Add a fifth candidate and every number moves. It is fine for ranking; it is dangerous as a gate threshold, because 0.90 here does not mean what 0.90 means from your calibrated encoder. If you gate on it, calibrate it against real outcomes first, or keep its min_confidence bars deliberately conservative and lean harder on the Slot and Auth gates.

Few-shot generative LLM - flexible labels, no native confidence. Give a current instruct model like meta-llama/Llama-3.1-8B-Instruct or Qwen/Qwen2.5-7B-Instruct a few examples and it will label unfamiliar phrasings well. The catch: a generative model returns text, not a probability. There is no confidence number to gate on unless you build one. To get a usable score, constrain the output to your intent set and read the token log-probabilities over that constrained space. Even then the score is uncalibrated and usually needs temperature scaling. A free-text label with an imagined confidence of "1.0" is the most dangerous input the Confidence Gate can receive. It defeats the gate by looking certain. So never let an LLM classifier emit a bare label; make it emit a constrained choice you can score. And remember that cloud-hosting the model may send customer messages out of your compliance boundary.

The reference slide below is the decision in one view.

Notice the reframing. The old question was "which classifier is most accurate?" The better question is "which classifier gives me a confidence I can gate on, inside my compliance boundary?" For fixed banking intents that usually points at a fine-tuned, calibrated, on-premises encoder - not because it labels best, but because its confidence is the only one the gate can trust literally.

The Practitioner Checklist: Auditing a Handler Contract

Use this in a code review of any chatbot that can take an action a user would not want reversed. Exercise each handler against a local banking sandbox before you trust it in production.

• Every action-taking handler has a declared contract. No handler is reachable except through the gate. If a handler can be called directly with a raw label, the contract is advisory, not enforced.
• Confidence thresholds scale with blast radius. Read-only intents sit low. Irreversible or money-moving intents sit high. If every intent shares one threshold, the risky ones are under-protected and the cheap ones are over-protected.
• You know what your confidence number is. For a fine-tuned model, it is calibrated on a held-out set. For zero-shot or LLM output, you have written down that it is not a posterior and set the bars accordingly. Gating on an uncalibrated number as if it were a probability is the most common silent failure.
• Required slots are validated, not just present. An amount is positive and within limits; a target account resolves to a real beneficiary; a card id belongs to this user. The gate checks presence; your entity-extraction step must have already checked validity, the same validation-and-repair discipline you apply to any model output. Presence is not validation.
• High-blast-radius handlers confirm the parsed action, even at high confidence. Set confirm=True on anything a user would not want reversed. This is the only check that stops a confidently wrong label, so do not rely on the confidence threshold alone for irreversible actions.
• Auth is in the contract, not the handler. Step-up (OTP, biometric) is a declared precondition the gate enforces, so no handler can forget it under a refactor.
• Every abstain branch has an exit. Clarify, slot-fill, confirm, and step-up all lead to a human handoff after repeated failure; the retry counter lives in those helpers. A gate with no handoff traps the user in a loop.
• The gate decision is logged. Which gate fired, on what confidence, is your audit trail and your training signal for what to calibrate next.

Choose the Classifier for Its Confidence, Enforce Safety at the Contract

The tutorials are right that you need a good intent classifier. They stop one step short of the step that matters. The label tells you where the user probably wants to go. It is a routing hint, not a verdict - and in a regulated chatbot, treating it as a verdict is how a correct classification still becomes an incident.

So invert the flow. Let each action declare the confidence, the validated slots, and the authentication it requires, and put one gate in front of all of them. Then the classifier's job shrinks to the thing it is actually good at: proposing. The decision to act - or, more often, the decision not to - lives in the contract, where you can read it, review it, and prove it. Chase the accuracy that keeps you safe: the accuracy of knowing when not to act.

References

• Chow, C. K. (1970). On Optimum Recognition Error and Reject Tradeoff. IEEE Transactions on Information Theory, 16(1). https://doi.org/10.1109/TIT.1970.1054406
• Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML 2017 (PMLR v70). https://arxiv.org/abs/1706.04599
• Xiong, M., et al. (2024). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ICLR 2024. https://arxiv.org/abs/2306.13063
• El-Yaniv, R., & Wiener, Y. (2010). On the Foundations of Noise-free Selective Classification. JMLR. https://www.jmlr.org/papers/v11/el-yaniv10a.html
• Geifman, Y., & El-Yaniv, R. (2019). SelectiveNet: A Deep Neural Network with an Integrated Reject Option. ICML 2019. https://arxiv.org/abs/1901.09192
• Louvan, S., & Magnini, B. (2020). Recent Neural Methods on Slot Filling and Intent Classification for Task-Oriented Dialogue Systems: A Survey. COLING 2020. https://arxiv.org/abs/2011.00564
• Tunstall, L., et al. (2022). Efficient Few-Shot Learning Without Prompts (SetFit). https://arxiv.org/abs/2209.11055
• Hugging Face. facebook/bart-large-mnli - zero-shot method and entailment scoring. https://huggingface.co/facebook/bart-large-mnli
• Hugging Face. Trainer and TrainingArguments - eval_strategy, processing_class. https://huggingface.co/docs/transformers/main_classes/trainer
• Civil Resolution Tribunal of British Columbia (2024). Moffatt v. Air Canada, 2024 BCCRT 149. https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html
• Reserve Bank of India (2018). Storage of Payment System Data. https://www.rbi.org.in/scripts/NotificationUser.aspx?Id=11244

Related Articles

Genai

• Building Agents That Remember: State Management in Multi-Agent AI Systems
• Building Production-Ready Agentic AI: The Infrastructure Nobody Talks About

Llms

• Stop Pasting Screenshots: How AI Engineers Document Systems with Mermaid
• Choosing the Right LLM Inference Framework: A Practical Guide