Why Your AI Chatbot Fails in Production and What to Build Instead

The Short Answer

AI chatbots fail in production because teams measure the wrong thing. Grounding is close to solved: Vectara's hallucination leaderboard shows the best model contradicting a document sitting in its own context only 1.8% of the time. What actually breaks is conversational state, coordination with the customer, and run-to-run variance. Sierra's tau-bench found that agents succeeding on under half of tasks fall below 25% when the identical task is run eight times. Build a deterministic workflow with a narrow tool surface instead.

What Actually Breaks, and It Is Not Hallucination

The consensus fix for a failing support bot is a better model and retrieval. Two published results say that fix is aimed at the wrong target.

Sierra's tau2-bench measured gpt-4.1 at 0.67 pass^1 in its telecom domain when the agent held all the tools and worked from a written ticket, and 0.34 in the default setting where a simulated user holds half the tools and has to be talked through them. Same model, same task, half the success rate. The difference was having to coordinate with a person.

Microsoft Research and Salesforce ran more than 200,000 simulated conversations for their paper "LLMs Get Lost in Multi-Turn Conversation" and found every top model performs significantly worse multi-turn than single-turn, averaging a 39% drop across six generation tasks. Their diagnosis is the sentence to keep: when a model takes a wrong turn in a conversation, it gets lost and does not recover. The failure mode is not a wrong fact, it is a wrong assumption made on turn two and never revisited.

Failure is conversational, not factual. Your demo is one turn long. Your customers are not.

Variance Is the Production Killer, Not Accuracy

Sierra's tau-bench introduced pass^k, which asks whether all k independent attempts at the same task succeed, rather than whether any one of them does. It reported pass^8 below 25% in the retail domain for models scoring under 50% on a single attempt.

Run that arithmetic against your own acceptance test. If failures are roughly independent, a flow that passes 80% of the time is at 0.8^5, about 33%, by the fifth time the same customer hits it. Real failures cluster, so treat that as a floor rather than a forecast. In a support queue it shows up as the same customer getting a different resolution on each contact, with nobody noticing because each transcript looks fine on its own.

Decision rule: ship on pass^5 over a frozen replay set of at least 200 real multi-turn transcripts, not a single passing demo run.

Every Answer About Policy Is a Binding Statement

In Moffatt v. Air Canada, 2024 BCCRT 149, the BC Civil Resolution Tribunal ordered Air Canada to pay $812.02 after its website chatbot told a customer he could apply for bereavement fares retroactively when the published policy did not allow it. At paragraph 27 the tribunal dismissed the airline's defense: Air Canada was in effect suggesting the chatbot is a separate legal entity responsible for its own actions, which the member called a remarkable submission. At paragraph 28 he added that Air Canada never explained why customers should have to double-check information found in one part of its website against another part.

The operative test was ordinary negligent misrepresentation. No AI-specific statute was involved, so the same exposure exists in every jurisdiction recognizing that tort.

The AI Incident Database catalogs a second case as Incident 1039. In April 2025 Cursor users started getting logged out unexpectedly, and the front-line support bot explained the behavior with a login policy it had invented. No such policy change existed. Users cancelled subscriptions over a rule that was never real, and because the answer was non-deterministic, customers comparing notes got different versions of it.

Rule: any answer stating a policy, price, date, or entitlement must be assembled from a retrieved record with a citation, or refused outright.

What to Build Instead of a Chatbot

Excessive Agency, meaning a model holding more permission than the task needs, has been on the OWASP Top 10 for LLM Applications since the 2025 edition, where it ranks sixth. The OWASP GenAI Security Project published the 2026 edition on August 3, 2026, grounded in thousands of real-world AI security incidents. The design conclusion to draw from that list is not to build a model that cannot be fooled. It is to make sure that when the model is fooled, nothing with authority moves.

Google DeepMind and ETH Zurich's CaMeL shows the shape of that answer. It wraps the model in a control and data-flow layer so untrusted content cannot influence program execution, solving 77% of AgentDojo tasks with provable security versus 84% with no defense. Seven points of capability for a guarantee is a trade worth making.

Applied to support, the build is a workflow, not an agent. Anthropic draws that line precisely: workflows coordinate LLMs and tools through predetermined code paths, while agents let the model steer its own process. Its engineering guidance is to start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when simpler solutions fall short.

The LLM proposes. Deterministic code with its own authorization check disposes.

Where the Context Window Betrays You

The reflex fix is to feed the model more of the help center. Chroma's Context Rot report evaluated 18 models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, and found that performance grows increasingly unreliable as input length grows even on trivial tasks. One of those tasks was replicating a list of repeated words, run across context lengths from 25 to 10,000 words. Models degraded on it. A large context window is not usable context.

There is a second reason models guess rather than abstain. In "Why Language Models Hallucinate", researchers from OpenAI and Georgia Tech reviewed the ten evaluations that dominate the major leaderboards, including GPQA, MMLU-Pro, SWE-bench, and HLE. Nine give no credit at all for saying "I don't know"; only WildBench gives partial credit. Models are optimized against that scoring, so bluffing is the trained behavior. Anthropic's guidance on reducing hallucinations names three counters: explicitly grant permission to say "I don't know", ask for word-for-word quote extraction before analysis on documents over 20k tokens, and require a supporting quote for every claim after drafting, retracting any claim that lacks one. Make abstention a scored success outcome in your own eval rubric, and measure it.

Assume the Injection Succeeds

Anything the bot reads can carry instructions: an inbound email, a ticket attachment, a knowledge-base article edited by a partner. CVE-2025-32711, published June 11, 2025 and known as EchoLeak, was an AI command injection in Microsoft 365 Copilot that let an unauthorized attacker disclose information over a network. Microsoft rated it 9.3 critical; NVD rated it 7.5. Both scored it as requiring no privileges and no user interaction, which is the part that matters: the victim did nothing but receive a message.

Budget for the capability tax, not a filter that catches every injection. Nothing the model reads should move money or change account state without a check written in ordinary code.

Set the Automation Target Honestly

Two vendor-published numbers converge on the same ceiling. Intercom's Fin advertises an average resolution rate of 76% across more than 12,000 customers, with many seeing over 85%. That is the vendor's own marketing figure for the most specialized product in the category, so read it as a ceiling. Anthropic's customer support guide, writing about what to target rather than what to sell, says to typically aim for a 70 to 80% deflection rate depending on the complexity of inquiries.

Plan staffing around 20 to 30% of contacts reaching a human, and treat any pitch above 90% autonomous resolution as a scoping claim to verify against your own transcripts. Volume figures are not quality figures: Klarna's own February 2024 release reported 2.3 million conversations in the assistant's first month, the equivalent work of 700 full-time agents, which tells you what the bot answered and nothing about whether it should have.

Two more constraints belong in the plan. The same Anthropic guide sets escalation accuracy at 95% or higher, tracked as the percentage of correctly escalated conversations against those that should have escalated and were not. And since August 2, 2026, Article 50 of the EU AI Act has required that people interacting directly with an AI system be informed of it, unless that is obvious to a reasonably well-informed person in context. Label it in the channel, not in a footer.

Cost Is Not the Reason to Compromise

Anthropic's pricing documentation works a support example at roughly 3,700 tokens per conversation on Claude Haiku 4.5 at $1 per million input tokens and $5 per million output, coming to about $37.00 per 10,000 tickets. Cache reads bill at 0.1x the base input rate and the Batch API is a flat 50% discount on input and output.

Set that against one escalation handled badly. Inference is almost never the cost driver in a support deployment; the eval harness, the transcript labeling, and the humans on the escalation path are. Pick the model on escalation accuracy and multi-turn stability, then control spend with caching and batching. A team that picks the cheaper model to save $37 per 10,000 tickets and loses two points of escalation accuracy has made an expensive trade and will not see it on the API bill.

Pre-Launch Checklist

• Frozen replay set of 200+ real multi-turn transcripts, user withholding information until asked
• pass^5 reported per flow, not a single demo run
• Escalation accuracy measured against conversations that should have escalated and did not
• Abstention scored as a success, not a miss
• Every policy, price, date, or entitlement answer carries a citation to a retrieved record
• Every write tool guarded by an authorization check that does not read the conversation
• Dual-control tasks routed to a human or a scripted wizard by default
• AI disclosure in the channel itself
• A route for corrected answers back into the retrieval corpus

How We Approach This at SASID

Across seven production AI systems in five industries, the pattern that holds is narrow scope with measurement attached. A cybersecurity query engine has run more than six months in production with zero bad queries because the surface it answers over is constrained and checked. A customer platform processes over 30,000 reviews a day across 200 or more locations with a sub-2-minute response SLA, and its response rate moved from 60% to 99.8% because escalation was designed in rather than treated as a defect. Call center QA reached 100% automated coverage where manual review had only ever reached under 5%. None of that came from a bigger model.

Get a Free Technical Assessment

If your support bot is failing in production, or you are deciding whether to build one at all, we offer a free technical assessment: a 30-minute call about your use case, followed by a written roadmap within 48 hours covering architecture, evaluation design, escalation policy, timeline, and cost. There is no obligation, and the roadmap is yours to keep. Book at sasid.ai.