An invented policy or delivery promise can turn an ordinary support question into a complaint. Incorrect answers can come from stale or missing sources, poor retrieval, misreading the customer's request or the model itself. A useful prevention plan checks those failure points separately, gives the assistant a route to a person and tests what happens when reliable evidence is unavailable.
Three distinct failures that get called one thing
Source and generation failures need different checks. NIST's Generative AI Profile describes how generative systems can present false content confidently, including misleading citations. Grounding helps provide evidence, but the generated answer still needs evaluation. The table below separates common failure points so a reviewer can identify what to investigate.
| Failure | What it looks like | The actual fix |
|---|---|---|
| Retrieval gap | The customer asks about a policy that exists, but the search step returned nothing relevant, so the model filled the space from general knowledge. | Check source coverage, indexing and retrieval. Add missing content or fix the search path, then test whether the assistant declines unsupported answers. |
| Stale source | The answer is confidently given and was correct last quarter. The article was never updated. | Review dates and one canonical page per fact. The model is faithfully repeating what you told it. |
| Unconstrained generation | The material was thin, the prompt did not forbid extrapolation, and the model produced a plausible-sounding policy. | Test grounding instructions, the model's use of evidence and a working handoff. Check the generated claims even when a relevant source was retrieved. |
Give it a way out, and make that the easy path
A model with no acceptable way to fail will fail unacceptably. If every turn must produce an answer, and the reference material does not contain one, something has to give — and what gives is accuracy. The fix is unglamorous: instruct the system to answer only from supplied material, never to invent policies, prices or order details, and when it does not know, to say so plainly and hand off to a person. That instruction is only worth anything if the handoff actually exists and works; an escape hatch that leads to a dead end trains the same behaviour as having none.
Separate what it can read from what it can assert
The most damaging invented answers in support are not wrong policies but wrong specifics: a delivery date, a refund status, an account tier. These should never come from a model's reasoning. They come from a system lookup, after the customer's identity is verified, or they do not get stated at all. The distinction matters enough that we built the identity check and the generic response on a mismatch into the retrieval path rather than the prompt — a prompt instruction is a request, not a control. There is more on where that sits in the features overview.
The controls, ordered by how much they buy you
- Check coverage before adding complexity. Review the questions that returned no useful source, including escalations and repeated requests. Use those conversations to prioritize missing articles, then test whether the revised sources and answer controls address the failure.
- Grounding instructions that name the prohibited categories explicitly — policies, prices, order details — rather than a general request to be accurate.
- A working escalation path, so declining to answer has somewhere to go.
- Identity verification before any account-specific statement, with a neutral response when the details do not match, so a failed lookup cannot become a fishing tool.
- Human approval for irreversible actions, which converts a wrong answer into a rejected suggestion rather than a refunded order.
- Reopen tracking, because a confident wrong answer scores identically to a correct one on resolution rate and response time. This is the argument for a settling window before counting anything as solved — the reasoning is in deflection rate vs resolution rate.
What to ask a vendor
Most demos are run on curated content where retrieval never fails, so the interesting behaviour never appears. Ask instead: what does the system do when retrieval returns nothing — does it decline or improvise? Can I see the source passages behind a given answer? Are those sources visible to my agents, and separately, can they be shown to customers? Can I set a confidence point below which it escalates rather than answers? And what does the log retain, so a complaint can be reconstructed? Our own answers to those are in the documentation, and the measurement side is on how we measure. Build a repeatable set of failure and recovery cases with the AI support regression-testing checklist.
One caveat worth stating plainly, since vendors rarely do: none of this reduces the error rate to zero, and any vendor implying otherwise is selling. The realistic goal is that errors are rare, visible, bounded to reversible actions, and traceable to a source you can fix. That is achievable. Perfection is not, which is why the approval queue described in AI agent vs chatbot matters more than any prompt.
Frequently asked questions
A wrong answer can come from missing or outdated content, retrieval that selected the wrong passage, incorrect customer context or the model's interpretation and generation. Inspect the source and the actual answer separately. Correct the relevant failure, test the question again and verify that unsupported requests reach a person.
Constrain it to answer only from supplied reference material, name the prohibited categories explicitly such as policies, prices and order details, and give it an explicit instruction to say it does not know and escalate to a person when the material does not cover the question. That escape route only works if the handoff genuinely reaches a human, otherwise the system learns that declining is not a viable option.
Yes, and this is the more tractable half of the problem. Order and account specifics should be produced by a verified system lookup rather than by the model, meaning the customer's identity is confirmed first and a mismatch returns a neutral response rather than a guess. Enforcing that in the retrieval path rather than in prompt wording is what makes it a control instead of a request.
Ask it about a policy you know you have never documented anywhere. A correctly configured system will say it is not certain and offer a human; a poorly configured one will produce a confident and plausible answer. Repeat this with questions whose answers changed recently, since stale-source errors are invisible to the model and will be delivered with the same confidence as correct ones.
Reopen rate within a stated window, because a wrong answer delivered confidently scores identically to a correct one on resolution rate, first-response time and often on immediate satisfaction ratings. The customer discovers the error later and comes back, so any resolution count that is not subject to a settling window will overstate quality precisely where it is worst.
