Skip to main content

BlogKnowledge base

How to stop your support AI from making things up

Reduce invented support answers by checking source quality, retrieval, model behavior and permissions. Use grounded replies, human handoffs and repeatable tests.

Written by the CustomerEagle teamPublished 9 min readUpdated
Knowledge baseQualityAutomation
Knowledge cards pass through blue inspection gates, with one card held beside a reference book for review.
AI-generated editorial illustration: answer testing and human approval.

An invented policy or delivery promise can turn an ordinary support question into a complaint. Incorrect answers can come from stale or missing sources, poor retrieval, misreading the customer's request or the model itself. A useful prevention plan checks those failure points separately, gives the assistant a route to a person and tests what happens when reliable evidence is unavailable.

Three distinct failures that get called one thing

Source and generation failures need different checks. NIST's Generative AI Profile describes how generative systems can present false content confidently, including misleading citations. Grounding helps provide evidence, but the generated answer still needs evaluation. The table below separates common failure points so a reviewer can identify what to investigate.

FailureWhat it looks likeThe actual fix
Retrieval gapThe customer asks about a policy that exists, but the search step returned nothing relevant, so the model filled the space from general knowledge.Check source coverage, indexing and retrieval. Add missing content or fix the search path, then test whether the assistant declines unsupported answers.
Stale sourceThe answer is confidently given and was correct last quarter. The article was never updated.Review dates and one canonical page per fact. The model is faithfully repeating what you told it.
Unconstrained generationThe material was thin, the prompt did not forbid extrapolation, and the model produced a plausible-sounding policy.Test grounding instructions, the model's use of evidence and a working handoff. Check the generated claims even when a relevant source was retrieved.

Give it a way out, and make that the easy path

A model with no acceptable way to fail will fail unacceptably. If every turn must produce an answer, and the reference material does not contain one, something has to give — and what gives is accuracy. The fix is unglamorous: instruct the system to answer only from supplied material, never to invent policies, prices or order details, and when it does not know, to say so plainly and hand off to a person. That instruction is only worth anything if the handoff actually exists and works; an escape hatch that leads to a dead end trains the same behaviour as having none.

Separate what it can read from what it can assert

The most damaging invented answers in support are not wrong policies but wrong specifics: a delivery date, a refund status, an account tier. These should never come from a model's reasoning. They come from a system lookup, after the customer's identity is verified, or they do not get stated at all. The distinction matters enough that we built the identity check and the generic response on a mismatch into the retrieval path rather than the prompt — a prompt instruction is a request, not a control. There is more on where that sits in the features overview.

The controls, ordered by how much they buy you

  1. Check coverage before adding complexity. Review the questions that returned no useful source, including escalations and repeated requests. Use those conversations to prioritize missing articles, then test whether the revised sources and answer controls address the failure.
  2. Grounding instructions that name the prohibited categories explicitly — policies, prices, order details — rather than a general request to be accurate.
  3. A working escalation path, so declining to answer has somewhere to go.
  4. Identity verification before any account-specific statement, with a neutral response when the details do not match, so a failed lookup cannot become a fishing tool.
  5. Human approval for irreversible actions, which converts a wrong answer into a rejected suggestion rather than a refunded order.
  6. Reopen tracking, because a confident wrong answer scores identically to a correct one on resolution rate and response time. This is the argument for a settling window before counting anything as solved — the reasoning is in deflection rate vs resolution rate.

What to ask a vendor

Most demos are run on curated content where retrieval never fails, so the interesting behaviour never appears. Ask instead: what does the system do when retrieval returns nothing — does it decline or improvise? Can I see the source passages behind a given answer? Are those sources visible to my agents, and separately, can they be shown to customers? Can I set a confidence point below which it escalates rather than answers? And what does the log retain, so a complaint can be reconstructed? Our own answers to those are in the documentation, and the measurement side is on how we measure. Build a repeatable set of failure and recovery cases with the AI support regression-testing checklist.

One caveat worth stating plainly, since vendors rarely do: none of this reduces the error rate to zero, and any vendor implying otherwise is selling. The realistic goal is that errors are rare, visible, bounded to reversible actions, and traceable to a source you can fix. That is achievable. Perfection is not, which is why the approval queue described in AI agent vs chatbot matters more than any prompt.

Frequently asked questions

A wrong answer can come from missing or outdated content, retrieval that selected the wrong passage, incorrect customer context or the model's interpretation and generation. Inspect the source and the actual answer separately. Correct the relevant failure, test the question again and verify that unsupported requests reach a person.

Eagle AI

Resolve more tickets automatically.

Connect your help centre, test answers on your own questions, and review handoffs before rolling it out.