Five layers between a retrieval system and a fabricated answer

A retrieval-augmented system fails loudly when it returns nothing and quietly when it returns something plausible and wrong. The quiet failure is the expensive one: it reads like an answer, and nothing in it signals that the supporting evidence was missing or said the opposite. Nobody files a bug against a fluent paragraph.

The controls that address it are a stack rather than a single change, each catching a different slice of the problem and each leaking in a different direction. The note below lists five, ordered by leverage rather than effort. What follows expands each into what it catches, what it cannot catch, and how to install it on a running system.

The original note

Mitigating Hallucinations

• Retrieval grounding: restrict answers to retrieved docs.
• Instruction tuning: “Do not guess if not in context.”
• Fact-checking: post-processing via smaller verifier LLM.
• Eval loop: monitor faithfulness score over time.
• Guardrails: prompt templates + output validators.

Hallucination is not one failure

Two unrelated failures share the name, and they have nothing in common except the symptom.

The first is a retrieval miss. The fact was never placed in the model’s context window — chunking split it across a boundary, the query embedding missed the passage, the document was not in the corpus. The generator had nothing to ground on and produced text anyway. No prompt discipline manufactures evidence that was never retrieved; those fixes live upstream, in chunking, query rewriting, hybrid retrieval, reranking, and corpus coverage.

The second is generator drift. The correct evidence was retrieved and sitting in the context, and the answer still contradicts it, overstates it, or fuses two passages into a claim neither one makes. Those fixes live downstream: citation-carrying prompts, claim verification, output validation.

The cheapest diagnostic

For each bad answer, ask one question: was the supporting evidence in the retrieved set? That single boolean partitions every failure into the two buckets. Teams that skip it tune prompt wording against what is actually a chunking bug and watch nothing move. The distinction is the reference point for everything below — each layer works on one bucket or the other, never both.

Faithfulness is not correctness

Faithfulness — also called groundedness — asks whether every claim in the answer is supported by the retrieved context. Correctness asks whether the claim is true. These come apart in both directions, and the gap matters.

An answer can be perfectly faithful and completely wrong, because the document it faithfully reproduces is stale, misattributed, or simply incorrect. Faithfulness has no opinion about the corpus; it checks only that the generator did not depart from it. In the other direction, an answer can be correct and unfaithful, when the model supplies a true fact from parametric memory that appears in no retrieved passage. That is still a defect: it is unauditable, and the mechanism that got it right here will get it wrong elsewhere with no change in confidence.

The practical asymmetry is cost. Faithfulness is cheap to measure automatically because the evidence is already in the request — an entailment check in the NLI style, or a smaller model acting as judge, decides it with no ground-truth label, so it runs continuously over production traffic. Correctness needs labelled answers or an authoritative source, so it runs on a curated set, on a schedule, with human involvement. Monitor the first continuously, sample the second periodically, and never let one stand in for the other: no faithfulness score detects a corpus that went out of date.

Layer 1 — Retrieval grounding

Restricting answers to retrieved documents moves the needle most, and it is worth being blunt: the other four are refinements on top of it.

Grounding is more than pasting documents into a prompt. It means giving every retrieved passage a stable ID, injecting the passages numbered and tagged, and requiring citations at the span level — sentence by sentence, not one bibliography at the end. The IDs make the rest of the stack possible: without them, claim verification has nothing to attach a verdict to and a validator has nothing to check.

You answer strictly from the numbered context passages below. Each
passage begins with its ID in square brackets, for example [S1].

Rules:
1. Use only facts stated in the passages. Do not use outside
   knowledge, and do not fill gaps from memory.
2. After every sentence that states a fact, cite the passage IDs
   that support it, written as [S1] or [S1][S4]. Cite the smallest
   set of passages that actually supports that sentence.
3. If the passages do not contain enough information to answer,
   reply with exactly: NOT_IN_CONTEXT
4. Never answer partially and then guess. Either every factual
   sentence is supported, or the reply is NOT_IN_CONTEXT.

CONTEXT
{context}

QUESTION
{question}

ANSWER

Rendering that template should be the only path by which a prompt reaches the model.

from dataclasses import dataclass
from typing import Protocol, Sequence

# GROUNDED_ANSWER_TEMPLATE is the template text shown above.


class LLM(Protocol):
    def complete(self, prompt: str) -> str: ...


@dataclass(frozen=True)
class Passage:
    id: str      # "S1", "S2", ... assigned at retrieval time
    text: str


def render_context(passages: Sequence[Passage]) -> str:
    return "\n\n".join(f"[{p.id}] {p.text}" for p in passages)


def answer(llm: LLM, question: str,
           passages: Sequence[Passage]) -> str:
    prompt = GROUNDED_ANSWER_TEMPLATE.format(
        context=render_context(passages),
        question=question,
    )
    return llm.complete(prompt).strip()

Grounding catches invention when the evidence was present, and makes every claim auditable against a specific passage. It cannot catch anything caused by bad retrieval: when the evidence is absent, grounding yields a refusal at best and, at worst, a confident answer citing a passage that does not support it.

Layer 2 — Instruction in the prompt

The note uses “instruction tuning” in the practical sense of instructing the model in the prompt, not the training-time procedure that shares the name.

An instruction like “do not guess if it is not in the context” is the weakest of the five, and should be treated that way. It is a prior, not a constraint: nothing enforces it, compliance varies with the model and where the instruction sits, and a confidently phrased question can override it. It is still worth writing, because it costs nothing and raises the rate at which the system abstains rather than improvises.

What turns it from decoration into infrastructure is making the refusal a fixed token — NOT_IN_CONTEXT — rather than free prose. A fixed token is machine-checkable, countable, and routable. “Say you don’t know” produces a different apologetic paragraph every time, which no downstream check can distinguish from an answer.

Layer 3 — Fact-checking with a smaller verifier

Verification works at the level of atomic claims, not whole answers. A four-sentence answer with one unsupported sentence scores as vaguely-mostly-right at the answer level, which is unactionable; at the claim level it resolves into three supported claims and one to strike or regenerate.

Each claim gets one of three verdicts against the retrieved context: SUPPORTED, CONTRADICTED, or NOT_FOUND. The last two carry different policies. CONTRADICTED means the generator went against evidence it was given — block and regenerate. NOT_FOUND usually means the retrieval bucket from the framing section: the claim may be perfectly true, but this system cannot stand behind it, so it is stripped or converted into a refusal.

import json
import re
from dataclasses import dataclass
from typing import Literal, Sequence

Verdict = Literal["SUPPORTED", "CONTRADICTED", "NOT_FOUND"]
CITE_RE = re.compile(r"\[S(\d+)\]")

CLAIM_CHECK_TEMPLATE = """\
Context passages:
{context}

Claim:
{claim}

Decide how the passages treat the claim:
- SUPPORTED: a passage states or directly entails the claim.
- CONTRADICTED: a passage states the opposite.
- NOT_FOUND: the passages neither state nor deny it.

Reply with JSON only, no prose:
{{"verdict": "...", "passage_ids": ["S1"], "reason": "..."}}
"""


@dataclass(frozen=True)
class ClaimVerdict:
    claim: str
    verdict: Verdict
    passage_ids: tuple[str, ...]
    reason: str


def split_claims(answer_text: str) -> list[str]:
    """Split an answer into separately checkable claims.

    Citation markers are removed so the verifier judges the
    assertion, not the generator's own bookkeeping.
    """
    text = CITE_RE.sub("", answer_text)
    parts = re.split(r"(?<=[.!?])\s+", text.strip())
    return [p.strip() for p in parts if len(p.strip()) > 2]


def check_claim(verifier: LLM, claim: str,
                passages: Sequence[Passage]) -> ClaimVerdict:
    raw = verifier.complete(
        CLAIM_CHECK_TEMPLATE.format(
            context=render_context(passages),
            claim=claim,
        )
    )
    try:
        data = json.loads(raw)
        verdict = data["verdict"]
        ids = tuple(data.get("passage_ids") or ())
        reason = str(data.get("reason", ""))
    except (ValueError, KeyError, TypeError):
        # Unparseable verifier output is itself a signal.
        return ClaimVerdict(claim, "NOT_FOUND", (), "unparseable")
    if verdict not in ("SUPPORTED", "CONTRADICTED", "NOT_FOUND"):
        verdict = "NOT_FOUND"
    return ClaimVerdict(claim, verdict, ids, reason)


def faithfulness(verdicts: Sequence[ClaimVerdict]) -> float:
    """Share of claims the retrieved context actually supports."""
    if not verdicts:
        return 0.0
    ok = sum(v.verdict == "SUPPORTED" for v in verdicts)
    return ok / len(verdicts)

The cost is one verifier call per claim, which is why the verifier is the smaller model: batch claims, cache repeat questions, and run the full pass on the eval set with a sampled pass in production. It catches contradictions and quietly-added claims when the evidence was present. It cannot catch an answer faithful to a wrong document, and it cannot catch its own mistakes — the verifier is a model, and it is wrong sometimes too.

Layer 4 — The eval loop

A faithfulness number measured once is trivia. The value is in the series.

Log per answer: the faithfulness score, whether the system refused, whether the known-correct passage appeared in the retrieved set, and the claim count. Track distributions across releases and query segments, because aggregate scores hide segment-level collapse. Two populations are needed — a frozen evaluation set, so numbers stay comparable across changes, and a rolling production sample, so drift in the real query mix shows up. Every component change is a candidate regression: an embedding model, a chunk-size tweak, a reranker, a reworded prompt, a version bump. This layer turns the other four from beliefs into measurements, which is why it belongs early rather than last.

Layer 5 — Guardrails

Guardrails are two halves that get lumped together.

Prompt templates make the input shape invariant. One code path builds every prompt, so context is always ID-tagged, the abstention instruction is always present, and passages are always numbered the same way. Ad-hoc prompt strings assembled at three call sites drift apart, and that drift causes regressions nobody can reproduce.

Output validators check structural invariants deterministically, before any model call is spent on the response. These are the free catches: the answer is non-empty; the refusal token, if present, stands alone rather than buried in a paragraph; every cited ID exists in the retrieved set; every factual sentence carries a citation. A citation to [S7] when only S1 through S5 were retrieved is a fabrication caught by a regular expression and a set membership test.

from dataclasses import dataclass, field

REFUSAL = "NOT_IN_CONTEXT"
# CITE_RE, Passage and split_claims are as defined above.


@dataclass
class ValidationResult:
    ok: bool
    errors: list[str] = field(default_factory=list)


def validate(answer_text: str,
             passages: Sequence[Passage]) -> ValidationResult:
    """Deterministic checks: no model call, no network, no cost."""
    errors: list[str] = []
    text = answer_text.strip()
    known = {p.id for p in passages}

    if not text:
        return ValidationResult(False, ["empty answer"])

    if REFUSAL in text:
        if text != REFUSAL:
            errors.append("refusal must stand alone")
        return ValidationResult(not errors, errors)

    cited = {f"S{n}" for n in CITE_RE.findall(text)}
    unknown = sorted(cited - known)
    if unknown:
        errors.append(f"citations not retrieved: {unknown}")

    sentences = re.split(r"(?<=[.!?])\s+", text)
    uncited = [s for s in sentences
               if s.strip() and not CITE_RE.search(s)]
    if uncited:
        errors.append(f"{len(uncited)} sentence(s) uncited")

    return ValidationResult(not errors, errors)

Run this before the verifier. It is deterministic, it costs nothing, and an answer that fails it should never consume a verification budget.

Installing the layers on a running system

Order matters more than completeness. Steps one and two come first because skipping them means tuning the wrong layer.

1. Instrument. Log the question, the retrieved passage IDs and text, the rendered prompt, and the answer for every request. Nothing downstream is measurable without it, and it cannot be reconstructed after the fact.
2. Partition the existing failures. Sample bad answers and label each a retrieval miss or a generator error. The split decides whether effort goes into the retriever or the generation stack.
3. Move every prompt onto one template with ID-tagged passages, span-level citations, and a fixed refusal token. Highest-leverage single change, and it unblocks steps four and five.
4. Add the deterministic validator. No model calls, and it immediately catches citation IDs that were never retrieved and refusals wrapped in prose.
5. Add claim-level verification, first in shadow mode on sampled traffic so its own precision can be observed, then as a blocking gate once the false-positive rate is known.
6. Stand up the eval loop over a frozen set plus a rolling production sample, tracking faithfulness, refusal rate, and retrieval hit rate together, with correctness sampled on a slower cadence.

Failure modes

Abstention eats coverage

Tighten refusal far enough and the system stops answering questions it could have answered. Faithfulness looks excellent — refusals contain no unsupported claims — while the product quietly becomes useless. Refusal rate belongs on the same dashboard as faithfulness and must be read alongside it: a faithfulness gain that arrives with a refusal-rate jump is not a gain.

Verifier collusion

A verifier drawn from the same model family as the generator tends to share its blind spots. It ratifies the same misreading of an ambiguous passage, and the pipeline reports high faithfulness while making one error twice. A different model family, or an entailment-style checker rather than a general chat model, reduces the correlation. A periodic human audit of SUPPORTED verdicts is the only way to notice that verification has stopped being independent.

Citation theatre

Models learn quickly that citations are expected and attach them to sentences the cited passage does not support. The output looks rigorous, the validator passes because the IDs are real, and the citation means nothing. Structural validation cannot detect this; only claim-level verification against the cited passages can.

Hedged non-answers

Any faithfulness metric can be gamed by vagueness. “The documents discuss several considerations relevant to this topic” is trivially supported and answers nothing, and a metric optimised in isolation drifts in that direction. The countermeasures are a claim-density signal — how many concrete, checkable assertions an answer makes — and periodic human review scoring usefulness rather than support.

Closing

No single layer solves hallucination, and treating one as the solution is how the problem survives several rounds of work. Grounding constrains the generator, instruction nudges it, verification catches what slipped through, guardrails catch what was structurally broken, and the eval loop is the only reason to believe any of it works. The load-bearing habit is the one from the first section: for every failure, know whether the evidence was there. That answer decides which layer to touch.