Detecting Hallucinations in LLM Outputs: A Practical Guide for Builders

Overview

Large language models (LLMs) occasionally produce fluent but unfounded statements—hallucinations. Detecting them is now a first‑class engineering task for anyone shipping LLM features in search, customer support, analytics, coding, or summarization. This article distills practical detection techniques, the signals behind them, and how to wire them into a real production pipeline.

What counts as a hallucination?

Hallucinations are outputs that are not supported by evidence or violate task constraints. Common forms:

• Factual: invented dates, figures, citations, or attributions.
• Logical: conclusions that contradict the provided context.
• Grounding errors: summaries that mention entities absent from the source.
• Procedural: confident instructions that misuse APIs, libraries, or commands.
• Spec non‑compliance: fabricating data when the spec requires abstention or a fallback.

Key principle: detection evaluates support, not eloquence. A confident, fluent answer can still be unsupported.

Why detection matters

• Trust and UX: false fluency erodes user confidence quickly.
• Risk and compliance: in healthcare, finance, and legal settings, ungrounded advice can be costly or harmful.
• Cost control: routing uncertain cases to humans prevents rework and incidents.
• Learning loop: flagged cases fuel continuous fine‑tuning and retrieval improvements.

A taxonomy of detection signals

Think in layers. Combining weak signals often beats relying on a single “magic” score.

1) Intrinsic uncertainty from the model

Available when you can access token probabilities or logits.

• Token‑level logprob/entropy: low average logprob or high entropy across the answer suggests uncertainty.
• Margin features: gap between the top‑1 and top‑2 token probabilities (small gap ⇒ ambiguity).
• Calibration metrics: expected calibration error (ECE) and Brier score help align confidence with reality. Pros: cheap, no external calls. Cons: poorly calibrated by default; confidence ≠ correctness.

2) Self‑consistency and variance across samples

Sample the model multiple times and compare.

• Disagreement rate: independently sampled answers that diverge semantically signal risk.
• Temperature sweep: answers that change with small temperature tweaks are less stable. Pros: model‑agnostic. Cons: raises latency/cost; consensus can be wrong.

3) Prompt‑based self‑verification

Ask the model to verify itself.

• Claim extraction + support judgments: “List atomic claims and label each as supported, contradicted, or unknown.”
• Counterfactual critique: “Provide reasons this answer could be false.”
• Chain‑of‑thought spot checks: critique the reasoning, not just the conclusion. Pros: simple to add. Cons: biased; models often over‑trust themselves—treat as one signal, not a verdict.

4) Retrieval‑augmented verification (RAV)

Check generated claims against sources.

• Evidence retrieval: search internal KBs, vector stores, or the web; constrain by domain and recency.
• Entailment over similarity: use an NLI (entailment/contradiction/neutral) model to test whether evidence entails the claim. Cosine similarity alone is insufficient.
• Coverage: compute fraction of claims with at least one entailing source. Pros: strongest factual signal. Cons: requires infra; retrieval quality bounds detection.

5) Tool‑assisted checking for specialized tasks

• Math/code: run‑time execution, unit tests, or solvers.
• Data ops: schema validators, SQL planners, and sandboxed execution.
• Policy/safety: deterministic rules for restricted content or PII.

6) Heuristics and content features

• Red flags: specific phrasings (“as of today” with no date), precise numbers without sources, fabricated citation formats.
• Out‑of‑scope detectors: classify when the question is unanswerable from provided context.
• Temporal claims: any “current” or time‑sensitive statement without a timestamp. Pros: fast and interpretable. Cons: brittle; tune per domain.

7) Human‑in‑the‑loop (HITL)

Route high‑risk cases to reviewers.

• Triage with score thresholds.
• Present evidence side‑by‑side for quick adjudication.
• Capture structured labels (supported/unsupported/needs‑more‑info) to improve models and thresholds.

Architecting a detection pipeline

A robust system blends pre‑generation safeguards, in‑generation controls, and post‑generation checks.

Pre‑generation

• Clarify or refuse: require the model to ask follow‑ups if the prompt is ambiguous.
• Retrieve first: gather context and instruct the model to only answer using provided sources.
• Set abstention contracts: “If unsupported, respond with ‘I don’t know’ and propose next steps.”

In‑generation

• Constrained prompting: explicitly require citations with every factual claim.
• Tool use: calculators, code runners, domain APIs.
• Evidence tracing: force the model to emit claim‑to‑source mappings as structured JSON.

Post‑generation

1. Extract atomic claims from the answer.
2. Retrieve candidate evidence per claim.
3. Score entailment per claim with an NLI model.
4. Aggregate to an answer‑level risk score with uncertainty features and heuristics.
5. Decide: accept, revise (ask the model to correct with evidence), or escalate to human.

A practical blueprint (code sketch)

from typing import List, Dict

class Detector:
    def __init__(self, llm, retriever, nli_model):
        self.llm = llm            # generation + verification prompts
        self.retriever = retriever  # search over KB/web
        self.nli = nli_model      # entailment/contradiction/neutral

    def extract_claims(self, text: str) -> List[str]:
        prompt = """
        Extract minimal, checkable factual claims from the ANSWER.
        Return a JSON list of strings, no commentary.
        ANSWER: {text}
        """
        return self.llm.json(prompt)

    def verify_claim(self, claim: str) -> Dict:
        docs = self.retriever.search(claim, k=5)
        scores = []
        for d in docs:
            label, conf = self.nli.entailment(d.text, claim)
            scores.append({"doc_id": d.id, "label": label, "conf": conf})
        # aggregate: best entailed conf minus best contradicted conf
        e = max((s["conf"] for s in scores if s["label"]=="entailment"), default=0)
        c = max((s["conf"] for s in scores if s["label"]=="contradiction"), default=0)
        return {"claim": claim, "entail": e, "contradict": c, "docs": scores}

    def risk_score(self, answer: str, verifications: List[Dict], token_stats: Dict) -> float:
        unsupported = [v for v in verifications if v["entail"] < 0.5 and v["contradict"] < 0.5]
        contradicted = [v for v in verifications if v["contradict"] >= 0.5]
        coverage = 1 - len(unsupported)/max(len(verifications),1)
        margin = token_stats.get("avg_top1_margin", 0.0)
        entropy = token_stats.get("avg_entropy", 0.0)
        # simple interpretable blend; in practice, fit a calibrated model
        score = 1.0 - (0.6*coverage) + 0.3*len(contradicted) + 0.1*entropy - 0.1*margin
        return max(0.0, min(1.0, score))  # 0=safe, 1=high risk

    def detect(self, prompt: str) -> Dict:
        out = self.llm.generate_with_stats(prompt)  # returns text + token stats
        claims = self.extract_claims(out.text)
        verifs = [self.verify_claim(c) for c in claims]
        risk = self.risk_score(out.text, verifs, out.token_stats)
        decision = "accept" if risk < 0.35 else ("revise" if risk < 0.6 else "escalate")
        return {"answer": out.text, "claims": verifs, "risk": risk, "decision": decision}

Metrics that matter

Evaluate at both claim and answer levels.

• Precision/Recall/F1 of hallucination flags at the claim level.
• AUROC/PR‑AUC for the risk score.
• FPR@95%TPR: how many clean claims you incorrectly flag at a high recall setting.
• Calibration: ECE and Brier score for predicted correctness.
• Coverage: fraction of claims with at least one entailing source.
• Abstention rate and utility: how often the system says “I don’t know” and whether users accept the fallback.

Useful evaluation datasets (task‑dependent)

• Question answering and open‑domain truthfulness: TruthfulQA.
• Claim verification: FEVER, SciFact.
• Summarization faithfulness: FactCC, QAGS, and human‑labeled contradiction sets.
• Dialog grounding: FaithDial and domain‑specific support chats. Use these to pre‑tune thresholds, then validate on your in‑domain data.

Setting thresholds and routing

• Start conservative: optimize for low false negatives in high‑risk domains.
• Per‑domain calibration: medical vs. e‑commerce need different cutoffs.
• Multi‑stage decisions:
  • Accept: high coverage, no contradictions, low entropy.
  • Revise: ask the model to regenerate citing specific evidence for unsupported claims.
  • Escalate: contradictions present, or temporal claims lacking dates.

Production logging and monitoring

Log enough to reproduce decisions:

• Prompt, raw answer, token‑level stats (avg logprob, entropy, margin).
• Extracted claims and their evidence doc IDs/URLs.
• NLI labels/confidences and retrieved snippets.
• Final risk score, decision path, and user feedback. Track over time:
• Hallucination rate per surface and intent.
• Coverage and contradiction trends by data source.
• Impact of model/version changes (A/B, canary deploys).

Common pitfalls and how to avoid them

• Over‑reliance on similarity: high cosine similarity ≠ factual support. Add entailment.
• No timestamping: require explicit dates for “current” claims and validate recency.
• Single‑model judge: self‑verification helps but should not be the only arbiter.
• Unverifiable domains: for forecasts or opinions, switch to compliance checking (“is this framed as opinion?”) rather than fact‑checking.
• Prompt injection (RAG): sanitize retrieved content and constrain tools; don’t let sources overwrite your policies.

Designing for correction, not just detection

When risk is moderate, attempt repair before escalation:

• Targeted rewrite: “Revise the answer. For each unsupported claim, either (a) remove it, or (b) add a citation with a quoted span and URL.”
• Cite‑and‑slice: regenerate per claim, conditioned only on retrieved evidence for that claim.
• Explanation requirement: block answers that contain numeric claims without a cited source.

Quick checklist

• Do you extract atomic claims and verify them with entailment, not just similarity?
• Do you compute and calibrate uncertainty features (entropy, margins)?
• Do you require timestamps for time‑sensitive statements?
• Do you have thresholds and routes for accept/revise/escalate?
• Are logs sufficient for replay and model audits?
• Is there a human review path for high‑impact cases?

Conclusion

Hallucination detection is best treated as a layered, probabilistic decision system—not a single classifier. Blend intrinsic uncertainty, retrieval‑based verification, structured self‑checks, and human review. Measure what matters (calibration, coverage, contradictions), and design for repair as well as refusal. With these pieces in place, you can push LLM features to production with confidence and clear guardrails.