Fact-Checking & Spotting Hallucinations

Large language models are trained to predict plausible next text, not to verify facts against a database. Most of the time, plausible and true line up, because the training data mostly contains true statements. But when they diverge, the model has no internal alarm bell — it produces the false statement with exactly the same fluent, confident tone as a true one.

This is a structural property of how the technology works, not a bug that a future update will simply fix away. Understanding this reframes the whole problem: you're not looking for a tell that gives away a lie, because there isn't one at the surface level. You're building a habit that doesn't depend on detecting anything at all.

This is why "it sounded so sure" is not evidence of accuracy — confidence is a stylistic feature of the model's output, generated the same way regardless of whether the underlying claim is true.

Hallucination risk isn't evenly distributed. It clusters heavily around specific numbers and statistics, exact dates, citations and sources, niche or obscure facts, and anything after the model's training cutoff unless it actually searched the web.

By contrast, broad, well-established general knowledge and structural tasks (summarizing something you provided, rewording, brainstorming) are much lower risk, because there's less room for a specific, checkable detail to be subtly wrong.

Knowing this map lets you allocate your verification effort efficiently — you don't need to fact-check every sentence of every response, but you should treat any specific number, date, or citation as a flag worth checking before you repeat it anywhere that matters.

The core defensive habit has three steps. Source it: ask for a linked source rather than accepting a bare claim. Date it: check when that source is from, since a true-at-the-time stat can be outdated. Cross it: verify against a second, independent source or a second model run, especially for anything important.

This habit is deliberately simple because complicated verification processes don't survive contact with a busy day — a three-step habit you can run in under a minute is one you'll actually use consistently.

The habit works because it doesn't rely on your judgment about whether something "seems" true — it applies mechanically regardless of how confident or reasonable the claim sounds, which is exactly the point.

Asking for sources is only half the habit — the other half, which people skip constantly, is actually clicking through and checking that the source says what's claimed. A citation that doesn't exist, or exists but doesn't support the claim, is a common and specific failure mode.

This step takes thirty seconds but catches a real category of error: models sometimes generate citation-shaped text (a plausible-looking title, author, and link) without that source actually saying the thing attributed to it, or without it existing at all.

The rule of thumb: a source you haven't clicked isn't verification, it's just a more official-looking version of the original unverified claim.

A surprisingly effective and nearly free verification step is simply asking the model to critique its own previous answer: "what in your answer above is most likely wrong or outdated? What would you verify first?"

Models are often much better at spotting weaknesses in an answer when explicitly asked to look for them than they are at avoiding those weaknesses in the first place. This isn't a perfect defense, but it's a fast, low-cost second opinion that catches a meaningful share of errors.

This works because the self-critique prompt changes the task from "generate a plausible answer" to "generate a plausible critique," which surfaces a different, complementary set of considerations than the original generation did.

The right trigger for verification is the stakes of being wrong, not a gut feeling that something seems off. A brainstorm of possible blog post titles needs zero verification — being wrong costs nothing. A statistic going into a public presentation, a number in a document you'll sign, or a claim you'll repeat to a client needs full verification, every time.

This stakes-based framing solves the core problem with gut-feeling verification: hallucinations feel exactly like facts, so "does this feel right" is not a signal you can rely on. "What happens if this is wrong" is a signal you can rely on, because you can reason about consequences even when you can't detect falseness.

A simple practical filter: anything you'll publish, pay, sign, or repeat as fact to someone else gets the full 3-check treatment, no exceptions, regardless of how confident the answer sounded.

Models have a training cutoff, and without an active web search, they're answering from memory — which might be outdated for anything time-sensitive like prices, current events, or recent releases. It's often not obvious from the answer's tone alone whether it searched or recalled from memory.

The fix is to ask directly, or to explicitly instruct the model to search the web before answering anything time-sensitive. "Search the web and tell me the current price" produces a meaningfully different (and more reliable) answer than just "what's the current price," which might get answered from stale training data.

This distinction matters most for exactly the kinds of questions people ask casually and trust immediately — prices, current events, "is X still true" — where staleness is invisible unless you specifically check for it.