When AI makes things up — and how to catch it

Ask an AI chatbot a question and you get an answer straight away: fluent, tidy, sure of itself. Ask it something it has no real answer to, and you get exactly the same thing — fluent, tidy, sure of itself, and sometimes completely made up.

That is the part that catches people out. Not that these tools get things wrong; everything gets things wrong sometimes. It’s that they get things wrong in the confident voice they use when they’re right. There’s no wobble, no “I think”, no “you might want to double-check this”. The invented answer and the correct one arrive dressed identically.

The industry even has a word for it: hallucination. It sounds like a rare glitch. It isn’t. It’s a normal side effect of how these tools work — so once you understand the how, you stop being surprised by it, and you start catching it.

It’s a prediction machine, not a knowing machine

Here’s the useful mental picture. An AI chatbot is not a librarian who looks things up and reports back. It’s much closer to the predictive text on your phone — the little suggestions above the keyboard — except vastly more capable.

It was built by being shown an enormous amount of writing, and from all that text it learned one skill extremely well: given some words, guess the words that most likely come next. That’s it. When you type a question, it isn’t searching a database of true facts. It’s generating the most plausible-sounding continuation, one word at a time, based on the patterns it has seen.

Most of the time that works beautifully, because true things are also common things — they show up again and again in all that text, so “the most likely next words” and “the correct next words” usually match. But when you ask about something rare, obscure, or genuinely unknown, those two part company. The tool doesn’t have a real answer to give you. And crucially, it doesn’t stop. It does the only thing it knows how to do: it produces the most plausible-sounding answer anyway.

A plausible-sounding answer to “what’s a good book about X?” is a book title. So it writes one. It might be real. It might be a title that has never existed, by an author who has never existed, described in perfectly convincing terms. The tool has no way to feel the difference, because it was never checking whether the book is real — only whether the sentence reads like a real recommendation.

That’s the whole trick, and once you see it, it can’t un-see itself: the machine is built to sound right, not to be right. Being right is often a happy by-product. It is not the goal.

Why the confidence fools us

Left to ourselves, we use confidence as a shortcut for competence. Someone who answers quickly and smoothly feels like they know their stuff; someone who hedges and pauses feels less sure. It’s a fair instinct with people, because a person who’s guessing usually gives off signals — a hesitation, a “I’m not certain, but…”.

An AI gives off none of those signals. It is equally smooth when it’s quoting a real fact and when it’s inventing one. So the one cue you’d normally rely on — does this person sound sure? — is exactly the cue that stops working. The fluency isn’t evidence the answer is good. It’s just the tool doing what it always does.

This is worth saying plainly because it flips the usual worry on its head. People often ask, “how do I know when the AI is unsure?” You mostly can’t. So the safer habit isn’t to look for signs of doubt — it’s to bring the checking yourself.

The ten-second habit

You don’t need to become suspicious of everything, and you don’t need to stop using these tools — they’re genuinely useful. You just need one rule of thumb:

Treat the AI as a fast, clever assistant who occasionally makes things up with total confidence — and check anything that actually matters before you rely on it.

“Matters” is the filter. If you’re brainstorming birthday ideas or rephrasing an email, a made-up detail costs you nothing — use it freely. But if the answer is a fact you’ll repeat, a name, a number, a date, a legal or medical or money question, or a link it wants you to click, that’s where a made-up answer can cost you something real. That’s where you check.

Try this the next time it matters. Ask the chatbot, “How do you know that — what’s your source?” Then do the part it can’t: go and confirm the source actually exists and actually says what the AI claims. Don’t take the source on trust either — a tool that can invent a fact can just as easily invent the study, the article, or the web address it credits. One quick look at a real, independent source settles it. If you can’t find any real source, treat the answer as a guess, because that’s what it was.

That check takes seconds, and it does two things. It catches the inventions before they reach anything important. And it slowly retrains your own instinct, so that “this sounds authoritative” stops being enough on its own — from anyone.

Where it shines, where to slow down

To keep it practical, here’s roughly where the line falls.

Lean on it happily for: drafting and rewording, brainstorming, explaining a tricky idea in plainer terms, summarising a long document that you paste in yourself, or getting unstuck on a blank page. In all of these, you’re the one supplying or judging the material — the tool is helping you think, not feeding you facts to swallow.

Slow down and verify for: specific facts, figures, and dates; quotes and who said them; anything recent or fast-changing; names of products, laws, or people; links and instructions to log in somewhere; and any question where being wrong has a cost. These are exactly the spots where a plausible-sounding invention slips through easiest.

The mindset that keeps you in charge

It helps to drop the idea that the AI is “lying” when it makes something up. Lying needs a liar who knows the truth and hides it. This tool doesn’t know what’s true — it has no notion of true at all. It’s arranging likely words. Calling it a liar gives it a mind it doesn’t have, and misses the real lesson.

The real lesson is quieter and more useful: the judgment stays with you. These tools are wonderful at producing a fast, fluent draft of an answer. Deciding whether that answer is actually correct — whether to trust it, send it, act on it — is a human job, and it always was. Used that way, an AI makes you quicker and sharper. Trusted blindly, it makes confident mistakes in your name.

So enjoy the speed. Just keep your hand on the wheel. The tool sounds certain for free; being right is the part you bring.