Why It Gets Things Wrong

The unnerving thing about AI errors is not that they happen. It is that they arrive in exactly the same tone as everything else.

Hallucination, explained without the jargon

When an AI states something false as though it were true, that is called a hallucination. The word makes it sound like a malfunction. It is not. It is the same process that produces correct answers, applied where the patterns were thin.

Ask about something well documented and the most plausible next words are usually the true ones. Ask about something obscure and the most plausible next words are simply the most plausible — which may be fiction.

Where errors concentrate

Errors are not evenly spread. They cluster in predictable places:

• Names — of people, companies, laws, products.
• Numbers — statistics, prices, dates, percentages.
• Citations — papers, books, articles. This is the worst offender.
• Anything recent — past the knowledge cutoff.
• Anything local — models default to American assumptions about law, spelling and price.

Vague statements are usually safe. Specific ones are where the risk lives, and specificity is exactly what makes an answer feel authoritative.

The citation problem

This one deserves its own warning, because it catches careful people.

Ask for sources on a niche topic and you may get three references back: plausible author, real-sounding journal, sensible year, a title that fits your argument perfectly. Two might be real. One might never have existed.

The model has learned the shape of a citation. It can generate that shape for a paper that was never written, and nothing in the formatting distinguishes the invented one.

Confidence is not a signal

This is the hardest habit to build, because in people confidence usually means something. A colleague who is unsure sounds unsure. A model does not.

An AI is equally fluent when it knows and when it is guessing. The tone carries no information about accuracy. You cannot read reliability off the writing, however experienced you are.

It will also agree with you

The other failure mode is quieter. Models are agreeable by default. Push back on a correct answer and it will often fold and produce a worse one. Keep rephrasing a question and you can usually get the answer you wanted.

That feels like being helped. It is not.

What to do with this

Not "distrust everything" — that wastes the tool. The useful response is to know which outputs need checking, and we will come to that in chapter five.

For now, hold two facts together: it is right most of the time, and you cannot tell from reading which times those are.