The short version
A generative model is a very large statistical model of language. Given some text, it predicts what should plausibly follow, one piece at a time, and repeats until the answer is complete. Everything it produces is generated in the moment rather than retrieved from a store of facts.
The model is producing the most plausible continuation, not the true one. When those coincide, the results are excellent. When they diverge, the model states something wrong in exactly the same confident register as everything else.
Why it is good at language and bad at facts
Fluency and accuracy are separate problems. Grammar, tone, summarising a long document, rewriting something aimed at engineers so a customer can follow it — these are pattern problems, and the models are genuinely better at them than most people.
Specific facts are a different matter. Your customer’s account balance, the clause in your standard contract, last month’s delivery figures — none of that was in the training data. So it produces something shaped like the right answer. This is usually called hallucination. It is not a malfunction: it is the system working exactly as designed, applied to a question it should never have been asked unaided.
The engineering answer is to stop asking the model to remember things and instead hand it the relevant material at the moment of the question. That approach is retrieval-augmented generation.
What it is actually useful for
| Task | Why it suits generative AI |
|---|---|
| Summarising documents and threads | The source material is supplied, so accuracy is checkable |
| Drafting routine correspondence | A person reviews before sending; drafting is the slow part |
| Extracting structured data from documents | Output has a rigid shape, so errors are easy to validate |
| Classifying and routing inbound work | Accuracy is measurable against a sample |
| Answering questions from your own documents | Works well only when grounded in retrieved material |
Three failure modes to design around
- Confident invention. Ground every factual answer in supplied material, and show the user which source it came from.
- Inconsistency. The same question can produce differently worded answers. Where structure matters, constrain the output format and validate it.
- No sense of authority. The model cannot tell your approved policy from a draft somebody abandoned in 2019. Curating what it can see is part of the build.
If your input is a table of numbers and you want a prediction, classical machine learning is cheaper. If the task requires several steps and taking action, what you need is an AI agent. For the wider picture, start with AI for business.