The Things It Gets Wrong
A story about a church that never was
Marcus had been researching his family history for months—the kind of slow, satisfying work that turns names on a page into real people with real lives.
An AI assistant made it easier. It summarized census records, helped him organize notes, and suggested promising places to look next. It felt like having a tireless research partner.
Until one day it confidently told him about a church his great-grandmother attended.
The pastor’s name. The founding date. The red brick façade. The neighborhood it supposedly served. Everything sounded right—so right that Marcus could almost picture her walking through those doors.
The problem: the church had never existed. The AI had invented it entirely.
That moment is the point where a lot of people either swear off AI forever… or learn how to use it well.
Why AI can sound so certain and still be wrong
Imagine a friend who has read every book in the library but has never lived a day of real life.
They “know” about snow and coffee and heartbreak—but only through other people’s descriptions. They don’t have memories. They don’t have senses. They don’t have a body that noticed the room go quiet or the way someone’s eyes changed when they said, “I’m fine.”
When you ask this friend a question, they give you their best answer.
Sometimes it’s brilliant. Sometimes they fill in the gaps with plausible-sounding details—not because they’re trying to deceive you, but because that’s how their mind works: pattern completion.
That’s essentially what large language models do. They generate the most likely continuation of text based on patterns learned from enormous datasets. They can imitate research, reasoning, and expertise. But imitation isn’t the same as grounded knowledge.
And the scariest part isn’t that AI makes mistakes. It’s that the mistakes can be eloquent.
The “confident nonsense” problem shows up everywhere
Marcus isn’t alone. People in many professions have run into the same shape of error:
A teacher asks for citations and gets references to books that don’t exist—perfect titles, realistic authors, plausible publishers. A whole bibliography that evaporates when you try to verify it.
A nurse uses AI to explain a condition in patient-friendly language, and it’s genuinely helpful. But she also knows what the model can’t know: it can’t see a patient’s face, hear the tremor in their voice, or feel the tension in the room that signals something is being left unsaid.
An architect prompts for design concepts and receives stunning renderings—clean lines, elegant curves, a sense of modern calm. Then the architect notices the problem: the design violates basic physics, ignores load-bearing realities, or places staircases where no human could safely walk.
These aren’t edge cases. They’re normal outcomes of a tool that is excellent at generating what sounds right, and inconsistent at guaranteeing what is right.
The deeper limitation: it doesn’t know what it doesn’t know
Humans have a built-in feature that often gets overlooked: uncertainty.
We hesitate. We get that nagging doubt. We say things like, “I’m not sure,” or “Let me double-check,” or “That doesn’t sound right.”
AI doesn’t experience uncertainty the way people do. It can simulate uncertainty in language (“I might be wrong”), but that’s not the same as actually knowing when it lacks sufficient grounding. Without the right constraints, it can answer with confidence even when it’s improvising.
That’s why “confidence” is such a poor signal with AI. Polished wording, structured explanations, and decisive tone can all show up in an answer that’s completely fabricated.
A veteran fact-checker summed it up well: use what AI gives you as a starting point. Check the important things. Never assume confidence equals accuracy.
How to use AI without handing it the steering wheel
AI can still be a powerful tool in thoughtful human hands. The trick is to treat it less like an authority and more like an assistant in a busy workshop.
Here are a few practical ways to do that:
- Ask for sources you can verify. If it cites something, look it up. If you can’t find it quickly, treat it as untrusted.
- Use it to generate options, not final answers. Brainstorming, outlining, summarizing, reframing—these are strengths. High-stakes claims are where you slow down.
- Cross-check anything “too neat.” When details line up perfectly with a cinematic kind of clarity, that’s often a signal to verify.
- Bring in real-world anchors. People, primary documents, direct observation, professional judgment. AI can support those, not replace them.
- Keep a “curious skepticism” posture. Not cynical. Not paranoid. Just awake.
This mindset doesn’t make you anti-technology. It makes you a responsible user of a technology that’s still learning what responsibility means.
What Marcus found when he stepped off the screen
Marcus eventually found the real church.
It wasn’t a red brick building with a tidy founding date and a pastor whose name fit perfectly into the story. It was a small wooden structure—humble, local, and long gone. It burned down in 1943.
And the discovery didn’t come from a database.
It came from a phone call with his elderly cousin, the kind that starts with, “I don’t know if this helps, but…” and ends with the feeling that you’ve recovered something precious.
Marcus still uses AI. He just uses it differently now.
When it expresses certainty, he stays curious. When something feels too clean, he pauses. The technology didn’t change. He did.
Closing thought
AI at its best can expand our reach—helping us draft, explore, organize, and imagine. But wisdom is knowing where the tool ends and where human judgment begins.
Be open to what it can do. And be gently, consistently questioning about what it might be getting wrong.
