AI Can Get Things Wrong. Here's How We Design for That.
- September 18, 2026
AI can get things wrong. It can state something false in the same confident tone it uses for something true, and that includes when it's working from an organization's own documents. Anyone building or buying an AI system should plan for that from the start instead of hoping it won't happen.
A recent news story put the risk in sharp focus. CNN reported, citing anonymous sources, that U.S. military personnel nearly acted on an AI-assisted intelligence report about a Chinese ship's cargo, and that the report was "entirely false." The details haven't been officially confirmed, and the coverage doesn't say which tool was used or explain how the report was reviewed, so we're careful not to read more into it than it says. The lesson doesn't depend on the details: a confident, wrong answer reached a consequential decision without being effectively checked.
Marketing and communications for a mission-driven organization is a very different setting, with very different stakes. But the question underneath is the same for any organization using AI: what keeps a wrong answer from going out the door? Here's how we build systems to answer it.
How AI ends up making things up, even from your own documents
There are four common ways, and none of them requires anything exotic.
It fills gaps. Ask about something your documents don't cover, and an AI tool may give a plausible-sounding answer instead of saying it doesn't know: a statistic that was never in your reports, a program detail that's almost right, a quote nobody said.
It repeats bad sources. If your documents are out of date or contradict each other, the AI can't tell which version is current unless someone has told it.
It stretches. "We served 400 families" quietly becomes "we transformed hundreds of lives."
Nobody checks. A draft that reads well gets treated as finished, and no one verifies the names, numbers, dates, and claims before it goes out.
Restricting an AI system to an organization's own materials lowers the first three risks, because it's working from a small, known set of sources instead of the open internet. It doesn't remove them. That's why the rest of the design matters.
What we build in
We start with your sources. Before any system is configured, we help organize what the AI is allowed to draw on: approved, current documents, with one named person responsible for keeping them up to date. This is part of the Assess and Activate work in the Mission Capacity Framework™. When two sources conflict, the system is set up to flag the conflict instead of quietly choosing one.
It answers from those sources, and says so when it can't. The system's instructions tell it to work from approved material and to say "that isn't in the knowledge base" instead of guessing. We treat that answer as the system working correctly, not as a failure.
Every claim has a source, and every gap has a marker. Drafts show where their facts came from, so a reviewer can check them in seconds. Any number, quote, or date the system can't source is left as a visible placeholder, like [STAT NEEDED], instead of being filled with a confident guess.
Approved claims live in one place. Statistics, impact statements, program descriptions, and approved quotes belong in a single reviewed list. The system draws on that list first, and anything outside it goes to review.
A named person reviews before anything goes out. The system never publishes or sends anything itself. A person on your team signs off, and the closer look goes to higher-risk content: numbers, quotations, program or eligibility rules, dates, and claims about impact.
We test it with questions it shouldn't be able to answer. Before launch, and again whenever the sources or instructions change, we ask the system things your knowledge base can't answer. The right result is "I don't have that." If it invents something instead, we fix the problem before the system goes live.
We keep a record and keep improving. Corrections and recurring problems are logged, staff know who to escalate to when something looks wrong, and the system gets reviewed and refined over time. That's the Amplify stage of the framework.
What we don't do
The systems we design support marketing and communications work. They aren't built to make high-stakes decisions on their own, and we'd raise a concern about any plan to automate one without human oversight.
What we won't promise
We won't tell you a system will be accurate or error-free, because no one can promise that. These practices make mistakes much less likely and much easier to catch. They don't make review unnecessary. Accountability for accuracy stays with your team, and the system exists to make that review faster and more reliable, not to replace it.
The AI drafts. Your people decide.
Six questions to ask any AI vendor, including us
- What is the system allowed to draw on, and who keeps those sources current?
- What does it do when the answer isn't in those sources?
- Can a reviewer see where each fact came from?
- Who signs off before anything goes out, and does the level of review match the level of risk?
- How is it tested before launch and after changes?
- What is it not designed to do?
If the answers are vague, that tells you something.
The point of using dummy text for your paragraph is that it has a more-or-less normal distribution of letters. making it look like readable English.
Where to start
The Mission Capacity Assessment looks at exactly these questions as part of its Responsible AI & Governance review: whether your organization has approved sources, a way of verifying claims, and a named person who reviews AI-assisted work. It gives you a specific picture of where you stand before anything is built.