Confident isn't correct: how to check an AI answer
Checked
Ask a chatbot a question and you usually get a smooth, confident answer in seconds. Often it's right. Sometimes it's wrong, and it sounds just as sure. The fix isn't to stop using AI. It's to know when to check, and how.
The examples below quote help pages from OpenAI, which makes ChatGPT, and Google, which makes Gemini. They are examples, not a ranking or an endorsement. The same habits work with any AI tool.
Why a chatbot can be wrong and sound sure
Chatbots are built to predict likely words based on patterns in the material they were trained on. NIST, the National Institute of Standards and Technology, is part of the U.S. Department of Commerce and develops measurements, tools and standards for AI. It explains that this kind of prediction can produce accurate answers, but it can also produce answers that are inaccurate or that contradict themselves.
NIST's name for the problem is "confabulation": "the production of confidently stated but erroneous or false content," which people often call hallucinations. NIST says confabulations "are a natural result of the way generative models are designed." In other words, this isn't a rare glitch that one company will fix next month. It comes with how the tools work.
The companies say so too. OpenAI says ChatGPT can give incorrect definitions, dates or facts, and can make up quotes, studies, citations and references to sources that don't exist. Its help page puts it plainly: "Confidence isn't reliability." Google's Gemini privacy page says AI tools like Gemini "can hallucinate and present inaccurate information as factual."
When it matters most
Plenty of AI use is low stakes. If a chatbot suggests a so-so dinner recipe or an awkward first draft of an email, the cost of a mistake is small.
The stakes go up when you might act on the answer: money, taxes, health, legal rights, a school assignment or anything you'll sign. NIST warns that risks can arise when people believe false content, often because it sounds confident, and then act on it or pass it along.
A case from the FTC (Federal Trade Commission) shows what can go wrong when an AI tool is sold as a stand-in for a professional. DoNotPay promoted its online subscription service as "the world's first robot lawyer." The FTC charged that the company didn't test whether its "AI lawyer" performed at the level of a human lawyer, and didn't hire or retain attorneys to test the quality and accuracy of its law-related features. In January 2025, the FTC approved a final order, announced on February 11, 2025. It requires DoNotPay to pay $193,000, to notify people who subscribed between 2021 and 2023, and to stop advertising that its service performs like a real lawyer unless it has enough evidence to back that up.
A chatbot can help you understand a letter from your landlord or a medical term. It can't take responsibility for the answer. You still have to.
A three-step check
1. Ask where it came from. Ask the chatbot for its sources. Some chatbots can search the web and show links. OpenAI says that with search turned on, ChatGPT can access and cite web sources for more recent information.
2. Open the source yourself. A citation can be invented, or it can be real but not say what the answer claims. OpenAI's advice is to "Always verify quotes, data, technical information or references to external documents," and to check sources "by visiting links directly." Find the sentence that supports the answer. If you can't find it, treat the answer as unconfirmed.
3. Confirm with the official source. For rules, deadlines and dollar amounts, go to whoever is actually in charge: the government agency, your bank or plan, your doctor's office or your school. Use a website or phone number you already trust, not one the chatbot made up.
Our rule of thumb: treat an AI answer like a tip from a friend of a friend. It's a useful lead, never proof.
Do this now: Next time you get an answer you plan to act on, ask "What's your source?" and then open the link.
Money answers get an extra check
Money rules change from year to year. OpenAI notes that its models are trained on data up to a certain point and don't know about later events unless a tool like search is used. An answer about this year's contribution limit or tax deadline may be out of date, so check the year on the official page.
Be especially careful with anything that sounds like a promise. The SEC's Office of Investor Education and Advocacy, the North American Securities Administrators Association (NASAA) and the Financial Industry Regulatory Authority (FINRA) jointly warn investors to be wary of claims, even from registered firms and professionals, that AI can guarantee amazing investment returns. Before acting on any investment idea, from a chatbot or anyone else, read our piece on how to check who you're dealing with.
Don't ask a chatbot who wrote something
Parents and teachers sometimes paste in an essay and ask, "Did AI write this?" OpenAI says ChatGPT "can't reliably determine whether text was AI-generated or whether it wrote the text," and that it may make up answers to questions like that. In OpenAI's words, "Those answers are not reliable evidence of authorship." A conversation with the student about how they wrote the piece will usually tell you more.
Do this today
Pick one AI answer you've used recently, about money, health or school, and run the three-step check. If it holds up, good. If it doesn't, you've learned something about the tool before it cost you anything. For what to keep out of a chat in the first place, see What to tell a chatbot, and what to keep to yourself.
Education only, not financial, legal or tax advice. Not a registered investment adviser.
Sources
Primary sources
- NIST AI 600-1: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile (PDF), July 2024, date checked: Oct 22, 2026
- OpenAI: does ChatGPT tell the truth?, help article, date checked: Oct 22, 2026
- FTC: FTC finalizes order with DoNotPay that prohibits deceptive "AI lawyer" claims, press release, Feb 11, 2025, date checked: Oct 22, 2026
Supporting links
- SEC Investor.gov: artificial intelligence (AI) and investment fraud, investor alert, Jan 25, 2024, date checked: Oct 22, 2026
- Google: Gemini Apps Privacy Hub, date checked: Oct 22, 2026