AIs: Actual Idiots

Notice anything weird about that? As you might have been able to guess, that is completely AI, the output of ChatGPT when asked to 'write a paragraph about ai hallucination'. Although there is some truth in it, it isn't a completely unbiased or correct paragraph, nor does it sound very natural or use specific information. So, why do AIs (or, more accurately, LLMs) do this, how can you spot it, and what should you do if you have?

If you ask an LLM a question, it might give you something that has no source and it just makes up. This occurs often when the question is particularly niche (or if it doesn't make any sense) and so the internet doesn't really have any data on it. In fear of appearing stupid or ignorant, it fills in its response with fictitious "information". It results in an answer that may contain what few facts it can find or figure out, and so may seem genuine, but has really been fabricated on a whim. It will often also double down on its responses, refusing to admit that it was wrong, and on the occasion that it apologises, it then feels the need to concoct another made-up answer. This occurs due to the way that an LLM functions - it doesn't know what it doesn't know, it has no way to check its own uncertainty. It has been trained with rewards for sounding conversational and fluent, which results in incorrect, but helpful-sounding, responses.

Even if you ask an LLM a question that it can definitely find data and sources on, that doesn't mean it will be correct. As I have already stated, LLMs are trained to be helpful, and as well as sounding eloquent, this means they have to be fast. During Reinforcement Learning from Human Feedback (RLHF), LLMs are rated by users, and these users often prefer quicker, more concise responses. So each new iteration tends to get more and more confident, even when it's wrong. There also exists such a thing called the lost in the middle problem, which is the phenomenon of LLMs only looking at the beginning and end of a dataset and prompts. This results in, again, partially correct answers, but the critical information is ignored and it sounds plausible, but is wrong. Models often also look at patterns from their training, rather than thoroughly consulting a source. So even if you give an LLM some data that contains the exact answer, it may give you a response based on what it expects the answer to be.

LLMs, as you will (I really hope that you do) know, are not all-knowing beings that can always filter out wrong from right. Because the internet is filled with outdated facts, misinformation, disinformation, biased views and just plain errors, and they draw their data from the internet, responses can often include what looks to be backed up information, but is really, really, really not. Unfortunately, there also exists a vicious circle called model collapse, which is where LLMs use other LLMs' responses as sources, and eventually one error turns into an accepted fact. There are also, occasionally, people who try to corrupt LLMs. A very good (well, not good, you know what I mean) example of this is Microsoft's Tay chatbot. It was trained to mimic humans to make it sound more natural, but groups of users turned Tay into a misogynistic, racist and anti-semitic bot and Microsoft had to quickly shut down and apologise for it. Obviously this is just one example from a decade ago, but the same theory of LLMs trusting things they shouldn't still applies.

There are a few signs that AI-generated text may be incorrect. Ask yourself:

1) Does it sound too perfect? If you have a gut feeling that it's sounding helpful but not being helpful, that may well be the case.

2) Does it give sources? If it hasn't, ask for them, and check if what it's telling you is the same as what the sources tell you.

3) Does it contradict itself? If it doesn't have a source, it may well have incoherent parts.

4) Does it answer too completely? Real knowledge has gaps, so if every part is fully formed, there may be fabricated bits to fill them.

5) Does it give you exactly what you want? That's a sign of it trying to please you rather than it giving you the disappointing, complicated, shocking, weird truth.

You may have heard these before, but there are also reliable, if simple, ways to make sure you are not plagued by LLMs being stupid. These include:

a) Checking the source. Do the sources and response line up? Are they by reliable authors?

b) Cross-referencing. Does anyone else talk about this? Is it accepted?

c) Asking it to steelman the opposite view. If it can talk just as fluently and specifically about it, that indicates that it doesn't really have any knowledge on either side.

d) Looking for things that it doesn't say. If it does hallucinate, it will often try to make the entire response as complete as possible.

e) Seeing if it will correct itself. That shows that it's not sure and is just trying to please you.

Ultimately, AI hallucinations are not a bug that will simply be patched away — they are a fundamental consequence of how these models work. An AI generates what sounds right, not what is verified, and it does so with a consistency of tone that makes the false indistinguishable from the true. The responsibility that creates falls on the reader: to question what feels too complete, to check what sounds too precise, and to remember that fluency is not the same as accuracy. AI is a powerful thinking tool, but it requires a skeptical mind behind it to be a trustworthy one.

Well, that was from an LLM too. I hope you were able to realise that after reading this by spotting its defensiveness of AI, its lack of opinion and its odd tone. Stay safe on the internet, people.

Thanks for reading!

LegoBuilderOne