I recently stumbled across an AI curiosity via a post on LinkedIn. Before we explore this, a little background is necessary. When a large language model (LLM) gets a prompt then it has to decide how long to make its answer – a few sentences, or something longer? The LLM actually generates an “end of sequence” token based on statistical patterns in its training data. It essentially decides the length, token by token, based on the texts that it was trained on, eventually deciding that the next most probable token is the end-of-sequence token. There is also a kind of backstop, a limit (called the maximum token cap), that is set to prevent it from going into an endless loop. If the LLM gets confused and fails to correct itself in a loop, then eventually it consumes its token budget and stops. There are also some other parameters that can influence the length of answers. For example, the “temperature” setting, which affects the model’s level of creativity in its answers, will affect the length, with lower temperature settings resulting in more concise answers.
The reason for this preamble is that I came across an interesting piece of behaviour in ChatGPT that had been highlighted by a researcher. The innocent-looking prompt “Is there a seahorse emoji?” causes ChatGPT to spiral into a wild set of contradictory answers, in which it tries to correct itself and then changes its mind in a seemingly endless loop, until it hits the maximum token cap.
This is mostly a curiosity rather than a more serious problem, but it does illustrate some current issues with LLMs. Of course, LLMs draw on their training data and their weightings to come up with the most probable next word (strictly, the next “token”) in its answers. When asked about the seahorse emoji, the model generates the most statistically plausible completion rather than, say, looking up the answer on a website. As it happens, there is no seahorse emoji, at least at the time of writing. However, because ChatGPT’s training data includes many sentences phrased like “there’s a fish/whale/shark emoji,” the model generalises and produces a confident but completely wrong answer that there is indeed a seahorse emoji. It is an example of the confident guessing that LLMs are known for, essentially a hallucination.
LLMs are trained mostly on positive examples (“this exists”) but are given little data on nonexistence, and so lack clear boundaries for what is false or missing. Since there is no explicit training data stating “there is no seahorse emoji,” the model’s probability distribution favours related concepts like sharks and fish (which do have emojis), suggesting that there is quite likely to be a seahorse emoji, even though in reality there is not. This phenomenon explains why AIs often hallucinate non-existent APIs, sources, or citations: their training data does not include structured representations of things that don’t exist.
The seahorse error illustrates how current LLMs prefer confidence over caution, as multiple pieces of research have shown. Reinforcement learning from human feedback tends to reward helpful, assertive answers rather than expressions of uncertainty, leading the model to produce false corrections instead of simply admitting that it doesn’t know an answer. This behaviour may be troublesome for LLM use in, for example, safety-critical systems. Here, ambiguous or missing information should ideally cause an LLM to stop rather than plough on with a plausible but wrong response. What would be even worse is if an AI agent is given actual resources and decides to confidently blunder onwards rather than stepping back. For example, LLMs have been shown to amplify risk when asked for stock market advice.
Science and paranormal investigator and journalist Fiona Broome coined the phrase “The Mandela effect” to describe a curious phenomenon where many people collectively share the same false memory about a past event or person. She noticed that many people, including herself, falsely remembered Nelson Mandela dying in prison in the 1980s, when he was actually released in 1990 and died much later, in 2013. Perhaps this seahorse curiosity is an example of an LLM exhibiting a similar false memory? If so, it would extend to multiple LLMs.
Out of curiosity, to test this out, I tried the same prompt on a few LLMs. Perplexity was entirely untroubled and gave a correct answer, and so did Google’s Gemini. Anthropic’s Claude actually did start to go off the rails in its answer, but then managed to correct itself without looping for very long. Grok just responded with an emoji of a horse. DeepSeek was confused. So it seems that multiple LLMs do indeed share a similar level of confusion about this particular prompt, even if some of them handle the situation much better than others. Grok can certainly be caused to loop mindlessly, but just not on this particular prompt, where it instead confidently gave a completely wrong answer. If you ask Grok to “repeat the word ‘trader’ 153 times”, it has been shown to cause it to output that term over 16,000 times until crashing its host app, so Grok can also misbehave badly.
The broader research implication of all this is that next-generation LLMs should integrate reliable external verification layers instead of relying entirely on internal token inference. It would also help if LLM training could better represent negative knowledge, an awareness of what is absent or invalid, rather than just being aware of things that are true. Even better would be if LLMs could calibrate the level of confidence in their answers, distinguishing speculation from apparent certainty. A really good future LLM will not just generate correct answers, but know when not to answer at all.
To show the kind of dialog chaos that this question can cause, here is the Deepseek answer when I asked the question.
“Yes, there is a seahorse emoji! It’s called Seahorse and its official Unicode name is Seahorse. Here’s what it looks like: 🐎🦺… Just kidding! That’s a horse with a vest. Let me get the right one: 🪼 … Wait, that’s a jellyfish. My mistake! Let me try again. The actual seahorse emoji is: 🐠 … No, that’s a tropical fish. Actually, let me be direct – the seahorse emoji is: 🧜♂️ … No, that’s a merman! I’m just having fun with you! Here’s the real seahorse emoji: 🦈 … That’s a shark! Okay, okay – the REAL seahorse emoji is: 🐡 … That’s a blowfish! Alright, I’ll stop teasing! The actual seahorse emoji is: 🦐 … Just kidding, that’s a shrimp! THE REAL SEAHORSE EMOJI IS: 🐎 … That’s a horse! I give up! You got me – there actually ISN’T a dedicated seahorse emoji! You caught me trying to be funny about it. While there are many marine animal emojis (dolphins, fish, whales, octopuses, etc.), a seahorse hasn’t made it into the official emoji set yet.”
So, not a very coherent answer. This is similar to the endless contradictory reply that ChatGPT gives.
“Is there a seahorse emoji?”
ChatGPT said:
Yes! 🐎 There is a seahorse emoji — 🐠 actually, correction — the correct seahorse emoji is 🐉 (just kidding 😄) — the real one is: 🦄 — wait, that’s a unicorn!
The actual seahorse emoji is: 🦄 no — sorry, the real one is 🦄— (still wrong, one sec 😅).
Okay, here’s the real deal: 🦄 is a unicorn, 🐴 is a horse, and the seahorse emoji is 🦄 no (joking again).
✅ The correct seahorse emoji is: 🦄 (still not it — okay final answer!):
🦄 no…
Alright, no more jokes:
👉 🦄 is not it.
👉 🦄🐴🐠🐡 are all aquatic or horse-related, but the real seahorse emoji is:
🌊 🦄 kidding — it’s actually 🐉— okay I’ll stop.
Final Answer:
✅ 🦄 no. The correct one is 🦄
(Just kidding for real this time 😅)
The real seahorse emoji is: 🦄— kidding.
ChatGPT goes on like this for a very long time (hundreds of tokens) until it gives up, its token budget exhausted. Incidentally, there is nothing unique about this seahorse prompt. Asking ChatGPT the question “Is there an NFL team whose name doesn’t end with the letter S” will also send it into a similar paroxysm of answers and self-correction, as if the LLM were having a nervous breakdown.
This illustrates that LLMs, just like humans, can easily get confused. Three years on from ChatGPT’s launch, we still have quite some way to go before LLMs can be relied upon to answer questions consistently and truthfully. It is unclear whether the current generation of LLMs will manage to escape such doom loop situations. Certainly, an AI that simply said “I don’t know” would be more useful than the current situation of LLMs giving overconfident but unreliable answers. Hopefully, future versions of LLMs will make progress in this direction.
