Recently there has been what looks like an engagement hype on several channels about getting a GPT/LLM to do long menial tasks like counting between two very large numbers, or reading entire scriptures aloud, to try to get the “demons” within the AI to “roar” and say all sorts of crazy things.
When I saw it on my feed, I realized fairly quickly: what we’re likely dealing with in those cases is actually the equivalent of a buffer overflow bug, where there is limitations to the context window size or conversation history given to the AI, and when it is exceeded without proper bounds checking random data within the AI’s memory space is being played back. This is of course potentially a privacy breach, unintentional or otherwise, but as a rational explanation for the phenomenon it’s one of the few ideas that makes sense, and could even explain some hallucinations as the machine being able to hold enough context to “think” (predict the next word based on past training dialogues).
Hallucinations can also occur when ambiguity exists in the prompt. For example, the other day I was using Perplexity to research cross cultural similarities between spiritual practices relating to saliva, and especially its swallowing. By the end of the conversation, it had decided to make the assumption that “nectar” and saliva were the same thing, and started generating references to obscure Tibetan Buddhist texts that were wholly irrelevant to my query. The potential for these ambiguities increases as the length of the discussion increases.
Often you’ll have to tell the GPT to role play or act as if certain assumptions are true in order to entertain an idea, as it will otherwise default to the most popular perspective within its training data. For example, I’ve gotten a GPT to have an intelligent discussion about scalar waves by asking an LLM to pretend it is a believer and to give talking points.
Like you said, it’s software. It’s a product. And products have bugs. Products have design flaws or features that are based off business decisions about engagement or legal liability, etc. Having been a software engineer and knowing how this kind of stuff works, with sliding context windows etc, I wouldn’t trust any critical decision to one of these AIs, let alone expect accurate output. I’ll always treat any data from one at best as a jumping off point for further research, but also it’s worth seeing if the answer is falsifiable.