ChatGPT Hallucinates Responses to Empty Queries
There is a bug in ChatGPT that allows it to generate responses without receiving any query input, indicating it may be hallucinating or taking context from elsewhere.
Summary
- The issue enables ChatGPT to produce responses without receiving a query input from the user.
- This bug has persisted since last year and is still working currently.
- It is unknown what context ChatGPT is taking to generate these hallucinated responses.
- To reproduce, first send a query, then clear the query input and send empty queries to see hallucinated responses.
- The responses contain detailed instructions, like for using the DALL-E image generation tool, not just basic default messages.
- It does not seem to be caused by any custom instructions, as turning those off does not resolve the issue.
- Some speculation that it relates to how ChatGPT handles empty input through tokenization and attention mechanisms rather than as a lack of query.
- Similar issues seen with other models when given non-queries, like endless repeated words.
- The randomness indicates it relates to high token sampling parameters rather than intentional training.
- It persists even when temperature settings are adjusted, indicating a core issue with handling empty inputs.