Hallucination

In the field of artificial intelligence (AI), hallucination or artificial illusion (also called fiction or delusion) is an AI-generated response that contains false or misleading information presented as facts. Generally speaking, it refers to the phenomenon that the content generated by the model is inconsistent with real-world facts or user input. How to effectively filter expert knowledge and use high-precision expertise for reinforcement learning to achieve iteration and updating of large models is an important method to eliminate scientific hallucinations.The illusion and emergence of large models are the same principle.

Hallucinations in Natural Language Processing

In natural language processing, hallucination is usually defined as "generated content that is meaningless or unfaithful to the provided source content". There are different ways to classify hallucinations: intrinsic and extrinsic, depending on whether the output contradicts the source or cannot be verified from the source; and closed-domain and open-domain, depending on whether the output contradicts the prompt.

Causes of Hallucinations in Natural Language Processing

There are several reasons why natural language models can hallucinate data.

• The illusion of data

The main cause of data hallucination is source-reference disagreement. This disagreement occurs either (1) as a product of heuristic data collection or (2) because the nature of some NLG tasks inevitably includes such disagreement. When a model is trained on data with source-reference (target) differences, it can be encouraged to generate text that is not necessarily well-founded and faithful to the provided source.

• Illusions created by models

Hallucinations have been shown to be a statistically unavoidable byproduct of any imperfect generative model that is trained to maximize the likelihood of hallucinations, such as GPT-3, and that active learning (such as reinforcement learning from human feedback) is required to avoid hallucinations. Other studies take an anthropomorphic perspective, arguing that hallucinations are caused by the tension between novelty and usefulness. For example, Teresa Amabile and Pratt define human creativity as the generation of novel and useful ideas.

Encoding and decoding errors between text and representations can lead to hallucinations. When the encoder learns false correlations between different parts of the training data, it can lead to false information being generated that is different from the input. The decoder takes the encoded input from the encoder and generates the final target sequence. Two aspects of decoding can lead to hallucinations. First, the decoder may process the wrong parts of the encoded input source, leading to incorrect generation. Second, the design of the decoding strategy itself can lead to hallucinations. Decoding strategies that increase generation diversity (such as top-k sampling) are positively correlated with increased hallucinations.

References

【1】https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)