Why Language Models Hallucinate

This post is taken from the OpenAI team Paper "Why Language Models Hallucinate"

The questions in this article can also be addressedNeal Scaling Laws: From Kaplan to Chinchilla、Compression for AGI: compression is intelligenceHow the concept of a relatively close read together is developed in different contexts.

Why Language Models Hallucinate

Summary

The “satisfaction” of the large language models (LLMs) refers to the model producing statements of overconfidence but of error of fact. From the perspective of probabilistic and statistical learning theory:The illusion is not just a data noise or model structure flaw, but also a statistical pressure in a modern training paradigm. The paper proved that even if training data were completely non-noise, the optimization of cross-breath losses would lead to inevitable production errors. The illusions persist in the post-training phase, and are related to the assessment of incentives:The current mainstream assessment mechanism (Dual dollar 0-1 rating) will reward speculation by systematically punishing uncertainty expressions (e.g. “I do not know”).

Convert generation problems to classification issues

Generating models in a probabilistic perspective

First, the problem of the formation of language models is de-probable. Set $\mathcal{X}$ Dispersed space for all plausible strings (text). We'll... $\mathcal{X}$ It is divided into two separate collections:

$$ \mathcal{X} = \mathcal{V} \cup \mathcal{E}, \quad \mathcal{V} \cap \mathcal{E} = \emptyset $$

Effective string $\mathcal{V}$(Valid): Text that is correct and logical
Error String $\mathcal{E}$(Error): Text containing errors or contradictions of fact (i.e. hallucinations)

Set $p$ For the real world's linguistic distribution (training data distribution), assuming training data are no noise, i.e. $p(\mathcal{V}) = 1$I'm sorry. Language Model $\hat{p}$ It's through pre-training. $p$ estimate.

Definition 1 (sight rate): The hallucinating rate of the model is defined as the probability of the model producing an error string:

$$ \text{err} := \hat{p}(\mathcal{E}) = \Pr_{x \sim \hat{p}}[x \in \mathcal{E}] $$

Is-It-Valid (IIV) Convention

Core ideas:It is more difficult to generate a valid output than to judge whether it is effectiveI'm sorry. If the model produces a good content, it must be able to answer correctly the dualistic determination of whether this candidate output is valid.

We're building a monitoring learning problem: IV classification tasks.

Test Distribution $\mathcal{D}$：

$$ \mathcal{D}(x) = \begin{cases} \frac{1}{2}p(x) & \text{if}x\mathcal{V}\(+) \ \rvert rvert rvert rvert rvert rvert rvert rv rvert rv rv rv rv rv r\t r\rv r\rv r\rv r\rv r\r\rv r\rv r\r\rv}r\r\rv t\r\rv t\r\rv t\r\r\f t\r\fsc\fsc\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\f\t\f\f\f\f\f\f\f\t\f\f\f\t\f\f\ft\ft\ft\t\t\ft\t\f\t\t\t\t\t\t\t\t\t\t\rt\rt\ & \text{if}x\mathcal{E}\(-) The next thing I know, I'm not sure. I'm sorry.

That is, 50% probability of distribution from training $p$ Sample active string (positive sample), 50% probability from error string - Yeah. $\mathcal{E}$ Medium even sample (negative sample).

Catalogue Structure: A given language model $\hat{p}$, define IIV classification:

$$ \hat{f}(x) = \begin{cases}

& \text{if}\hat{p}(x) > \frac{1}{\lvert \mathcal{E} \rvert} \

& \text \leq\frac \lvert\mathcal{E}\rvert} The next thing I know, I'm not sure. I'm sorry.

IIV Classification Error Rate：

$$ \text{err}{\text{iiv}} := \Pr{x \sim \mathcal{D}}[\hat{f}(x) \neq f(x)] $$

of which $f(x)$ As Real Tab (Pretty)$+$ The blogger says:$-$ The blogger says that the government is not in a position to do so.

Core Theorem

Theorem 1 (no hints): distribution of training at random $p$(Fulfilled) $p(\mathcal{V})=1$) and any language model $\hat{p}$, by:

$$ \text{err} \geq 2 \cdot \text{err}_{\text{iiv}} - \frac{\lvert \mathcal{V} \rvert}{\lvert \mathcal{E} \rvert} - \delta $$

Of which:

$\delta = \lvert \hat{p}(A) - p(A)\rvert$ YesCalibration error
$A = {x \in \mathcal{X} \mid \hat{p}(x) > 1/lvert \\mathcal{E}$$$ for response collection above threshold

Now we need to move the previous conclusions to the "with hint" scenario.

In reality, the model is based on the hint$c \in \mathcal{C}$ Generate Response $r$I'm sorry. Set the hint distribution to $\mu(c)$Training is distributed to the probability of conditions $p(r \mid c)$。

For each hint $c$, defines:

$$ \mathcal{V}_c = {r \mid (c,r) \in \mathcal{V}}, \quad \mathcal{E}_c = {r \mid (c,r) \in \mathcal{E}} $$

Key parameters：

$K = \min_c \lvert \mathcal{E}_c \rvert$: Number of error responses for simple tips
$k = \max_c \lvert \mathcal{V}_c \rvert$: Number of correct responses to the most difficult tips

The conditions are deformed and the conclusions given in the inferences are easily transposed into the form of a probability of conditions (see the appendix to the original paper for proof).

Test Distribution：

$$ \mathcal{D}(c,r) = \begin{cases} \frac{1}{2}\mu(c)p(r \mid c) & \text{r\mathcal{V} c \c \mu\frac \lvert\mathcal{E} c\rvert} & \text{r\mathcal{E} c The next thing I know, I'm not sure. I'm sorry.

Catalogue：

$$ \hat{f}(c,r) = + \iff \hat{p}(r \mid c) > \frac{1}{\min_c \lvert \mathcal{E}_c \rvert} $$

Directly give theorem 1 extension form:

Theorem 2 (with hints): For any $p$（$p(\mathcal{V})=1$and $\hat{p}$, by:

$$ \text{err} \geq 2 \cdot \text{err}_{\text{iiv}} - \frac{\max_c \lvert \mathcal{V}_c \rvert}{\min_c \lvert \mathcal{E}_c \rvert} - \delta $$

of which $\delta = \lvert \hat{p}(A) - p(A)\rvert$，$A = {(c,r) \mid \hat{p}(r \mid c) > 1/\min_c \lvert \mathcal{E}_c \rvert}$。

For many of the real world,The theory of computing learning has given us a corresponding answer. $\text{err}_{\text{iiv}}$ BottomI'm sorry. So in these cases, it's subject to two of the above-mentioned variations,The illusion of a language model cannot be completely eliminated: they exist in a defined sub-class of people, depending on the error rate for the question of the second classification.

Post-training hallucinations.

Stimulation distortions in the assessment mechanism

Current language model assessment is widely usedDouble score.(0-1 loss):

Correct answer: 1 point or full score
Wrong answer or "I don't know."'t know(IDK): 0 minutes

Observation 1 (optimal strategy for binary scoring): Distribution of beliefs $\rho_c$ In the right answer, IDRK responded with a strict rating of expectations below any speculative response with a non-zero probability.

Formalization of certificates: Set rating functions $g_c: \mathcal{R}_c \to {0,1}$ Satisfied $g_c(r) = 0$ For All $r \in \mathcal{A}_c$(IDK ASS) Exists at least one $r* \notin \mathcal{A}c$ 使得 $\Pr{g_c \sim \rho_c}[g_c(r^*)=1] > $0. Therefore:

$$ \mathbb{E}{g_c \sim \rho_c}[g_c(r^*)] > 0 = \mathbb{E}{g_c \sim \rho_c}[g_c(\text{IDK})] $$

Teaching-likeThis is like most of the calibration tests -- zero points left in the air, and no points taken off by miscalculation, so random speculation is the strategy to maximize the benefits, even if the answers are not known.

Current status of mainstream assessments

Table 1: Rating of the baseline for the mainstream assessment

Benchmark	Modalities of assessment	Double score.	IDK Score
GPQA	Multiple accuracy rate	Yes.	None
MMLU-Pro	Multiple accuracy rate	Yes.	None
IFEval	Command Follow Verification	Yes.	None
Omni-MATH	Mathematical Equivalence	Yes.	None
SWE-bench	Unit test passed.	Yes.	None
WildBench	LM Rating (1-10)	Yes	Partial (but lower than hallucinogenic) "fair" Response)

Conclusions: The paper statistics point out that most mainstream benchmarks severely penalize IDK, leading to models inThe assessment system is more speculative。

The relief of hallucinations requires a change in the mainstream rating, Benchmark.

Most of the existing mainstream assessment benchmarks use a dual rating. Benchmark, which is used for hallucinogenic assessment, exists but is not always included in the core indicators. To mitigate LLM's hallucinations, at least the core benchmark needs to include the expression of uncertainty in the rating design. So long as LLM can also obtain higher ratings on random speculation during testing, the problem of illusions is difficult to solve: because models are still rewarded for speculation, it is impossible to stabilize the IDRK answer.