Why Data Keeps Fooling You: The Counterintuitive Inspection Paradox

This English version was machine-translated from the Chinese original. Technical terms may require verification.

The questions in this article can also be addressedStatistics and Truth: How to use the accident (Statistics and Truth)、Anscom Quartet: Visualized power and statistical illusionHow the concept of a relatively close read together is developed in different contexts.

Intuitive thinking: Why do you always deviate from the average?

The logic of examining paradoxes is straightforward:When you look at or "check" something, those "greater" or "longer" samples, because they take more space or time and have a higher probability of being drawn by you. So, the probability of intuition underestimates the real probability, and the real probability itself is counterintuitive.

Let's see two classic intuitive scenes:

• The bus's delusion: Assuming that a bus on a line averages 10 minutes a shift. In fact, the intervals between departures fluctuated, sometimes five minutes, sometimes 15 minutes because of traffic jams. If you're in one day,RandomThe probability of you falling 15 minutes apart is three times the distance of five minutes. So, the average waiting time you experience as a passenger is necessarily greater than the real average dispatch time, as measured by the public transport company.
• Overcrowded university classes: The school has 10 classes for 10 people and 1 for 200 people, with a real average class size of about 27. But when randomly sampled, the 200 students who were stuck in the major classes would be significantly higher than the average.

In this paradox, no one lies, just becauseThe probability of a sample taken by the observer is proportional to the size of the sample itself.。

A close proof of probabilistic perspective.

In a strict mathematical framework, the test paradox can be expressed as a Length deviation sample (Length-Biased Sampling) Problem.

Assume that the variable in the real system is $X$(e.g. real bus departure interval), the real probability density function is $f(x)$, the (average) is $\mu$, the difference is $\sigma^2$。

$$E[X] = \mu$$

$$Var(X) = \sigma^2 = E[X^2] - \mu^2$$

When we were random observers, we saw a length of time when we were going into the system. $x$ The probability of the sample, and the length of it itself. $x$ Positive. So we...Actual observationRandom variable $Y$ Probability density function $g(x)$ The following is a long-weighted version:

$$g(x) = \frac{x \cdot f(x)}{\int_0^\infty x \cdot f(x) dx} = \frac{x \cdot f(x)}{\mu}$$

Based on this new probability density function, we calculate the averages that observers experience. $E[Y]$：

$$E[Y] = \int_0^\infty x \cdot g(x) dx = \int_0^\infty x \cdot \frac{x \cdot f(x)}{\mu} dx = \frac{1}{\mu} \int_0^\infty x^2 \cdot f(x) dx$$

Because... $\int_0^\infty x^2 \cdot f(x) dx$ Yeah. $E[X^2]$, the alternative variance formula $E[X^2] = \mu^2 + \sigma^2$, can be inferred from the examination paradox:

$$E[Y] = \frac{\mu^2 + \sigma^2}{\mu} = \mu + \frac{\sigma^2}{\mu}$$

The conclusion is clearly contrary to instinct:

Because of the difference. $\sigma^2 \ge 0$♪ So so ♪ $E[Y] \ge \mu$ Always set up.As long as the system fluctuates, the averages you observe are necessarily greater than the real averages. The bigger the variance, the more the paradox you feel.

Statistical analytical attention

We understand the mathematical logic of the bottom, and we have to guard against this sampling deviation when we do actual statistical analysis or construct algorithm models. Especially when large-scale data cleansing at the bottom is carried out, if this bias is not eliminated, it will be the cause of disaster for all subsequent analytical and model training.

• Quite clearly analyze the entity (Entity): Before calculating the indicators, define your main subject. If the efficiency of the system is to be assessed, the entity is the system itself (see para. $\mu$); the entity is the user if the user experience is to be assessed (request) $E[Y]$I'm not sure. Never leave a vague average on the data board.
• Watch out for structural deviations in the construction of the AI training set: Assuming you're trying to build an AIAgent training set (e.g., a model for security gap analysis) by capturing codes from open-source libraries like GitHub. If you randomly swipe codes or functions, you have a high probability of swiping out large, swollen code warehouses or giant files. This will result in a serious bias in your model training data towards specific large project styles, overlooking small, delicate modular codes.
• Introduce reverse probability weighting (IPW): If the data set in your hands has been examined for paradox contamination (e.g., observational data that can only be obtained from a user perspective), a weight inversely proportional to the size (or frequency) of each observation can be given to statistical extrapolation or model fine-tuning (e.g., data from the user's perspective).$w_i = 1/x_i$) to restore the real bottom distribution in mathematics.
• Use more medians, with a cautious average: Since the average is highly vulnerable to distortions in the range and long tail data, medium (Median) or percentage (e.g. P50, P90) are often more robust than averages in the face of business scenarios with asymmetric or high differentials.