The Computational Revolution in Statistical Inference: Jackknife, Bootstrap, and Subsampling

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Introduction: Inference crises in application

The questions in this article can also be addressedFrom Oscillospace to Flowing: The Declining of High DataHow the concept of a relatively close read together is developed in different contexts.

In today's data science applications, the common scenario is that you have designed an operational indicator to measure algorithmic effects or have trained an in-depth neuronet model. In addition to obtaining a point estimate (e.g. CTR up 2%, or model accuracy 85%), operators or scientific reviewers often ask:

"What is the confidence of this result? How much is it in the range of fluctuations?"

That sounds like a standard statistical inference. In the traditional statistics course, we learned to process the mean of the function with CLT (centre of extreme limits), and the difference of the function with Delta Method (Delta method). But in the face of modern applications, these classic tools are often not effective:

• The indicators are too complex.: In the A/B test, many of the indicators we are concerned with (e.g., the decay factor for retention rates, the fractional value of the LTV for the user life cycle) are difficult to process, and it is impossible to even write a decomposition expression, let alone ask for a guide.
• The hypothesis is too fragile.: Many extrapolations rely on the normality or independence of data (IID). However, in financial time series or social networking data, these assumptions are often broken.
• Blackbox Model: For deep learning models, we do not even know what the specific "parameters" distribution is, much less the broad error boundary that can be extrapolated through the Hessian matrix.

In the face of these “unpredictable” difficulties, statistics have undergone an important shift in the second half of the 20th century:Computer statisticsRise. It's a direct idea, but it changes the way it is extrapolated:

If distribution is not easily extrapolated by mathematical formulas, can we use the powerful computing power to "calculate" distribution directly by means of repeated simulations of data?

The three leading actors in this revolution are discussed here:Jackknife、Bootstrap and SubsamplingI'm sorry. We start with the pain of their application, and see how computing can replace the decomposition and become a new engine of modern statistical extrapolation.

2. Jackknife: sharp instrument for correction of deviations

Application scenario: deviation from ratio estimates

In sample surveys and econometrics, we often need to estimate the ratio of the two variables. For example, to estimate the “input output ratio” of society as a whole, we may divide the average output of the sample by the average input:$\hat{R} = \bar{y} / \bar{x}$。

However, statistics tell us,Rate expectations do not match expectations(i.e., $E[\bar{y}/\bar{x}] \neq E[y]/E[x]$I'm not sure. This means that the government is not a party to the law.$\hat{R}$ It's one.Biased EstimatorI'm sorry. In the case of small samples, this deviation may seriously mislead decision-making.

How do you correct this deviation? If you're going to deduce $\hat{R}$ Taylor has begun to make corrections, which are cumbersome and prone to errors. This time,Jackknife (scissor) It's going to work.

Algorithms and principles

Jackknife's idea is,Insulate the nature of the whole by observing the impact of the absence of some data on the whole。

Suppose we're gonna take a sample. $\mathcal{X}_n = {X_1, \dots, X_n}$ Medium estimate parameters $\theta$。

1. Build a Go-I sample: Yes $i = 1, \dots, n$♪ We'll be $i$ A observation value removed from the sample and the size of the observation was $n-1$ The sample. $\mathcal{X}_{(i)}$。
2. Calculates the volume of copies: recalculated statistics on each “Done One” sample, obtained $\hat{\theta}_{(i)}$。

Tukey (1958) proves that through these $\hat{\theta}_{(i)}$, we can construct a revised estimate of deviation:

$$ \hat{\theta}{jack} = n\hat{\theta} - (n-1)\bar{\theta}{(\cdot)} $$

Where's the bar?{(\cdot)} = \frac{1}{n}\sum \hat{\theta}{(i)}$。

It seems simple, but strong: It can take the deviation from the estimate. $O(n^{-1})$ Directly from elimination to $O(n^{-2})$I'm sorry. In applications with small sample volumes (e.g. early medical clinical trials), Jackknife can provide much more accurate estimates.

At the same time, Jackknife gave a non-parameter estimate of the variance:

$$ \widehat{Var}{jack}(\hat{\theta}) = \frac{n-1}{n} \sum{i=1}^n (\hat{\theta}{(i)} - \bar{\theta}{(\cdot)})^2 $$

Limitations and application of borders

The calculation cost for Jackknife is low (just need $n+1$ (a) A positive performance in correction. But in modern applications, it has obvious limitations:

It will fail on non-smooth indicators.

The most typical example isMedium (Median)I'm sorry. Even the sample. $n$ The median difference estimated by Jackknife is not consistent (it will not shrink to the real difference). This limits its application in modern wind control indicators based on fractional numbers (e.g. VaR, Value at Risk). We need a more universal approach.

Bootstream: Swiss military knife, general extrapolation

Application scenario: confidence-building for complex operational indicators

In the Internet company A/B testing platform, analysts often define complex “artic star indicators”. For example:

$$ \text{Metric} = \frac{\text{GMV}}{\text{DAU}} \times \log(\text{Retention}_{7\text{day}}) $$

Is this real or random increase when the test group and control group have a 1% difference? To answer that question, we need to draw a composite indicator. 95% confidence compartmentI'm sorry. It is almost impossible to extrapolate the distribution of this indicator.

1979, by Bradley Efron. Bootstrap (self-help) A common solution to the problem was provided.

Core idea: Plug-in Prince (inclusion principle)

The philosophy of Bootsrap is very intuitive:Since we can't get the real distribution, $F$Then you can tell the distribution of the experience you've seen. $\hat{F}_n$ Consider it a true sum.

• Real world.: From the general $F$ Data from the Chinese sample $\mathcal{X}_n$, calculate the amount of statistics $\hat{\theta}$。
• Bootstrap World: Distribution from experience $\hat{F}_n$ Medium- Yes. Sampled $matcal{x}_n$，计算统计量 $\hat{\theta}^$。

Efron proves that under quite a wide range of conditions, the world of Bootslap Medium $\hat{\theta}^*$ Around $\hat{\theta}$ The distribution of the world is perfect for the real world. Medium $\hat{\theta}$ Around True Values $\theta$ the distribution.

Algorithms & Credibles

With this principle, it becomes very simple to apply:

1. Re-sampling: Using computers, put back to extract from raw data $B$ Group data (e.g., $B=1000$）。
2. RedecountCompute your complex indicators on each set of data, and get 1000 $\hat{\theta}^*$。
3. Distribution extrapolationThis is the distribution of 1,000 values, the simulation sample distribution of your complex indicator.

For confidence compartments, most commonly.BitmapTake the first 2.5% of the 1000 digits and the 97.5% fraction point as the upper and lower boundary between the zones.

We can also use scenarios that require a high degree of precision (e.g. biopharmaceuticals). BCa (Bias-Corrected and Accelerated) MethodI'm sorry. Using the bias and deviation information estimated by Jackknife, it fine-tuned the fractions to obtain a second-order accuracy.

Bootstrap has significantly liberated the productivity of data scientists and has become a standard tool for modern statistical extrapolation of this "Swiss Army Sword".

Here's another side: in the machine learning model assessment, Bootstream was used as a data-segregation strategy -- $D with a drop sample.'$ Trained, tested with approximately 36.8% of undiscovered samples (out-of-bag), suitable for small data sets and integrated learning. This usage complements the extrapolation perspective of this section, and a more systematic model assessment approach is presented in the following paragraphs:Monitoring of learning performance assessmentThe self-help approach section of the Law.

Subsampling: Last line of defence in extreme cases

Apply scene: when Bootstream is invalid Time

Bootstream is useful, but it's not almighty. In some high-risk applications, blind use of Bootstream can have serious consequences:It may give a confidence interval that seems precise and completely incorrect.

Typical failures include:

1. Extreme estimate: For example, the distribution boundary for estimating peak traffic in cybersecurity. The maximum value of the Bootslap sample will never exceed that of the original sample, leading to a degradation of its distribution at the end.
2. Parameters at Borders: When the true parameter is located at the boundary of the parameter space (e.g. a 0-square test).
3. Strong reliance on data: Simple Bootslap destroys the time-series structure of the time series.

At this point, we need to ask the most theoretical and robust members of the heavy sample family:Subsampling (subsampling)。

Core thinking and operational guidelines

Subsampling's logic is clearly different from Bootstream:Bootstream tries to simulate the real world.$n \to n$The real world is only a microcosm (the real world).$m \to \infty$）。

Its core operation is based onNo Return Sample (Sampling Without Replacement)and sample quantities $m$ Much less than $n$。

Specific Operational Steps (Algoritthm)

Subsampling is used in the actual project and can be operated as follows:

1. Determine subsampling size $m$: This is the most critical step.$m$ It has to follow. $n$ Increase and increase, but faster than $n$ Slow.Common experience:$m = \sqrt{n}$ or $m = n^{2/3}$。

2. Build subsamples: From the original $n$ The blog is a good example of the situation.No returnExtract $m$ Data. That means you're generating a much smaller data set than the original one.

3. Calculate statistics: Count your numbers on each subsampling. $\hat{\theta}^*_{m}$。

4. Repeat and Distribution Build: Repeat the above process $B$ Number of times (e.g. $B=1000$(c) The distribution of experience in obtaining statistics.

5. Rescaling: This step is often forgotten. Because you're in $m$ The distribution calculated on a sample is definitely more varied than the one calculated on the sample. $n$ A sample on the big. To extrapolate the original sample, $n$ The nature of the process, you need to use the speed of extraction. $\tau_n = \sqrt{n}$) To scale up. What we need to see is... $\tau_m (\hat{\theta}^*_m - \hat{\theta}_n)$ The distribution of the problem. Use it to simulate it. $\tau_n (\hat{\theta}_n - \theta)$ the distribution.

Why is it more "strong" in theory?

The effectiveness of Bootstream depends on a stronger assumption:The distribution of statistics must be smooth.(i.e., $\hat{F}_n$ Weak harvests $F$ At times, the distribution of statistics must also be reduced). This is often not valid when non-slipper parameters (e.g. polar, median, etc.) or when parameters are located at the boundary.

By contrast, Politis & Romano (1994) proves that Subsampling's effectiveness requires only that the statistical volume itself be distributed at limits.

This is a very weak condition:

• If Bootstream works, Subsampling works (although efficiency is somewhat lower because it is not a whole sample).
• Subsampling is still valid if Bootstream is invalid (e.g., the polar issue).

It is thus often called the “last line of defence”.

Modern front application cases

1. General error estimates in depth learning

In in-depth study theory, we often need to estimate the boundaries of the model across error. The standard progressive normality assumption is completely invalid because the loss function curvature of the nervous network is non-compressed, non-silent and very high in parameters. The blogger says:Subsampling An effective means of constructing such non-ruled statistical confidence-building areas is provided. It does not need to assume second-order guidance for loss functions and is able to capture more robustly model fluctuations resulting from training data disturbances.

2. Enhanced learning and time series: Block Subsampling

In the Policy Assessment, the trajectory data generated by intelligent bodies are highly relevant in time. The random and shattering of heavy samples by Bootstream directly undermines this time dependence and leads to a serious underestimation of the Value Function equation. You'll think the strategy is stable, not really.

The solution is to use Block SubsamplingI'm sorry. We are not taking single sample points, but a continuous “block of time”. This preserves the local dependency structure within the data, which allows for the correct estimation of the differential returns of Long-horizon and provides a real security boundary for the tactical trajectories.

3. Extreme value theory (EVT) and financial regulation

In calculating the value of the financial market at risk (VaR) or expected loss (ES), we are concerned with the distribution of the “tails” — the black swan event that occurs at one-tenth of the 10,000. Subsampling has become the preferred statistical extrapolation of extreme values due to the inconsistency of Bootstrap ' s estimates of boundaries. Select the appropriate subsampling size $m$We can use it. $m$ The polar distribution pattern in the sample is extrapolated by the scaling relationship of the polar theory. $n$ Extreme risks in samples even on larger scales in the future.

5. Summary

From Jackknife to Bootstream, and then Subsampling, this evolution clearly shows how statistics, with their calculus, conquer the difficulty of applying:

• ♪ When we face ♪Small sample deviationThe problem is that the government is not a party to the law.Jackknife The simplest way to go is to give a graceful solution.
• ♪ When we face ♪Common extrapolation of complex statisticsThe problem is that the government is not a party to the law.Bootstrap Using “simulations instead of extrapolation”, it has become a standard weapon for data scientists.
• ♪ When we face ♪Extreme, non-sliding, heavy dependenceThe blogger says that the government is not the only one who is not a member of the opposition.Subsampling Relying on weaker theoretical assumptions, it is often more stable than Bootstream.

In today ' s time, where the algorithms are within reach, it is far more valuable for data science practitioners to understand the rationale and boundaries of these methods than to remember a few normal distribution formulas. The real world is often not a normal distribution, but it can always be counted.