Neural Scaling Laws: From Kaplan to Chinchilla

This English version was machine-translated from the Chinese original. Technical terms may require verification.

I'm not studying Deep Learning Theory, and I'm not familiar with the content; the following is just a brief summary of a theoretical field. Write anything about fun.

The questions in this article can also be addressedCompression for AGI: compression is intelligenceHow the concept of a relatively close read together is developed in different contexts.

If you want to see Scaling Law first, you can read it first, between data, pre-training, post-training and deployment.LLM Life Cycle Overview;This paper expands only the size pattern itself.

From 0 to 1 from Neal Scaling Laws

The Neurocaling Laws are a common set of empirical patterns that understand large-scale in-depth learning (especially the LLM model). It describesThe relationship between model performance and Power Law calculations of resources, data and parameters.

Within a given range, multiple increases in the algorithm, data or parameters, and model Loss (loss function) usually drops at a relatively stable rate. This has moved the refining from a purely intuitive approach to the process, to an additional tool for estimating work that is not necessarily accurate but sufficiently useful (P.s. The material of the new age is refined and extremely friendly to the experimenters, except for their hair).

The most basic formula for Neal Scaling Laws is usually: $$ L(x) \propto x^{-\alpha} $$ of which $L$ It's on the test set, Los.$x$ is a size variable (e.g., parameter volume) $N$Data volume $D$ Or, uh, the math. $C$），$\alpha$ is the scaling index. Main findings: Within the scope of the experiment, model performance is mainly influenced by size (Scale) and has a weaker relationship to specific model structures (e.g., layers, width ratios) (as long as the architecture is not too disproportionate).

Baseline: Kaplan Law for OpenAI - Multiplication and Parameter Prefer

Kaplan Team Hypothetical Test Set Los ($L$) with the amount of parameters ( )$N$) and the size of the data set ($D$Following the laws of independence and the two are bound together. Original article is Scaling Laws for Neural Language Models (Kaplan et al., OpenAI)

It's often compared to the age of big models."Moore's Law."I'm sorry. More precisely, it uses experiments to combine performance and performance of Transformer models. $N$(Agrilateral quantities),$D$(Size data),$C$There is a stable arctic relationship (calculated amount).

Its single variable forms: $$ L(N) \approx \left( \frac{N_c}{N} \right)^{\alpha_N}, \quad L(D) \approx \left( \frac{D_c}{D} \right)^{\alpha_D} $$ of which $\alpha_N \approx 0.076$, $\alpha_D \approx 0.095$, using experiments to prepare.

To enable Scaling Laws' experience formula to further guide the next stage of architecture design, Kaplan proposed a coupling.Joint Zoom FormulaTo describe $N$ and $D$ Time limits: $$ L(N, D) = \left[ \left( \frac{N_c}{N} \right)^{\frac{\alpha_N}{\alpha_D}} + \frac{D_c}{D} \right]^{\alpha_D} $$ Math meaning: This is a similar, even-handed form. Because $\alpha N < \alpha_D$，这意味着随着算力增加，参数 $N$ 的边际收益递减速度比数据 $D.L.L., slow down.

Inference: To minimize the loss of Los,Arguments $N$ It should grow faster than the amount of data. $D$(i.e., $N \propto C^{0.73}, D \propto C^{0.27}$I'm not sure. They therefore suggested that, in increasing the ability to calculate, priority should be given to the model being larger than to the unlimited increase in data.

This directly led to the early large model (e.g. GPT-3, PLM, MT-NLG) crazy stack parameters (175B, 540B), but training data were relatively small (usually only 1 Epoch trained).

Amendment time: Chinchilla Law of DeepMind - Added in equal scaling Fire!

The Chinchira team is here Training Compute-Optimal Large Language Models (Hoffmann et al., DeepMind) points to the loophole in the Kaplan method when it's ready to fit the hyperparameter and suggests a more intuitive one.Adding Model, contains non-acceptable errors.

Core formula: $$ L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta} $$

• $E$: Irreducible Los, i.e. the entropy of the natural language itself (Beyers error), even if the model is perfect.
• $\frac{A}{N^\alpha}$: Model Approximate Error, which results from inadequate model capacity.
• $\frac{B}{D^\beta}$: Data estimate error, from the differences caused by limited samples.

It's the last thing I can do. $C \approx 6ND$ Under a fixed constraint, use the Lagrams multiplier to reach the extreme, found in DeepMind $\alpha \approx 0.5, \beta \approx 0.5$I'm sorry. Because $\alpha \approx \beta$，$N$ and $D$ It's a similar contribution to Los.SymmetryI'm sorry. The best strategy under the fixed sum is to... $N$ and $D$ It should be on the scale.（$N \propto C^{0.5}, D \propto C^{0.5}$I'm not sure. That's famous."Chinchilla Scaling Laws"。

Chinchilla Scaling Laws overturned parts of Kaplan's conclusions. DeepMind found that previous models (e.g. GPT-3) were allUndertrainingI'm sorry. For the current Transformer,Each parameter requires approximately 20 trainings(i.e. number of Tokens/ argument numbers) $\approx 20$I'm not sure. This has dramatically changed the direction of the industry away from the blind pursuit of trillion parameters, but rather of relatively small models + very large amounts of data.

The power of the Scaling Laws"Inconsistence" (The Discontinuity)

In the laws of Kaplan and Chinchilla, the decline of Loss is silky. But in practical applications, engineers have discovered an incomprehensible phenomenon:Losing the lower does not mean that models learn to do things.

Models are going through one of these tasks (e.g. triple digit plus, complex reasoning). Phase Transmission: From"Totally random."♪ And suddenly jumps to ♪"Close to humans."。 Emergent Abilities of Large Language Models (Wei et al., Google, 2022) partly answered the question -- why is Loss smooth, but the result is steep?

We can use a simple probability model to extrapolate this non-linear.

Suppose a logical task needs to be continuous. $L$ Steps is all right to score. You're the one who's gonna get you. $p$ Forecasting for modelsSingle Token (or one-step reasoning) The correct rate. According to Scaling Laws, with size $N$ Increase, one-step correct rate $p$ It's smooth. $p \propto 1 - N^{-\alpha}$）。

But the probability of success for the whole mission $P(\text{Task})$ is the product of all steps: $$ P(\text{Task}) = p^L $$

Here, a non-linear dimension of a trimester is introduced. Assumptions of mission requirements $L=5$ Step: Small model ($p=0.5$): $P(\text{Task}) = 0.5^5 \approx 0.03$ (near 0, as shown)"No, I won't.")

Medium Model ($p=0.8$): $P(\text{Task}) = 0.8^5 \approx 0.32$ ( still failing)

Critical points ($p \to 0.95$): $P(\text{Task}) \approx 0.77$ (The power is bursting.)

Conclusions It's not magic, it's magic.Projection of microprobability multiplier effects on macro-indicatorsI'm sorry. That explains why we need to do the model so great -- because only one step is accurate. $p$ The success rate of long chain reasoning when it's extremely close to 1. $p^L$ It's only meaningful.

From the perspective of Metric, Stanford's Schaeffer (2023) points out that if we put the evaluation indicators from the perspective of the"All right."In the smooth."Token Edit Distance", the curves rise back to smooth curves. It tells us:Capabilities have been built up, only because of the harsh assessment criteria, which have led us to lose sight of them at the tipping point.

Data repetition and quality: Data-Constrained Scaling

Scaling Laws is demanding too much data, and the Internet is running out of high-quality text (Data Wall). We have a new problem:If the data are not sufficient, how many times can we learn from old data?

Scaling Data-Constrained Language Models (Muennighoff et al.) Amends the Chinchilla formula to include repeated training (Epochs, as $R$as a variable. He found out that the data...ValidityWith the number of repetitions declining.

We can create a simplified conceptual formula: $$ D_{eff} \approx D_{unique} \cdot (1 + \lambda \cdot \log R) $$ Or more intuitively, the conclusion is: $R \le 4$ (4 Epochs in): Data returns are almost non-depleted. Models extract residual value from duplicate data.

$R > 40$: The revenue is almost zero, and even over-compatibility leads to a backsliding of Los.

In an era of data depletion, we can only repeat four times at most the high-quality data in our hands. After that, new ways of finding solutions (e.g. synthetic data) must be found. Since... $D$The only way out of the constraints (quantity) is to increase the data quality coefficient. Microsoft Phi series proves: $$ L \propto \frac{1}{(Q \cdot D)^\beta} $$ If data quality $Q$ High enough (e.g., synthetic data at textbook level), very small $D$ And it's going to be very low, Los. It breaks blind Scaling superstition.

Mixing expert model: MoE Scaling

Why did GPT-4, Gemini 1.5, Mixtral and DeepSeek all turn to MoE structures? Because Scaling Law showed up on MoE.It's called a bit of a bad efficiency.。

Traditional Dense model, parameters $N$ It's a direct calculation. $C$(FLOPS) The bigger the model, the slower it goes. MoE broke this tie, and it introduced two dimensions. $N$：

• $N_{total}$: Total parameter volume (decided on model)"Knowledge capacity"And memory.
• $N_{active}$: Active parameter (calculating cost/velocity at the time of reasoning).

Unified Scaling Laws for Routed Language Models (DeepMind, Google) found that MoE's Loss dropped following a law that included two items: $$ L(N_{total}, N_{active}) \approx \frac{A}{(N_{active})^\alpha} + \frac{B}{(N_{total})^\beta} $$

There's one key here.Asymmetricality： Costs of reasoningMain $N_{active}$ Decision.

Model performanceBut I can enjoy it at the same time. $N_{total}$ The dividends (although marginal gains are lower than Dense, they are significant on a large scale).

This gives MoE a chance to be Paretto Frontier (Pareto front) Better than Dense model:MoE often mobilizes a larger total parameter capacity with equal reasoning.

Concluding remarks

The laws of neuroscaling have written some of the patterns of experience in large model training into an estimateable engineering relationship. Kaplan emphasizes the size of the parameters, Chinchilla revises the focus to the calculus-data ratio;"♪ Emergence ♪"The data quality and MoE architecture were all problems that were revealed as the scaling line continued to advance. As Training Scaling’s marginal gains become more expensive, the follow-up route will continue to seek better quality and synthetic data, while also focusing more attention on the Inference Time Scaling.

Bias-Variance No Trade-off：Neural Scaling Laws and Statistical Learning Theory

The Neurotic Scaling Laws are important not only because they guide engineering estimates, but also because of the obvious tension in the intuition of the Statistic Learning Theory, SLT.

The traditional SLT tells us"The model is too well developed."And making a better model requires controlling the complexity of the model, and Scaling Laws tells us that"The bigger the better."I'm sorry. This tension comes from...Defeat and re-engineering in the age of deep learning。

Classical Review: Traditional Perspectives of Dividence and Differences

Before we go into the conflict in depth, we need to look at how the classical statistical learning theory views generalization errors.

Offset-Equal Disaggregation (Bias-Variance Decomposition) It is a common tool for interpreting the generalization of learning algorithms. For a learning algorithm, its expectations on the test set are generalized error. $E(f;D)$ It can be broken down into three sumes:

$$ E(f;D) = \text{Bias}^2(\boldsymbol{x}) + \text{Variance}(\boldsymbol{x}) + \text{Noise} $$

• Distortion (Bias): The capability to develop models is measured. High deviations mean that models are too simple to capture the true pattern of data (see figure 1).Outstanding）。
• Difference (Variance): Measurement of the sensitivity of the model to changes in training data. The difference means the model is too complex to remember the noise in the training data.Compromise）。
• Noise: The data itself is non-accuracy, and is a general sub-continuation.

Offset - Square Discretion (Bias-Variance Dilemma): This is the most painful trade-off in traditional theory.

• When training is not enough: Bias leads, Varance low. The learning device is not sufficiently formulated and the disturbance of training data is not sufficient to make a significant difference in the learning device.
• When you're overtrained.: Bias low, Varance dominated. The learning device was so well designed for training data that it captured the noise in the data.

This leads to famous. The U-type learning curve: The total error drops and rises as the complexity of the model increases. Therefore, training needs to stop at a middle point in search of a less-than-high “Sweet Point” (Trade-off) for Bias and Varance.

Collision with traditional SLT: from uniform condensation to the paradox of scale

In classic SLT, we tried to find one."Worst case scenario"The guarantee. The broad error (Risk) is usually broken down into: $$ \text{Risk} = \text{Bias}^2 + \text{Variance} + \text{Noise} $$

Classic. VC Panorama (VC Generalization Bund) Tell us about the complexity of the model. $h$(approximate to parameter volume) $N$) Hypothetical space, generalization error $R(f)$ Training error $\hat{R}(f)$ There is a relationship between: $$ R(f) \leq \hat{R}(f) + \underbrace{\sqrt{\frac{h (\log(2n/h) + 1) + \log(1/\delta)}{n}}}_{\text{Complexity Penalty}} $$

The prophecy of the classic theory: With the Arguments $N$ Increase, training error $\hat{R}$ It'll drop to zero, but in $N > n$ 时，复杂度惩罚项（包含 $\sqrt{h/n} will tend to be endless. It's a direct result of the famous. U-type curve Inference: The total error must have been first down and then up.

But in 2017, Zhang et al. Understanding deep learning requires rethinking generalization A very powerful phenomenon was suggested. They found:Even if you put the label on the training set. $y$ All randomly disrupted. Deep network still reaches. $\hat{R}(f)=0$I'm sorry. This means that the model is sufficiently big to remember pure noise, but it can still be generalized in real data; tradition is based on"- We're all in."It is difficult to explain what is actually happening in depth learning.

Escape the curse: a virtuous match

To explain Scaling Law $N \to \infty$ And then, Ross, he fell in one-way, and he was told, It's a good thing. Theory (Bartlett et al., 2020). This theory is trying to re-understand. Varance The behavior in the high space.

Amount of time $d$ Much greater than the sample $n$ And then the SSD will be hidden. $\ell_2$ Minim Norm ♪ The world ♪ $$ \hat{\theta} = \arg\min_{\theta} |\theta|_2 \quad \text{s.t.} \quad X\theta = y $$

With this setup, the breakdown of Risk has changed:

• Bias single-hut drop: As the model becomes larger and the subspace cover becomes stronger, the model is better able to approach the real function.

• Varance disappears (not explodes): That is the most important part. Bartlett proves that as long as the data match the matrix to meet a specific spectrum decay,Noise energy will be used."Paint"It spreads across countless extra dimensions. Go, go, go!。

Full mathematical image: double drop (Double Descent)

Combining the theory, we got Scaling Laws complete. Double down. Image:

• Arrear parameters range ($N) < n$)： Observed by the classic VC-dimensional boundary, presenting the U-type curve. Bias falls, Varance rises.

• Critical Range (Classory Range)$N \approx n$)： $XX^T$ The smallest feature value is close to 0, and the counter-format is extremely large, resulting in Risk explosionI'm sorry. This is the region most feared by traditional statistics.

• Good over integration (in %2)$N \gg n$)： Enter the Scaling Law field. Bias continues to decline because of dimensions $d$ It's huge, the anti-formation is becoming good and the noise is spread to high-dimensional zero.Varance, instead of blowing up, is approaching zero.。

Scaling Laws proves that, under the combination of deep nervous networks + SSD, we are on the right side of the double-down curve. Increase $N$ And so, the fact that the two of us are able to reduce the Bias and Variance at the same time, and thus the Loss is showing a one-way drop in the law.

Theory Reconstruction: Scaling Laws Revision of Math Intuitives

Scaling Laws proves that under the combination of deep nervous networks + SSD, the government is not going to be able to use the Internet to control the situation.The differential gains (soft and integrated) from over-parametricization exceed the differential risks associated with it。

Let's compare the mathematical perspectives of the old and new ages:

Feature	Traditional Statistical Learning (SLT)	Scaling Laws	The mathematical nature difference.
Curve form	U-shape curve (U-shape)	The law is falling.	Mono-telephone vs non-uni-telephone
Main contradictions	Vs variance	Calculate resource allocation (allorcative Efficacy)	Optimization issues vs statistical extrapolation issues
Overparameter	Dangerous (over-sizing)	Must.	Benign Overfitting
Error limit	Dependency $\sqrt{N/D}$	Dependency $N^{-\alpha} + D^{-\beta}$	Here. $N$ It's a denominator.

Formula for the Chinchila Law $L = E + A/N^\alpha + B/D^\beta$ It actually rewritten the general error: it no longer contained the explosion. $N/D$ item, but will $N$ and $D$ Considers two independent variables that have a positive contribution to reducing Ross. This is a major revision of the traditional statistical learning theory in the context of deep learning.

United: Chartering Laws ' structure irrelevant and computational scaling down laws

These two elements bring Scaling Laws from the empirical formula to the allocation of engineering resources and historical evolution.

If you say so."Smooth"Scaling Law exists.Math BaseWell, then..."Calculate Zoom"It's how you build a building.Construction drawingsand"Structure irrelevant"And that explains why we finally chose Transformer.Construction materials。

Compute scaling: Optimizing resource allocation issues

In practical, large model training, we're not just concerned with..."Parameters $N$"or"Data $D$"In itself, it's... "I have $100 million in the budget. What am I supposed to do?" And then Scaling Law became a resource optimization issue.

Train a total calculation volume for a Transformer model $C$(Floating-point operations) can be approximated as: $$ C \approx 6 \cdot N \cdot D $$

• $N$: Model parameter volume.
• $D$: Token number of training data.
• $6$: Empirical factor (upward transmission of the approximation) $2N$Inverse transmission of the contract $4N$）。

Optimizing geometric interpretation of the problem: Imagine a two-dimensional system:

• X-axis: calculated quantity $C$( logarithmic coordinates).
• Y axis: Los (relative coordinates).

If we fix the size of the model, $N$(e.g. 10B parameter) increasing data $D$And we'll get a curve. ♪ With $D$ Add, and Los drops and slows down (limited to model capacity). If we draw curves of 1B, 10B, 100B models of different parameters, they're like a line down.

The Scaling Law curve (Compute Frontier) is the lower condensed line of this curve.

It means that it's in any given budget. $C$ I'm not sure if you're going to be able to do this.It's inevitable.The best (N^), D^) $, which brings Los to the smallest global level.

There is a corresponding cost of departing from this optimal combination.Over-large model (over-sized) It means that if you use a huge model but with little data (e.g., Kaplan early proposal), you waste your calculus to multiply the matrix without giving the model sufficient information (Undertrained). The blogger adds:Sub-models It means that if you run through big data with a model of tiny (e.g., a few small models before LLAMA), the model is full and the return on continuing to read is low.

In the early stages of deep learning, bottlenecks are often the same.Algorithm"Don't know how to train deep webs" or "I don't know how to train deep webs"Data(no ImageNet). But in the age of Scaling Law:

• Algorithms are known: Transformer + SGD。
• Data adequacy: CommonCrawl contains the entire Internet.
• The only limit is $C$： Each unit of Los is falling, and needs an index increase in FLOPS.

The Chinchila law can be written as this optimised equation: $$ \min_{N,D} L(N,D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} \quad \text{s.t.} \quad C = 6ND $$ The best part of the job is... $N \propto C^{0.5}, D \propto C^{0.5}$I'm sorry. This means that under this scenario, the budget for computing will directly limit the amount of work to be done by models and the amount of data to be fed.

Structure irrelevant: Why is Transformer the last winner?

Structure irrelevant (Universality)

Scaling Law, a very useful observation:The trend of law enforcement is not just one structure.

Research by OpenAI and DeepMind shows that it is not just Transformer, but:

• LSTM / RNN(Circulatory Neurological Network)
• CNN(CROWD NRUNET)
• Linear Transformers
• Even some pure connections.

Their Loss can be seen with the scale. $L \propto C^{-\alpha}$ The curator form. This means that structures such as LSTM can also benefit theoretically from more calculus and parameters.The scaling up of the gains is not exclusive to Transformer.

If you're following the law, why is LSTM dead?Theoretical performance caps vs. Engineering efficiency。

Although they all follow the formula of the law. $L = a C^{-k}$but coefficients $a$(transmitting) and $k$It's different, it's more important.Feasibility of practical training。

Efficiency is everything: serial bottlenecks and gradients Stream

Serial vs. Parallel (The Wall-Clock Time Wall)

This is the key to the Transformer victory.LSTM/RNN The Mathistic Form $h_t = f(h_{t-1}, x_t)$ It was decided that it had to be calculated in a serial line: to calculate the status of 1000 token, it had to be calculated first, 999 before. This one. $O(T)$ Time dependence is difficult to reconcile, and it leads to a very slow physical training to the same scale even if it is theoretically capable of Scaling.

The blogger adds:Transformer Attention mechanism $Softmax(QK^T)V$ Allows a one-time parallel calculation of all token relationships. Although the complexity is... $O(T^2)$However, it is more likely to eat the array computing capacity of large GPU clusters, and the training efficiency is clearly superior to the current hardware.

Gradient flow and long distance dependence

Apart from calculating efficiency, gradient flows are important. Yes. LSTM , the gradient is to be transmitted back through time steps (BPTT) and is vulnerable to early information loss when dealing with extremely long sequences (e.g., a book), resulting in a Scaling curve being flater than Transformer. And... Transformer , any distance between either token is 1 (connected through the Attention), and a smoother gradient flows make it more efficient to use data in Scaling.

Hardware Lottery

Google Fellow Sara Hooker presented "Hardware lottery" Theory: Scaling Law is not about Transformer being mathematically perfect, but about... Transformer is the most suitable structure for current GPU hardware (matrix multiplication accelerator)。

If we use some kind of brain chip, maybe Scaling Law wins LSTM or Spiking Natural Networks.

But the reality is, we have a GPU that's good at multiplication of dense arrays. Transformer's architecture features (high parallels, dense calculations) make it in this race of Scaling, and it is a very important way to get the world to the bottom of the world.Decline the speed of Loss in unit time (Loss per GPU-hour) Far beyond other structures.

Wrap-up: The cruel truth about Scaling Law

Together, we can draw a cruel conclusion about the modern AI development:

The structure is no longer the core moat: If the architecture supports the gradient decline and has some expression, it can be scale. Transformer's victory is based on the fact thatEngineering efficiencyThe victory (it can run Scaling Law as fast as possible).

The calculation of intelligence: Under the optimal resource allocation strategy (Cinchiilla), the model ' s intellectual level (los) is almost entirely accounted for by input (in the form of a budget).$C$(b) Decision.

Rules of the game: Deep learning becomes one.Transforming electricity and chips into intelligent industrial processesI'm sorry. Whoever runs more efficiently (higher parallels and higher hardware utilization) along the Scaling Law curve, is the winner.

That's why the AI lab is more likeSystems EngineeringNot the traditional algorithm lab. Competitiveness from"Design a fine network structure"Turn"Building infrastructure to operate the Vanka cluster in a stable manner"。

The negative: GDDT"Hard Cut"With invalid Scaling

This is a very real issue. Scaling Laws does not form directly in front of classic gradients such as the GGBDT, LightGBM, which helps us to clarify what conditions Scaling relies on.

The brief answer is:GDDT is still stuck in the U-type of classic statistical learning theory, and deep neural network (DNN) is passed"It's a good thing."Run away.

To understand why Scaling Law is only interested in deep learning, we need to go deep into the bottom of the Function Application.

U-size curve vs. & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & and & & and & & &

If you keep increasing the number of trees in XGBoost (n_estimatorsOr deep, you'll see the textbook.Distortion - Square trade-off (Bias-Variance Trade-off)：

• Initial: As the trees increase, the model captures data characteristics, Bias drops, test set Los decreases.
• Later: Once the threshold has been crossed, the model begins to compress noise. While the training set is close to zero, the test set is quick to bounce back.
• End of story: That's classic. U-type curveI'm sorry. GGBT relies heavily on Earthly Stopping to prevent a crash.

LLM is in contrast"Double down."Modern spaces. When the amount of data is sufficient (in line with the Chinchilla ratio), the increase in the parameter volume usually results in a continuous decline in Los. This one."Stacking arguments"The dividends are more evident in in-depth learning.

Division constant vs continuous flow: a general neutral war

Why? The point is both mathematically. Summarize bias (Industive Bias)It's the model that's pre-set."Worldview"Different.

GDDT: Space Partitioning

The tree model can be understood as cutting up the high space into a lot of irrelevants.Hyper-cubes。 $$ f(x) = \sum c_m \cdot \mathbb{I}(x \in R_m) $$ It assumes that the world is made up of countless unrelated flat slices, and then gives a constant projection (sub-consistent function) in each block, which is expected to jump (StepFunction) at the regional boundary.

• Deadback: Increasing parameters (deepening trees) is tantamount to cutting space even more. As the tree becomes deeper, the space is cut into powder, and each leaf node may contain only one sample.
• Non-slipble: For unseen areas (Gap), the model can only output rigid constants and cannot be inserted according to gradients. This one.Inconsistent, non-smoothThe approach led to the direct explosion of the difference by the parameterization.

DNN: Fluid Soft Approach

The neural network is constructed by a layered matrix multiplication and activation function.A smooth, smooth, continuous flow of height。 $$ f(x) = \phi_L(\dots \phi_1(W_1 x)) $$ It assumes that the world is continuous and micro-synthetic.

• Good idea: Even though the big models are capable,"Remember."Noise, but with the driver of SSD, the model tends to constrict to Minim Norm The solution.
• Inhibited regularization: Math instincts tell us that SPD automatically chooses among the countless solutions that can be drawn.Minimum curve, smoothestThat one. Adding parameters gives the model more freedom to draw more fine and smooth curves than to make sawn teeth.

Conclusion: Smoothness is a ticket to Scaling.

Smoothness will have a direct impact on whether Scaling Law is effective. In high-dimensional space, data is extremely thin. Models must guess the value of the blank area (plug-in value) based on the training point. If the model is..."Jump.", for example, for blank areas (high variance). If the model is..."Smooth", (e.g. DNN) allows smoothing of the plugs using existing data points, thus maintaining low error in the blank area.

We can sum up by a visual analogy:

• Scaling for GDDT It's like stacking blocks. To make a curve, you're going to pile it with smaller and smaller blocks. The smaller the wood, the more visible the sawn teeth on the edge, the more sensitive the deviation from position (oversizing).
• Scaling for DNN It's like pulling a rubber band. To draw up the data points, you add the elasticity of the rubber band (freedom). Under SSD tension, the rubber band not only crosses the data points, but also maintains a smooth transition in the blanks.

Scaling Law is very dependent on smoothness. GDDT lacks global smoothing, overparametrics lead to greater space debris decoupling; and DNN has global flow structure, which makes it easier to convert past parameters into real functionsHigh Precision Smooth Plugin Value。

That's why GDDT is still the master of the tables (low, non-continuous), but on the road to AGI, their fate is very different:

GBDT: Tree models are good at processing low-dimensional, dense, table-type data. However, in very high dimensions (e.g. text, pixel-level characteristics of images), the depth and number of trees required to cover the entire space are exploded exponentially. This increase in size does not bring about a generalization of capacity, but rather a convergence.

DNN: The network of neurons (especially Transformer) is good at compressing high-dimensional thin data into low-dimensional currents through Embedding, and its characteristic combination capabilities are enhanced exponentially with depth and remain general.

Only deep nervous networks can break the dimension curse and set off on the Scaling road to AGI.

Scaling: The nature of non-parametricism versus parametricization

Since tree models can increase the number of trees,n_estimatorsIsn't that a Scaling?

The answer is:Yes, it's a Scaling, but it's closer."Non-parametric Scaling", with deep learning"Parametric Scaling"It's different in kinetics.

We can try to establish this connection, but we have to see them.MathsDifferences over. The following is a further summary of the discussions that took place earlier:

Re-examine: Are the tree models really only U-shaped curves? (Connects Double Discent)

The tree model mentioned earlier usually follows the U-shaped curve, but this is just one"The conditional truth."I'm sorry. The latest research (e.g. the work of the Mikhail Belkin team) shows that the government is not willing to take the initiative to stop the violence.Under certain conditions, tree models can also show a single-tune drop or a scaling Law-style drop or"Double down."The phenomenon.

Random Forest"It's a good thing."

Random forests are the traditional way to get closer to Scaling Law instincts.

Mathistic form: RF is Bagging. $$ f_{RF}(x) = \frac{1}{M} \sum_{m=1}^M T_m(x) $$

Scaling Behaviour: With the number of trees $M \to \infty$, the RF test error is usuallyUnarguedand constrict to a constant (non-approximate error + model deviation). It hardly fits because of too many trees (U-style right side not to go up).

Links to Deep Learning: It's like a deep study.Width ScalingI'm sorry. The hyper-widening neural network can also be understood as the convergence of many sub-networks from the perspective of Ensemble.

Variance: RF's Los, the speed of decline usually follows statistics. $1/\sqrt{M}$ or $1/M$ It's a very fast harvest, but the ceiling is hard to break. And Deep Learning Scaling Law often crosses more orders.

GDDT"Double down."Possibilities

For Boosting (e.g. XGBoost), the traditional view was that it was too much to say. But recent experiments have found that if ** completely uncut** and with very low learning (Shrinkage) $\to 0$Boosting can also observe a similar double decline.

Explanation: When the tree is extremely deep (over parametrically) each tree is over-composed to some of the defects. But if the learning rate is low enough, this is a match."Slow"And the next tree will be going on the noise in front."Align"I'm sorry. This simulates in some way the iterative process of SSD.

Core spread: two distinct"Scaling Paradigm"

Even though they can be Scaling, tree models and nervous networks are climbing two different peaks. We need to understand the geometry of the difference between the efficiency of the two Scaling.

Tree model 's Scaling (Tiling / Partitioning):

Operation: Tree models approach target functions by cutting space continuously (Axis-aligned splits).

Complexity: It's ** Local**. To get close to a high-dimensional sphere, the tree model needs to cut out thousands of tiny squares to take the sphere."Spell"Come out.

Scaling Trouble: Curse of DemensionI'm sorry. For each additional characteristic dimension, the number of trees (parameters) required to maintain the same near accuracy will need to be increased exponentially (in the case of the number of parameters).$2^D$)。

Conclusions The Scaling of the tree model is extremely effective at low dimensions (tables) but very low at high levels (images/text).

Neurosciente Scaling (Commonposition / Folding):

Operation: Neural networks distort space through linear transformation (rotation/extension) and non-linear activation (crash).

Complexity: It's...Global and Composition Yeah. Deep-end networks do not need to cut space, but simply learn a function to learn the flow structure of data.

Scaling Advantages: This one. Combiningability This allows the neural network to express the complex function of the index (as evidenced by Telgarsky et al. in 2016) by linearly increasing the amount of parameters.

Conclusions The tree model. Scaling is..."Add French"The neuronet Scaling is..."Multiplication/complex") (approach efficiency with depth enhancement).

Memory vs. Understanding (Memoration vs. Internationalization)

Back to what we discussed.Smooth。

Tree extension: When we extend the GDDT to the extreme, it actually becomes one. k-Nearest Neighbor (kNN) Or check the tab. It remembers training data perfectly.

On test: For the new sample, it's just going to check."Which leaves are closest to me?"I'm sorry. The plug is...Non-slippin steps。

Network expansion: When network parameters are sufficient, it is more likely to learn through the sample point under suitable optimization and normal conditions.Relative smooth flow。

On test: For new samples, it is moving upstream in the flow.

Conclusions Scaling Law really needs to explain, not just"Bigger"And it's also..."Why do you keep a relatively smooth forecast structure when you get bigger?"I'm sorry. When the tree model becomes larger (if not made more normal), space becomes easier to make.Broken。

Scaling Law?

What do you want to do if the tree model has the same Scaling capability as Transformer? This points to some of the integration directions of the current AI study.

Soft tree / Neural tree (Differentiable / Soft Trees)

If we put the tree in the model,"Hard"Yes. if x > 0.5 then left else right I'm gonna..."Soft"Yes. Sigmoid(x - 0.5)(that is, to the left with this probability)

• And the tree becomes nuanced.
• Trees become a special form of neural network (a complete layer of connections).
• Results: This one."Neural tree."You can use SSD training, and you can get closer to the Zooming Conditions of the Neural Network.
• A revelation: This suggests that the benefits associated with Scaling Law may depend not only on the name of the structure (tree vs web) but also on the name of the structure.Micro-diverseBring it here.Gradient OptimizationandContinuous flow pattern。

Missing Representation Learning

Scaling Law's success on LLM is largely due to the fact that models grow with size and learn better. Embeding (identity)。

• Tree model: Usually, it is directly divided on the original characteristic. It does not create new features, but only combines old ones.
• Reform: If we add a Transformer to the GDDT front to make the feature extraction, then then the GDDT to the header. So this whole system is compatible with Scaling Law. But at this point, it is thanks to Transformer, not to GGBDT.