Tree-Based Models Are Still SOTA for Tabular Data: XGBoost, LightGBM, and CatBoost

This English version was machine-translated from the Chinese original. Technical terms may require verification.

If the main battlefield of image, voice and natural language processing is already studied in depth today, then table data always resemble another completely different rhythm. There is no unified spatial structure, nor is there a natural continuous local model, which is replaced by hybrid type fields, missing values, abnormal values, long tail categories, operating rules, limited sample volumes, and unenviable engineering costs.

The questions in this article can also be addressedAutoGluon: Simplify machine learning Baseline to several lines of code、From Bagging to Stacking: an integrated learning methodology mapHow the concept of a relatively close read together is developed in different contexts.

And so, for the first time in the industrial machinery, many people feel that they are “era-fault”: Transformer, proliferation models and multi-modular systems on the one hand, and a large number of projects in wind control, advertising, referral, search, marketing, medical and industrial surveillance on the other, are still carefully compared. XGBoost、LightGBM and CatBoost。

It's not nostalgia, it's not a historical remnant of tree models. More precisely:In a large number of structured table tasks, tree upgrading remains the most common strong baseline and, in many scenarios, the best option or near best option. They have not been absent for a long time, for the simple reason that they remain valid.

This paper will not expand the whole ICF or discuss in detail the pedagogical extrapolation of random forests, Bagging and original GDDTs. I would like to focus on three of the most representative modern tree-upgrading systems:XGBoost, LightGBM and CatBoostI'm sorry. They all come from the GDDT spectrum, but the focus of the problem is completely different:

• XGBoost is a common, robust, manageable baseline selection.
• LightGBM is the industrial system that pushes mass training and memory efficiency to the top.
• CatBoost is a re-programming of the system around the type characteristics and target leaks.

Understanding the differences between them is far more important than remembering “who is faster, who is more accurate and who supports the class characteristics”.

This term “still SOTA” is not to say that they are always first in all data sets, all tasks and all input modes. More precisely, they remain the strongest baseline, common best solution or at least the closest type of method in the large number of structured table tasks.

Trey, why aren't you out yet?

Before entering into three models, answer one root question: Why is it that in 2026, Tree-based Boosting is still strong on the table task for a long time?

Isomer field: tables are never evenly spread

The image is the rule grid, the text is sequenced, the voice is continuous; the table is much more confusing. Each column in Table Gerry may have a completely different semantic meaning: some are continuous values, some are high base category, some are Boolean fields, some are timetamps, times, percentages, and even string IDs and a small number of text. It is not a natural type of input space, but rather a isomer space that is composing business syntax.

The tree model is exactly what fits this imbalance. It does not require all features to share the same geometric structure or a uniform smooth transformation, but rather it cuts the sample space directly by “whether a field is larger than the threshold” and “whether a category falls into a pool”.For tables, Tree tends to be more inversely biased than a neuronet to the shape of the original problem.

Small and medium samples: data from the real world may not be large enough to reach End

The strengths of in-depth learning are largely based on big data, pre-training resources and end-to-end expressions of learning ability. However, many real operations do not have such conditions. The sample of tabular tasks in the enterprise may be tens to hundreds of thousands, with a large number of features but of variable quality and distribution with marked long tails and operational deviations.

In such cases, a lifting tree with strong summary bias, clear parameters control and a more friendly approach to small and medium samples tends to give more stable results.Many tasks are not as large as they have to rely on heavy loads to indicate learning.

Missing and thin: default status of industrial data

In the real table data, the " Unfilled " " " Not occurs " " System " , " System " is valid only for certain categories of users " , is almost a default status, not an exception. The missing values themselves often carry business information; the rare features are also daily in the tables.

The Tre-based model is natural here. A number of Boosting frames can directly incorporate missing values into the divide logic without a single round of rough fillings. For high-dimensional thinness, tree models are also often closer to the luxuries of information only.

Feature interaction: first meet the conditions before the next step is discussed

A number of table tasks are not derived from a single field, but from a conditional interaction between fields. For example, “overdue and recent trades have been abruptly reduced” “Equipment types are of certain types and geographically in some cities” “high number of hits but low cost per passenger unit”. Such models are neither linear nor simple non-linear, but more like a set of conditioned f-else rules.

The tree model is structured precisely in this language: it cuts out a local area and continues to subdivided it. And that's why Trey is particularly sensitive to such issues as “determinating the next step when conditions are established”.

Project constraints: Explanatory, training costs and iterative efficiency

Industrial machines never learn more than their limit precision. Training time, parameter searches, effectiveness stability, feature importance, SHAP access, line delay and deployment price will all alter model selection.

The trees are strong for the long term, also because they form mature ecology in these dimensions. Teams often need not the SOTA in the sense of the paper, but rather the stable handling of the task today.

From Random Forest to GDDT: Before the official content begins

The main character of this paper is not random forest, nor is it original GDDT, but a short mattress is still needed to keep the discussions from going.

Random Forest: The classic strong baseline on Bagging's route

Random Forest's idea is to train multiple trees in parallel, to vote or to average, along the Bagging route, with a technical focus on reducing the difference. It remains a strong baseline and suitable for use as an introductory model and control group.

But it is not the focus of this paper, because I will then write a separate article on integrated learning, which will make it clear in a single framework that Bagging, Boosting, Random Forests, and the Difference-Equal Balance.

GLDT: Common parent of three modern lifting trees

If the random forest is Bagging, then the GDDT is Boosting:

$$ F_t(x) = F_{t-1}(x) + \eta f_t(x) $$

The tree here. $f_t$ Instead of independent training, the current model has not been completed. Intuitively, it is constantly correcting errors; more strictly, it is moving in the direction of loss in the functional space.

Original GDDT is important because XGBoost, LightGBM and CatBoost are all on this main line. But in today ' s engineering practice, what really affects experience and performance is how these modern achievements deal with target functions, split search, missing values, class characteristics, memory layouts and parallel efficiency.

XGBoost: Move GGBDT from "good experience" to "methodology."

XGBoost: A Scalable Tree Boosting System The impact is not just from speed. It's a practical way of moving GDDT forward.Target functions are clear, regularized, approximate algorithms are complete, thin process mature, work solidThe tree system.

Why is it the first stop of many people's defaults?

The observation form is a learning exercise and shows a steady pattern: When the task is not fully tacted, people are often willing to try an XGBoost first. The reason is not mysterious, but rather the sense of balance that is present in the whole picture, rather than the occasional speciality of an indicator.

It is strong and stable, it is flexible in target functions, it is relatively clear in the logic of hyperparameters and it is natural to process missing values. XGBoost often offersA reliable Baseline.It is used to clarify the structure of the problem and then to determine whether to pursue speed, class characteristics or some specific optimization more aggressively.

Math Identity: Structure Regular is written directly into the target function

One of the most significant contributions of XGBoost is to clearly define the optimization of each new tree round as a target with a complex punishment:

$$ \mathcal{L}^{(t)} = \sum_i l\big(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)\big) + \Omega(f_t) $$

The tree's regular item reads:

$$ \Omega(f) = \gamma T + \frac{1}{2}\lambda \sum_{j=1}^{T} w_j^2 $$

Here. $T$ The leaves count,$w_j$ It's the first. $j$ A leaf's weight. The idea behind this is:The trees are not as complex as they are, and each leaf must prove worth it. $\gamma$ The complexity of the punishment structure,$\lambda$ The leaves are not too heavy to weigh.

Training core: second stage roll-out, leaf weight closed and split main

Another key point for XGBoost is that it does not look at the first step gradient, but rather at the second step of the loss approximation:

$$ \tilde{\mathcal{L}}^{(t)} \approx \sum_i \left[g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i)\right] + \Omega(f_t) $$

of which $g_i$ It's a step gradient.$h_i$ It's a second-stage gradient. This second-tier perspective brings two direct benefits.

First, the optimal weight of the leaves is closed to the structure of the given tree:

$$ w_j^* = -\frac{G_j}{H_j + \lambda} $$

Second, the benefits of the candidate division can also be written in a prominent way:

$$ \text{Gain} = \frac{1}{2}\left(\frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L + G_R)^2}{H_L + H_R + \lambda}\right) - \gamma $$

This thing makes the "separation" in the XGBoost a clear optimisation.Gains must be substantial enough to cover structural complexity penalties.

Project core:exact、approx、hist With the right-winged quantile sketch

Many people still have the impression that XGBoost is in precise greddy search, but this is not the most common view in modern practice. According to the official Tree Methods Documents, XGBoost, have three main split paths:

• exact: accurate greeny search, accurate but slow, least scalable;
• approx: Construct a candidate point using a weighted bitsketch;
• hist: First, do a global partition, then build on histogram training, which is usually the most common route today.

It's particularly worth naming here. weighted quantile sketchI'm sorry. It does not disperse the continuous feature by hand, but is designed to be a similar structure specifically designed for the weighted fraction, aligning the structure of the candidate cut points with the optimisation logic of the second tier.

This is also a typical feature of XGBoost: it is not necessarily the most radical, but it is often complete enough.

Scatter and Missing: To treat absence as part of the building

XGBoost sparsity-aware split finding The default direction for each node to learn the missing values. That is, when a sample is missing on the current split, left or right, not pre-coded, but in training.

This brings two practical advantages: first, you do not need to fill all missing values in a rough round to get the model running; second, for high-dimensional thin input, it is closer to the partially utilised amount of information only.

Therefore, the "XGBoost Supported Missing Value" is stated as a simple functional introduction. More precisely:It would see the missing as part of the need to optimize together in a divisive structure.

What do you think of today?

It was often summed up by a phrase: XGBoost supports non-ingenital type characteristics. That statement is outdated today. According to the official Categorical Data The program, which supports the class data from 1.5. 3.1 Release Notes,3.1 Version started this capability by removing the executive label.

But this needs to be more detailed:

• Categorical spit dependency hist or approx, do not support exact；
• Models should normally be stored using JSON / UBJSON, otherwise information on categories may be lost;
• It supports one-hot and part-based split, but the "class feature processing" is still not its core design centre.

Today, XGBoost cannot handle the categorical; but it would be too much to understand if it were to be understood as a CatBoost alternative.

It's the strongest scene.

The most natural battlegrounds in XGBoost are the following types of tasks:

• Numerical characteristics are the main table tasks;
• A robust baseline is needed to determine whether to pursue further projects that are either insinuated or treated by category;
• Special task(s) requiring special objective, such as sorting, survival analysis, counting modelling, fractional regression;
• (b) The scenario where the missing value is high and the model is expected to absorb missing structural information;
• Industrial systems that require fine particle control training processes.

Misunderstanding and borders: robust, not the equivalent of almighty

With respect to XGBoost, these are often the most common types of misunderstandings:

• "XGBoost is just faster than GDDT." That's way underestimating it. It's worth making a structure, a goal-full, a mature system.
• "XGBoost does not support class characteristics." This is outdated today, but its methodological focus on the characteristics of the category is still less central than CatBoost.
• "XGBoost's strength is still the exact greeny." In modern practice,hist It is often the mainstream.
• "Leave value is 0." For trees, the two concepts are not equal, and the missing direction is to learn in training.

Its borders are also clear: on a very large, high-dimensional and thinly broad scale, its ingestion and memory efficiency is often less radical than that of LightGBM; nor is it the most natural first choice in the task of the high base category.

Summary of the sentence

The advantage of XGBoost is not always the fastest, but...Steady, universal, manageable.I'm sorry. If you don't know which tree you should start with, XGBoost is often the most natural starting point; if you don't know what model you're going to use, it's always the first stop.

LightGBM: When bottlenecks turn into swallowing, memory and width size

If XGBoost is complete and robust, then LightGBM: A Highly Efficient Gradient Boosting Decision Tree It's more like a distinct engineering system from the beginning: it's focused on design.Large-scale training speed, memory efficiency and wide-ranging vomitingGo on.

Why is it more engineering from the beginning?

LightGBM is so common in advertising, recommendation, search sequencing, CTR estimates, etc., not because it simply takes a little speed on XGBoost, but because it has reorganised the training package. According to the official Features Document, whose advantages are derived from a set of synergistic designs: histogram, histogram sub-section, leaf-wise growth, GOOSS, EFB, thin optimization and distributed friendly.

LightGBM closerA re-engineered Boosting system for training on large-scale industrial forms, not just another GDDT library.

Speed source: histogram and histogram subtration

LightGBM will first disperse the continuous feature to a limited number of bins, then find the best cut points on histogram. The benefits of this are straightforward: the cost of building histogram remains related to the sample number, but once hetogram is built, only the bin is scanned, without having to scan all original values repeatedly.

The bigger speed point is from histogram subtractionI'm sorry. The parent's node, histogram, is quickly available through the Parent's node minus bronode, as long as hetogram is clearly constructed for one of the subnodes. That sounds like the details to be achieved, but it is one of the important sources of the speed of training.

Structural philosophy: Leaf-wise why it's stronger and why it's more dangerous

LightGBM's most famous and easily misunderstood design is its default adoption. leaf-wise / best-first Growing, not more conservatively by layer. Each step of the way it picks the leaves that currently bring the largest loss of the losaic, and continues to split, rather than extending the whole layer down.

The result is straightforward: in the same leaf budget, leaf-wise tends to lower training losses more quickly and thus often provides a greater alignment capacity within a fixed time frame. But the problem is also from here:More radical, and usually more easily conciliated, is also implied.

So in LightGBM, controls are not always just seen. max_depth, and control it simultaneously. num_leaves、min_data_in_leafThe government has also been working on a project to improve the quality of the media. A lot of starters think it's a mistake. max_depth It's like turning LightGBM back into a level-wise, which is not true, because its growth philosophy itself is not changed.

Sample dimensions optimization: GOSS is not just subsample

It's from the LightGBM paper. GOSS（Gradient-based One-Side Sampling）, often misunderstood as another random sample. But it's different from the ordinary subsample's problem consciousness.

During the Boosting process, the absolute size of the gradients usually means that the models have not learned them well. So the GOSS strategy is not to delete samples evenly and randomly, but to:

1. (a) To retain a portion of the large gradient sample;
2. (a) Randomly extract a portion from a small gradient sample;
3. The latter is compensated for weighting to keep the estimates of gains as high as possible from being heavily distorted.

The instinct behind it is:What can't be lost easily is those samples that are not yet understood. This is more abosting than simple subsample training logic.

Characteristic dimensions optimization: why EFC is particularly suitable for width

Besides the sample dimensions, LightGBM has structural compression of the characteristic dimensions, which is EFB（Exclusive Feature Bundling）。

It observes that many of the rare features are mutually exclusive, such as a column after one-hot has been expanded, and that the same sample usually has only one column activated. Since these features are almost non-zero at the same time, they can be packaged into the same bookle, thereby significantly reducing histogram construction costs.

This is why LightGBM is so natural about high-dimensional slush tables such as recommendation, advertising, and searching for sorting. It is best handled, often not by dozens of neat numerical features, but by hundreds of, if not more, coded and spelled industrial broad-table fields.

Category support: it can handle the categoric, but not the CatBoost type of treatment

LightGBM original support for the categoric world, of course, it's worth it. But it's a category of characterization route and CatBoost is completely different. According to the official Advanced Topics document, which is essentially the best two-pointer in the order of the statistically ordered category, instead of ordered statistics.

It addresses issues more in favour of:How to efficiently divide categories; instead of treating category code leakage and Boosting deviations under the same principle as CatBoost.

At the same time, several borders must be remembered:

• Category characteristics should normally be coded as non-negative integers;
• Negative values are considered missing;
• High base category does not necessarily naturally fit natural, and official documents themselves caution this matter.

Missing value:zero_as_missing

LightGBM default supports missing values, but zero_as_missing Changes the semantic boundary between numeric 0 and missing values.

This is particularly dangerous in the rare matrix, the LibSVM format or the industrial characteristic of "0 without presence". If 0 is a state of clarity in business, and you treat it as a missing item, then the model sees it not as the watch you think it is.

With regard to LightGBM, it is not “it is fast” that deserves to be emphasized, but:It's quick, but it requires you to be sufficiently clear about the semantics of data.

It's the strongest scene.

LightGBM's most natural advantage scenario, which is basically these:

• (a) The number of samples is large, reaching hundreds of thousands, millions or even higher;
• The features are numerous and thin.
• Training in ingestion and memory occupation are major contradictions;
• The tasks are typical industrial issues such as CTR, recommendation, search sequencing, advertising, wind-control bandwidth;
• A large number of experimental iteratives need to be completed quickly.

In many cases, the value of LightGBM is not just in the final indicator, but in the way that you can run the experiment faster, adjust the parameters and sift the features.

Misunderstanding and boundaries: fast, not automatically saving hearts

With regard to LightGBM, the most common misconceptions include:

• "A primary supported category feature is equal to any string that can be eaten directly. " It still requires a reasonable integer code.
• "leaf-wise must be more comprehensive than level-wise." More precisely, it usually lowers the loss, but it's easier to match.
• "GOS is an alias of subsample." No, it is essentially "retain the hard sample + weight".
• "the high base category is naturally suitable for the LightGBM natural." This must be done with caution.
• “zero_as_missing It's just a little switch." It may actually change the semantic of data directly.

It's also clear: it's not always the least cost-effective in small data, especially when it's a validation protocol,num_leaves When the formal constraints are not controlled, the joint meeting is faster.

Summary of the sentence

LightGBM is the most like "High-intensity Boostling System for Large-scale Industrial Forms Learning". The main advantage of this is thatSpeed, memory and width matching- Not more stable.

CatBoost: Combining the category characteristics with the target leaking problem Rewrite

Put three together, CatBoost's highest-resolution. It's the most valuable place, not just to support class characteristics, but to be like, CatBoost: unbiased boosting with categorical features The paper highlighted:It deals with the deviations from the category characteristics and the Boosting itself, and it is reworked under the same design principle.

Why does it have a different sense of consciousness than the other two?

When people first met CatBoost, they remembered that it was good at dealing with categoric feature. Of course it is, but it is not enough. CatBoost, the real difference is that it's not about how to cut categories more efficiently, but about how to start:Will the statistics of the training phase leak back into the target information?

This makes it natural to focus on two other families. XGBoost places greater emphasis on target integrity and controlability, LightGBM places greater emphasis on insulation, memory and width efficiency, while CatBoost has been dealing with predation Shift and target leakage from the beginning.

First level: ordered handling predation shift

Traditional GDDTs are used for each round of training using the current model for the training set's disability or gradient. The problem is that these disabilities are themselves affected by the same training sample labels, and the statistical structure observed at the training and testing stages is not entirely consistent. CatBoost calls this deviation prediction shift。

The solution is... ordered boosting: randomly sorted training samples first; for a sample, in the $t$ The training statistics are constructed using only sample information before the array. Intuitively, it's to try to make the training sample more like a real new one.

This requires a boundary to be filled: the distinction between CatBoost's official parameters Ordered and Plain Two types of boosting, the default behaviour is affected by CPU/GPU, sample size and task type. Therefore, the more precise statement should be:Ordered Boosting is CatBoost's representative mechanism, but it is not the same as any configuration is defaulted on.

Second floor: avoid surface space

The most easily pedaled pit in the type characterization is the target leakage. Normal tranged encoding if the category target is calculated directly on the whole set, then the current sample's own label will leak back to its own signature expression.

CatBoost ordered target statistics This problem is mitigated by “using data only before the current sample”. One way to visualize is:

$$ \text{CTR}i = \frac{\text{countInClass}{<i} + \text{prior}}{\text{totalCount}_{<i} + 1} $$

Here<i. $ indicates only the records that appear before the current sample in the array. It is very clear:The current sample does not leak its own label into its own code.

Further, CatBoost also constructs group features. It makes it right. user_id × item_id、city × device、ad_slot × hour Such interaction is particularly sensitive and explains why it often behaves naturally in high-base category assignments.

Level 3: SymmetricTree gives it a high degree of visibility.

CatBoost's classic default tree structure is SymmetricTree(also commonly known as obblivious tree) It is characterized by the fact that all nodes on the same level use the same partition condition and therefore the depth is as follows: $d$ ♪ The trees will be ♪ $2^d$ A leaf.

This structure brings three things:

• (b) The reasoning path is well structured and online forecasts are usually fast;
• (a) A more structured and self-contained regularity;
• Expressional ability is also constrained, not only by profit.

CatBoost has been supported in the current official parameter document grow_policy = SymmetricTree / Depthwise / LossguideI'm sorry. The exact statement should therefore be:CatBoost's classic default form is derived from symmetric trees, but it does not support symmetric trees alone.

Time and text:has_time And the true meaning of the built-in text lines

CatBoost has two other points that deserve the attention of the industrial reader.

The first one is... has_timeI'm sorry. If the data are in a natural chronological order, then you do not want to do anything to disrupt the sample during the training phase and then construct the type statistics. Otherwise, much of the future information may be re-routed into the training process through a code sequence or cross-check process.has_time The point is to remind you:Order itself is part of the data structure.

The second is its text feature pipe:tokenizers -> dictionaries -> feature calcers -> numerical features -> boostingI'm sorry. The features calculators it supports include BoW, NaiveBayes, BM25, etc. This is practical, as many table tasks do mix short text fields, search terms, titles, label descriptions, etc.

But the border must also be clear:CatBoost's text support is not a tree model, Bert, but a classic text statistical feature project embedded in the Boosting system.

It's the strongest scene.

CatBoost's most important priority scenario, which is basically the following:

• (a) The characteristics of the category are numerous and many are high-base category;
• (a) The discrete fields of user ID, commodity ID, equipment, geography, channels, advertising posts are very strong;
• Not to write a large number of projects by hand;
• Data type heterogeneity, values, categories and small amounts of text;
• Hope to get a strong baseline quick without pre-treatment.

Misunderstanding and boundaries: good at categories, not the best for all tables

With regard to CatBoost, the most common misconceptions include:

• "CatBoost is just automatic target encoding." Actually, ordered Prince works both in code and boosting two layers.
• "CatBoost must be defaulted for ordered booping." This depends on the current configuration and training backend.
• "CatBoost only trains symmetric trees." This is not the case with official documents at the moment.
• "With CatBoost, there's no time-spacing." Still, the right time is to be split.
• "Standing support for CatBoost is equivalent to deep semantic modelling." No, it is still a text feature project in the tree model context.

Its boundaries are also clear: in pure numbers, super-sized, trained-to-smuggling priorities, LightGBM is often more attractive; and CatBoost’s core advantages are not necessarily released when the characteristics of the categories are not too numerous.

Summary of the sentence

CatBoost's real value is not "can also eat class features," but...The goal of avoiding leakage has been elevated into a uniform design principle and the matter has been incorporated into the category code and the Boostling training process.

Three routes: what are they optimizing?

If only three core differences are observed, they can be summarized in the table below.

Model	Design Centre	Tree Growth Policy	Category characterization processing	Speed/RAM	Default robustness	The best job.	Main costs
XGBoost	Universal Boosting, Visible Regulars, Flexible Targets	More conservative, frequently depth-wise	One-hot and partSpit support, but not design center	Medium Stable	High	Table with main values, missing values, needing robust baseline or special objective	A large-scale broad-band stale may not be a good idea.
LightGBM	Industrial ingestion, memory efficiency, broad-scale optimization	Defaultleaf-wise / best-first	Native class splits, but does not resolve	Usually the fastest, most economical memory	Medium	Million-degree samples, high-dimensional swirl list, CTR, recommendation, sorting	Small data is easier to match, and more semantic pits are available.
CatBoost	Category characterization modelling, leakage control, reduced manual code	Classic default is SymmetricTree	Ordered target statuses / CTR is the core advantage.	Training is usually slower than LightGBM. The reasoning is very well-defined.	High	High base category, ID feature intensive, isomeric tables, small text mixing	Training costs are higher. Pure values are too large to be optimal.

So the three really represent not “who is higher than whom”, but three completely different technical positions:

• XGBoost(a) robust, universal and manageable;
• LightGBMHigh-intensity, broad-mindedness, industrial efficiency priority;
• CatBoost: Class characterization modelling and leak control priority.

What do you choose when you actually land?

In engineering practice, the most important is not back-model history, but rather the development of enforceable selection principles.

1. If the features are numerical, it is hoped that a robust, manageable, ecologically mature baseline will be obtained first, with priority for the XGBoost test.
2. If it is faced with a large, high-dimensional, thin-wielding scale, training for ingestion and memory efficiency is the first contradiction, giving priority to the LightGBM.
3. If the class characteristics, ID characteristics, high base numbers are so numerous, and you don't want to write a bunch of code logic yourself, try CatBoost first.
4. If the task is clearly sequenced, no model can substitute for the correct time-slicing validation. CatBoost's ordered idea is helpful, but assessing protocols is more important than model names.
5. If it is not clear who is better, all three are considered as a line candidate, with fair comparison under the same identity, cross-checking and evaluation indicators. The real credit is the out-of-fold result of your mission.

In other words, the key to the selection is not to ask who is the most advanced, but to ask:What kind of watch is my watch? Are my bottlenecks generalized, insinuated, class-specific, or are they the cost of characteristic engineering?

Concluding remarks

The competition in table data has never been a pure model complex. In many cases, the decision is not about who has more parameters, who is closer to the data structure, who is more fit for engineering constraints, who can more steadily bring the signal out of the table.

XGBoost, LightGBM and CatBoost are still important not because they are only three GGBDT libraries, but because they represent three very clear technical positions:

• XGBoost Representative Steady, universal, manageable.；
• LightGBM Representative High-intensity, broad-mindedness, industrial efficiency priority；
• CatBoost, on behalf of Category characterization modelling and leakage control priority。

A truly mature engineering judgement is never the dogma who says it's always the best, but rather the knowledge that:Better definitions are not always the same in the different tabular forms and data distributions.

References

XGBoost

• XGBoost: A Scalable Tree Boosting System
• XGBoost Tree Methods
• XGBoost Parameters
• XGBoost FAQ
• XGBoost Categorical Data
• XGBoost 3.1 Release Notes

LightGBM

• LightGBM: A Highly Efficient Gradient Boosting Decision Tree
• LightGBM Features (v4.5)
• LightGBM Advanced Topics (v4.5)

CatBoost

• CatBoost: unbiased boosting with categorical features
• CatBoost Unbiased Boosting
• CatBoost: Transforming categorical features to numerical features
• CatBoost Common Training Parameters
• CatBoost Parameter Tuning
• CatBoost: Transforming text features to numerical features