Practical Handling of Class Imbalance: When and How to Act

This English version was machine-translated from the Chinese original. Technical terms may require verification.

And the first thing that happens to people is that they're not even."The few are too few."I'm sorry. But if the understanding is to stop at this point, it is easy to follow a series of costly operations that do not necessarily serve business objectives. The real available frame isJust follow the sequence and take a few steps.First, to decide whether to process it, then to hold the assessment, then to start with the cheapest means, and then to consider changing the data.

The questions in this article can also be addressedIntroduction to Machine Learning: Monitoring Learning and the Bayesian ApproachHow the concept of a relatively close read together is developed in different contexts.

Step one: to decide whether to deal with it or not.

Before addressing imbalances, make a shift:Don't ask first."Should I, SMOTE?", and ask"I'm going to optimize ranking, probability, or the final benefit of decision-making."。 These three objectives are not evenly priced in the uneven scenario:

• Just care.Sort(Is the formality above the negative ones: with indicators such as AUROC/PR-AUC, many times without doing anything;
• Care.Probability(0.8 of output is really close to 80%): a calibrated probability output is required;
• Care.Final proceeds(A different cost of misreporting than of underreporting): the movement of thresholds and sensitivity of costs are key.

- Put it on."Samples are uneven"Dismantling into three phenomena that are often mixed should be approached differently:

1. Uneven training cluster categories: Most common scenes, many of the head type samples, few of the tail type.
2. Estimates of probability of rare events: a small probability of event (failure rate, risk of complications, failure rate) is to be estimated steadily.
3. Pre-deployment deviation: Training sets differ from test sets, type a priori (label Shift) for online traffic.

When is it really not necessary to deal with it?

• The character is sufficiently different.: Standard models can be learned well when a few categories and most categories are virtually non-overlapping in characteristic space, even at a ratio of 110000;
• There's enough of them.: When the total is a million, the tail class is 0.1% and there are thousands of them, not bottlenecks;
• The tree model can carry a part of it.XGBoost / LightGBM, which enhances trees, is highly adaptive to moderate imbalances;
• Business indifferent tail-type performance: When you only care about the whole sort or the type of accuracy of the head, you deliberately raise tail and lower the main target.

Conclusion: First, to verify whether the imbalance is really hurting the indicators of your concern, then to decide to stay. Sometimes the problem is not proportional, but the quality of the feature or the noise of the label.

Step two: hold the evaluation. Don't be fooled by Accuracy.

The blogger says that the government is not taking the initiative.The quality of ranking, probability and decision-making must be separated Talk.I'm sorry. Details of the algorithms and selection of specific indicators are available.Monitoring of learning performance assessmentHere is the conclusion of practice:

• Under the strong imbalance, the PR curve tends to reflect actual pressure more than ROC: ROC takes in a lot of real negatives, and many models are going to look like"It's nice.";PR Space more directly exposes the misstatement of costs.
• At least report concurrently on the selection of indicators, PRUC, per-class and thresholds, multi-classification long end with the macro indicator and head/tail layer results. The only way to get a report is to be too optimistic.

Step 3: Starting with the cheapest - threshold movement and weight

As long as the model gives a reasonable probability sequence, many questions simply do not require data modification:

• Threshold moving: Move decision-making thresholds from default 0.5 to business cost equivalent. The most cost-effective first step is not to repeat training or disrupt training distribution. Attention. F1 Best Threshold is not"Common threshold"It has some anti-intuitive nature - the threshold is decision-making, not default.
• Category weightsXGBoost scale_pos_weightLightGBM is_unbalance / Category weight, equal to the weight given to the sample, to preserve the original distribution and to avoid pre-testing the test set.
• Cost-sensitive learning: If the cost of the omission and the misrepresentation are different, the default 0.5 threshold is not natural. The absence of a high-risk transaction in the fraud detection is far more expensive than reviewing several normal transactions; medical early screening is more likely to be called back in exchange for false reporting. The costs are directly written into the loss function or decision threshold and the weight sample is cleaner.

Step 4: Change data before it is enough - re-sampling

The first two steps are not a solution (extreme imbalance, small categories surrounded by most categories, high cost asymmetries) before data are moved. Three types of approach:

• Undesired sampling (underersampling): Remove some negatives that bring the number of positives and negatives closer. The disadvantage is that most types of information are lost, and the improved version is EasyEnsemble• Disaggregation of the inverse into several sets for different learning devices, each of which is unsampled, but information is not lost across the board.
• Oversampling: Add some positive examples. Note that the initial rule cannot be simply copied, otherwise it would cause serious overload; normally, the plug is used to generate additional rule.SMOTE It is the insertion of composite samples of values in a few local adjacent areas. It works if there are plugged structures in a few types of neighborhoods. - I'm sorry. If a few categories are caught in most categories or are labelled with noise, synthetic samples only contaminate the boundary.
• Integrated + Sample: Embedding samples into integration processes, such as SMOTEBoost, which is more focused on hard-to-train tail samples per round.

The most critical sequence discipline: first split the training/test set and then sampled only within the training set. SMOTE, then split, will allow training and testing to share local neighbourhood structures, and the results are very high -- the most classic data leak.

Special target: Probability is what rare events require.

If the target is not..."Distinguishing the positive and negative."and estimates the probability of rare events (failure rate, failure rate, risk of complications),Standard models systematically underestimate the probability of events when the normal pattern is scarce.— The uneven impact is not only the classification of boundaries, but also the parameter estimates themselves. The blind re-sampling is not the first option at this time:

• Keep original aforecheck, do it prior correction(Assisting the sample back to real a priori);
• After training, do it alone.Probability calibration（Platt / Isotonic / Temperature Scaling）；
• Don't take the rebalancing score directly as a risk probability.— Oversampling and weighting are all rewritten training a priori, and model output scores may not be interpreted as a real world backsliding probability.

Deep model and pre-training age: Longtail is not that important.

The core difficulty of traditional long-term learning is that tail class means unstable- There are too few samples to learn reliable characterizations. But the massive pre-training model (CLIP, DINOV2, Big Language Model, etc.) changed this premise:

• tail class means instability is mitigated by pre-training(a) Pre-training has learned fairly common expression space and there is a good enough starting point for less downstream samples of the tail type;
• Zero samples / Few samples capacity directly bypassed long tails: Visual language models can even describe text description of the type of training that has never been seen before in the training.

So in the scene that I'm watching,Longtail study is not so important anymore.I'm sorry. If the depth model still has to address the imbalance, the point is:"The decorating means learning and categorizing. Set"And focus on calibration, not stacking of Focal Los, LDAM, such weighting techniques.

Common error areas (checklist)

1. I took samples before the splitting.- Data leak. Always cut first, then take samples only inside the training set.
2. Only Accuracy or AUROC— may be overly optimistic; at least report both the provalence, PR indicators, per-class indicators.
3. Telling stories on the balance test set, but defaulting on the actual deployment a priori.- Balanced benchmark is not a realistic a priori.
4. Consider the increase of tail-class precision as a probability-based increase— Improved classification and improved probability must be tested separately.
5. Make imbalance the only problem, ignoring overlaps and noises.— When a few categories are mixed in most categories or with noise, the weighting is only more rigorous in the learning of noise.

Page Selection

scene	First option	What can be folded?	Don't come up here and do it.	Suggested indicators for reporting
Small sample table II classification	Layer-specification, class weight, threshold-adjusted	Moderate sampling or EASYEnsemble in training set	SMOTE before split; only Accuracy	PR-AUC、Balanced Accuracy、MCC
Raise Tree-Major Data Task	Verify first whether the processing is really needed; move the scale pos wait + threshold if necessary	Cost-sensitive, small sample	Blind SMOTE	PR-AUC, per-class, confusion matrix
Probability of rare events	Keep original aforesee, logical regression / GGBDT+ calibration	Case-control sampling, project control, sootonic/temperature scaling	Make the rebalancing fractions the probability of risk.	Brier、Calibration、PR-AUC
Wind control / Medical care / required for reliable probability output	Define deployment a priori and cost before training	calibration、threshold moving、label-shift correction	I'll just watch my own, and I'll be on the line.	Calibration curve、Brier、ECE

Default order of operation:Define the target first(Sorting / Probability / Proceeds) Hold the evaluation.(Accessorator set close to deployment) First, make a simple baseline.(class weight, threshold movement, correct indicators, header correction) And then the complicated way.(sampling, weighting) Last chance of re-entry( calibration).

Concluding remarks

Let's just say the first line:A truly stable imbalance is not a one-man skill, but a one-man skill."Training objectives, evaluation indicators, deployment decisions"Three-man consistent design. In most cases, holding assessments, moving thresholds and re-aligning weights have resolved most of the problems and changing data is always at the end. When you see the imbalance as a whole link from data distribution to deployment decision-making, the method is easier to choose.