Decision trees and gradient boosting

What is a decision tree?

A decision tree is a chain of yes or no questions about one row of data, ending in an answer. Has this customer ordered from us more than three times before? If yes, is the quote under 10,000 euro? If yes, predict a 68 percent chance of winning it. You can print the whole thing on a page and trace any prediction from top to bottom, which is why scikit-learn's documentation calls a tree a white box model.

The questions are chosen by the computer, not by you. The algorithm looks at every column and every possible cut-off point in it, takes the split that best separates won quotes from lost ones, and repeats that inside each branch until there is nothing useful left to split or it hits a limit you set.

Why one tree is not enough

Left to grow freely, a tree keeps splitting until nearly every leaf holds a single training row. At that point it has memorised the training data instead of learning a pattern. Prune it back to five or six levels and the opposite happens: genuinely different customers land in the same leaf with the same score.

Then there is instability. scikit-learn puts it bluntly in its own list of disadvantages: small variations in the data might result in a completely different tree being generated. Retrain next month with three hundred extra rows and the column at the top can change. A model that reads so nicely but tells a different story every quarter is not one to build a sales process on.

From one tree to a forest, and from a forest to boosting

The fix is to stop relying on one tree. There are two ways to combine many of them, and the difference between the two is the whole story.

Bagging, which gives you a random forest. Train a few hundred trees at once, each on a random sample of the rows drawn with replacement, and each allowed to consider only a random subset of the columns at every split. The trees end up wrong in different directions. Average their answers and a good share of those errors cancel each other out. scikit-learn describes the point of that double randomness exactly so: it decouples the errors of the individual trees, so averaging removes some of them.

Boosting. Train one small tree. Look at what it still gets wrong. Train a second tree whose only job is to predict that error, and add its output to the first tree's output. Keep going for a few hundred rounds, adding each tree in small steps controlled by a learning rate, so no single tree can swing the answer far.

What the word gradient means here

Nothing you need calculus for. After each round the model has a prediction for every training row and you know the real answer, so per row you know two things: which way that prediction has to move, and how far. That pair is the gradient, and the next tree is fitted to predict it. When you are predicting a number and measuring error as the squared difference, the gradient is exactly the leftover, the real value minus the current prediction, which is why boosting is often explained as fitting the residuals.

Random forest versus gradient boosting

Compare the two on one dimension: how the trees are combined. In a random forest they grow in parallel, never see each other, and count equally in the final vote. In gradient boosting they grow in sequence and every tree exists only because of the mistakes the earlier ones made, so the ensemble is a running sum in which order matters.

A random forest is therefore hard to break: adding more trees does not make it worse, and the defaults are usually fine. Gradient boosting ends up a few points more accurate on the same table, but it will overfit if you let it run too long or grow the trees too deep, so it needs a validation set and a stopping rule. Train the forest first as your floor, then see how much boosting adds on top.

The three implementations you will meet

Three open source libraries dominate this corner of machine learning, and all three do gradient boosting over decision trees. They differ in how they grow the trees and in what they do with categorical columns.

XGBoost gives the plainest statement of the method in its own tutorial: fix what has been learned, and add one new tree at a time. What sets it apart is that the penalty for tree complexity sits inside the objective the algorithm optimises rather than bolted on afterwards, something the same page notes most tree packages treat less carefully or skip. It has handled categorical columns natively since version 1.5, switched on with the enable_categorical flag.

LightGBM buckets continuous values into a fixed number of bins before training and searches for splits over those bins, which makes it fast and light on memory. It also grows leaf-wise: instead of finishing one level before starting the next, it splits whichever leaf promises the largest reduction in error, wherever that leaf sits. Its own tuning guide is candid about the cost, warning that leaf-wise growth may overfit without the right parameters and pointing at num_leaves and min_data_in_leaf. The library started at Microsoft and moved to its own GitHub organisation in March 2026.

CatBoost comes from Yandex, and its handling of categorical columns is why people reach for it. Instead of turning a column of two hundred postcodes into two hundred new columns, it replaces each category with a number derived from the target, computed using only the rows that come earlier in a random shuffle. A row never contributes to its own encoding, which is what stops the trick from leaking the answer into the feature.

Which one you pick matters less than the internet suggests: on a table of a few thousand to a few hundred thousand rows they usually land within a point or two of each other. scikit-learn also ships HistGradientBoostingClassifier, which uses the same histogram approach and needs nothing extra installed.

Why these models beat deep learning on business tables

In 2022 a team at Inria, the French national institute for research in digital science and technology, benchmarked tree-based models against neural networks across 45 tabular datasets, with a very large hyperparameter search behind every learner so neither side could win on tuning effort alone. Tree-based models still came out ahead on medium-sized data of around ten thousand rows. The three reasons the paper gives all describe the data an SME actually has: columns that carry no signal, which throw a neural network off while a tree simply never splits on them; hard thresholds like a discount above fifteen percent or an invoice past thirty days, which trees cut at by construction while neural networks prefer smooth functions; and columns that each mean one thing, where mixing them into new combinations is right for the pixels of a photo and wrong for a table.

Next to the accuracy argument sit the properties that decide whether an SME ever gets a model into production. These models work from a few hundred rows up to millions, take numeric and categorical columns in the same table without a preprocessing pipeline in front of them, and handle missing values by learning at each split whether the rows with a gap belong left or right. They train in seconds to a few minutes on a laptop, so you can try twenty ideas in an afternoon. And they hand you a ranked list of which columns mattered, which is the part you can put in front of a sales manager.

Where a language model fits

Not in this job. A language model has never seen your history, cannot learn a threshold from five thousand rows pasted into a prompt, and costs orders of magnitude more per row scored. Where it does earn its place is one step earlier, turning free text into a column: the topic of the last support ticket, the tone of the customer's last email, whether a tender document mentions a maintenance contract. That column then joins the table the boosted model trains on.

Worked example: which quotes will convert

An installation firm sends around 1,400 quotes a year and wins about thirty percent of them. Four years of history gives roughly 5,600 rows, of which around 1,700 were won. The owner wants to know which open quotes deserve a follow-up call this week. The target column: was this quote accepted within 60 days of being sent, yes or no. Twelve features, all known on the day the quote went out:

1. Quote total in euro

2. Number of line items

3. Discount percentage applied

4. Days between the customer's request and the quote being sent

5. Whether a site visit happened before quoting

6. New customer or existing customer

7. Number of previous orders by that customer

8. Euro value of those previous orders

9. Sector of the customer

10. Province

11. Sales rep who sent it

12. Month it was sent

A LightGBM model with five-fold cross-validation trains in under ten seconds on a normal laptop and lands at an AUC of about 0.74. That is not a spectacular number and it does not have to be. It means that if you pick a won quote and a lost quote at random, the model gives the won one the higher score about three times in four. On a follow-up list of thirty names a week, that is the difference between calling at random and calling with a reason.

Reading the feature importance honestly

The chart puts days between request and quote at the top, then quote total, then whether a site visit happened.

Importance is not causation. The model has found that quotes sent fast get accepted more often. It has not found that sending quotes faster causes acceptance. The likelier story runs the other way round: a customer ready to buy pushes for a quote and gets one within two days, while a vague enquiry sits in someone's inbox for a fortnight. Both patterns produce the same column at the top of the same chart. The only way to tell them apart is to change something and measure it: quote half of next quarter's requests within 48 hours no matter how warm they feel, and compare.

Watch the columns with many distinct values. The importance number a tree model gives you by default is computed on the training data and, as scikit-learn's documentation warns, it favours features with many unique values. A province column with eleven values is treated fairly. A postcode column with four hundred would float to the top on cardinality alone. Permutation importance, which shuffles one column and measures how far the score drops on data the model never trained on, is the version to trust.

Be careful with the sales rep column. If one rep shows high importance, the model may have learned that this person handles the repeat customers who were going to buy anyway. That is a fact about how the work is allocated, not about the rep.

What to watch out for with gradient boosting

Overfitting when nothing holds the model back. Every extra tree fits the training data a little better whether or not there is anything left to learn. Hold back data the model never sees, or use cross-validation, and stop when the score on that data stops improving.

A leaked column. A boosted model finds any column that quietly contains the answer, and faster than a person would. In the quotes example, a field called order_number is filled in only for quotes that were accepted. Leave it in and the model reaches an AUC of 0.99 and is worthless. A suspiciously good score is a leakage check first and a celebration second.

Predictions outside the range the model has seen. A tree predicts by dropping a row into a leaf and returning that leaf's average, so it cannot extrapolate. scikit-learn states it plainly: the predictions are piecewise constant, so trees are not good at extrapolation. If the largest quote in four years is 60,000 euro, a new 250,000 euro quote gets scored as though it were a 60,000 euro one.

Categories the model never saw while training. A new sales rep, a new sector code, a supplier added last month. LightGBM treats an unseen category at prediction time as a missing value, which is sensible as a default but means those rows get a generic answer until they have history behind them.

Too few rows for the number of columns. Three hundred quotes and forty features gives you a model that has learned the individual quotes. Cut the list back to the ten or twelve columns the business genuinely believes in and keep the trees shallow.