Cross-validation

What is cross-validation?

Cross-validation estimates how well a machine learning model will perform on data it has not seen, by splitting the same dataset several ways and scoring each one. Instead of trusting a single held-out slice, you rotate which rows do the testing and average the results, so the number you report does not hinge on one lucky split.

It is a routine step in supervised learning. You fit a model on labelled examples and want an honest read on how it behaves on new records. A single train-test split gives one score from one arrangement of the data, and on a small dataset that score can swing a lot depending on which rows land in the test fold. Cross-validation uses the data more efficiently and shows that variance, though you still hold out a separate test set for the final number and to watch for model drift.

How k-fold works

The most common form is k-fold. You split the rows into k equal folds and train k times. Each round holds one fold back for testing and trains on the other k-1, so every row is tested once and trained on k-1 times.

Say you have 500 rows and pick k of 5. That gives five folds of 100. Round one trains on folds 2 to 5 (400 rows) and scores on fold 1, round two trains on the rest and scores on fold 2, and so on. You end up with five scores: their average is your estimate, and their spread shows how stable it is.

Most teams use k of 5 or 10, and scikit-learn uses 5 folds by default. Higher k trains more models and costs more compute, which bites when each model is a slow neural network.

Stratified, grouped and time-ordered folds

Plain random folds assume every row is independent. When that breaks, k-fold gives a misleading score and you need a different splitter.

• Stratified k-fold. Each fold keeps roughly the same class proportions as the full dataset. If 5 percent of your rows are fraud cases, every fold keeps about that 5 percent instead of one fold drawing almost none. That is what you want under class imbalance, and scikit-learn uses it automatically for classifiers.

• Grouped k-fold. When several rows belong to the same entity (one customer, one patient, one machine), they must stay in one fold. Otherwise the model sees patient A in both training and testing and looks better than it is. GroupKFold keeps each group on one side of the split.

• Time-series split. When rows are ordered in time, a random split lets the model train on the future and test on the past, which never happens in production. TimeSeriesSplit trains on earlier rows and tests on later ones.

Fit preprocessing inside each fold

Cross-validation only gives an honest number if every step that learns from the data is refit inside the loop. Scale or impute on the whole dataset first, and information from the test folds has already leaked into training. The scores come out too optimistic, and you find out only once the model is live.

The fix is to wrap the preprocessing and the model in a single pipeline and pass that to the cross-validation call. scikit-learn then refits the scaler on each training fold and applies it to the held-out fold, exactly as at prediction time. Preprocessing by hand across the full dataset is the most common source of data leakage in model evaluation.