Train-test split

What is a train-test split?

A train-test split holds part of your data back from training so you can score the model on examples it has never seen. The model learns on the training set, and you measure it on the test set. That score estimates how the model will do on new data, not on answers it already had.

You need the split because a model graded on its own training data flatters itself. A system that just memorised the labels would post a perfect score and still predict nothing useful on unseen data. That failure mode is called overfitting, and a held-out test set is how you catch it in machine learning.

In a supervised learning workflow the split is usually the first line of code you write:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

The ratio matters less than one rule: the test set has to resemble the data the model meets in production. An 80/20 split on the wrong axis still lies to you, and a strong score cannot rule out model drift.

Train, validation, and test

Once you start comparing models or tuning settings, two parts are not enough. The training set teaches the model. A validation set is what you check while you pick between algorithms or hyperparameters. The test set stays sealed until the end.

Every time you read the test score and change something to push it up, knowledge about the test set leaks into the model. Tune against it long enough and it turns into a second training set: the number still looks good, but it no longer says anything about new data. So you touch the test set exactly once, for the final read.

If you would rather not carve off a fixed validation slice, cross-validation is the common alternative, and a held-out test set still earns the final check.

When a random split is the wrong choice

The default train_test_split shuffles rows at random. That is right only when rows are independent and the future resembles the past. Two situations break that assumption.

• Time series. Sales, churn, and demand data are ordered in time, and nearby points are correlated. Shuffle them and the model trains on later days to predict earlier ones, which it never does in production. Split by time: train on the earlier period, test on the later one.

• Grouped data. When many rows belong to the same customer, patient, or device, that entity must sit entirely on one side. If one customer appears in both training and test, the model recognises the customer instead of the pattern, and the score is flattered. Split by group so an entity's rows stay together.

Both are forms of data leakage, easy to miss if you only read the headline accuracy.

Keeping rare classes in every split

When one outcome is far rarer than the other, like fraud or a positive medical screen, a random split can leave the test set with almost none of the rare class, so the score swings on a few cases.

A stratified split keeps the same class proportions in every part as in the full dataset, so a 1-in-20 positive rate stays about 1-in-20 on both sides. In scikit-learn you pass stratify=y, as in the snippet above. It is the standard first move whenever you are working with class imbalance.