Hyperparameter

What is a hyperparameter?

A hyperparameter is a setting you choose before training a model, and it shapes how the training runs. The learning rate, the depth of a decision tree, the number of layers in a neural network, how many passes to make over the data: these are all hyperparameters. You set them. The model does not learn them.

That is the line between a hyperparameter and a parameter. Parameters are what the model learns from the data during training: the weights, the coefficients, the split points inside a tree. Hyperparameters are the dials you set beforehand that decide how those parameters get learned.

Think of baking a cake. The oven temperature and the baking time are settings you choose up front, the way hyperparameters are. The rise and the colour of the finished cake are the result, the way learned parameters are. You pick the settings, and the outcome follows from them.

Parameters versus hyperparameters

A small example makes the split clear. In a simple linear model written as y = b0 + b1x, the values b0 and b1 are parameters: the model learns them by fitting the data. How hard you penalise large coefficients to keep the model simple is a hyperparameter: you set it before training starts.

A random forest shows the same divide. The number of trees and how deep each tree may grow are hyperparameters you choose. The actual questions each tree asks at each split are parameters the model works out from the data. Set the dials, and training fills in the rest.

Common hyperparameters

Different models expose different dials, but a few come up again and again.

• Learning rate. How big a step the training takes on each update. Too high and it overshoots the target; too low and it crawls, or never gets there in the time you allow.

• Model size and complexity. Tree depth, the number of trees, the number of layers or units in a neural network. More capacity can capture more, but it also makes overfitting easier.

• Regularisation strength. How hard the training penalises complexity. It is the main lever for keeping a model from memorising its training data.

• Batch size and number of passes. How much data the model sees per update, and how many times it works through the full dataset.

How you tune hyperparameters

Since you cannot read the best settings off the data, you search for them. The plain approaches are grid search, where you try every combination from a list you define, and random search, where you sample combinations at random and often find a good one faster. More guided methods, such as Bayesian optimisation, use earlier results to decide what to try next.

Whichever method you use, judge each combination with cross-validation, not the final test set. Rotating the held-out portion across folds gives a steadier read on which settings actually generalise. Because a tuning run produces dozens of combinations, experiment tracking earns its place: record every setting and its score so you can tell why one model beat another and reproduce the winner later. This bookkeeping is a core habit in MLOps.

What to watch out for with hyperparameters

Do not tune on the test set. If you pick settings by their score on the same data you use for the final verdict, that verdict is inflated. It is a form of data leakage. Keep a clean split between tuning and final evaluation.

Returns diminish. Searching a huge space of combinations burns compute for smaller and smaller gains. Sensible defaults exist for most algorithms, and better features often beat a marginally better learning rate.

Write them down. A model is only reproducible if you know the exact settings that produced it. Undocumented hyperparameters are the reason a result cannot be repeated a month later.