Experiment tracking

What is experiment tracking?

Experiment tracking is the practice of recording every training run in machine learning together with what produced it: the parameters you set, the code version, the dataset version, the metrics it scored, and the files it created. With that record you can compare two runs honestly and rebuild the one that won.

It works like a lab notebook for model training, but one that captures the ingredients for you. It is a core part of MLOps, the work of moving models from a notebook into production.

What a run record contains

A run is a single execution of your training code. A useful record holds four kinds of information:

• Parameters. The settings you chose before training, such as learning rate, tree depth, number of estimators, and model type.

• Metrics. The scores the run produced: accuracy, precision, recall, AUC, or a loss value logged at each step.

• Artifacts. The output files, from the trained model itself to a confusion matrix or an evaluation report.

• Context. The git commit of the code, the dataset version, the environment, the owner, and the timestamps. This is the part people skip, and the part that makes a run reproducible.

Tools such as MLflow, Weights and Biases, and Azure Machine Learning capture most of this for you; Weights and Biases even records the git commit and a diff of any uncommitted changes.

The failure it prevents

Picture a model that scored an AUC of 0.91 in someone's notebook and impressed everyone on a slide. Weeks later you want it in production, but nobody can rebuild it: the notebook was edited, the data snapshot is gone, and the random seed was never written down. You retrain, get 0.83, and cannot explain the gap.

That is the exact problem experiment tracking solves. A result you cannot reproduce is an anecdote, not an asset; a run record turns "it worked once" into something you can rerun.

What reproducible actually requires

Logged metrics on their own do not make a model reproducible. A score tells you the result, not the recipe. To rebuild the same model you need three things pinned down:

• The exact code commit, not just "the training script", so a later edit cannot change what you rerun. This is where Git (version control) earns its place.

• The dataset version, a specific snapshot or hash, because "the customers table" looks different next month.

• A fixed random seed, so the same code on the same data lands on the same weights every time.

Tracking is what keeps those three auditable after the author has moved on.

Experiment tracking versus model registry

Experiment tracking is the research log: every run you tried and what it scored, winners and dead ends alike. A model registry is the release shelf: the few model versions you keep, stage, and roll back. The best run in your tracker can graduate into a registered version, but most runs never leave it.

Both sit next to other MLOps parts. Feature engineering shapes the inputs a run trains on, and model serving is what happens once a registered version goes live.