SyntaxPilot
Courses/Machine Learning/Lesson 80
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 80 of 90

Experiment tracking and model registry

experiment trackingrunslogging parametersmetrics and artifactsMLflowmodel registrymodel versioningstages (staging, production)data versioningDVCend-to-end reproducibility

Machine Learning Full Course · Episode 80. Every program and command below was run on a 2-core Linux machine with Python 3.13, scikit-learn 1.9.1, MLflow 3.18.0, DVC 3.67.1 and Git, and the output shown is the real output. Run IDs and random results without a seed will differ on your machine. The loan data is made up: random numbers (NumPy seed 78) with a hidden rule.

The big idea: after weeks of experiments, a team must answer "which model is live, how was it made, and can we make it again?" Without records, even the same code gives different results (unfixed randomness). Experiment tracking writes every run down: its parameters (inputs), metrics (numbers) and artifacts (files). MLflow does this in a few lines. A model registry gives each shipped model versions and movable aliases (champion), replacing the older stages (Staging, Production). DVC versions the data: Git stores a tiny .dvc file with the data's hash, DVC stores the data. Reproducibility needs four things pinned: code (Git commit), data (DVC hash), parameters and seed (MLflow run) and environment (Docker image, Episode 79).

Sources: MLflow docs, MLflow Tracking ("an API and UI for logging parameters, code versions, metrics, and output files"; runs are "executions of some piece of data science code"; each run records metadata such as metrics, parameters, start and end times, and artifacts, output files such as model weights and images; "An experiment groups together runs and models for a specific task."). DVC, User Guide ("Data Version Control is a free, open-source tool for data management, ML pipeline automation, and experiment management."). The stage deprecation is the real FutureWarning printed by MLflow 3.18.0 (shown below).

Words you need first

  • Experiment (everyday sense): one attempt at a model, with one choice of algorithm and settings.
  • Hyperparameter: a setting you choose before training, like C (Episode 23) or max_depth (Episode 30).
  • Random seed: a number that makes "random" choices repeat exactly (Episode 2). Same seed, same split, same forest.
  • Hash (MD5): a short fingerprint computed from data. Change even one byte and the hash changes completely.
  • Git: the tool that versions code: every commit is a saved snapshot you can go back to.

The story: the lost experiment

A month of experiments, files called model_final.joblib, model_final_v2.joblib, model_FINAL_real.joblib, and the boss asks: which one is live, with which settings and data, and can you make it again? The obvious attempt: run the same code again.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(78)            # made-up loan data: income, loan, credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
for attempt in ["last Monday", "today"]:   # "the same experiment", run twice
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2)        # no seed
    model = RandomForestClassifier(n_estimators=50, max_depth=6).fit(X_tr, y_tr)  # no seed
    print(f"{attempt:12} test accuracy = {model.score(X_te, y_te):.4f}")

One real run (yours will differ, which is the point):

last Monday  test accuracy = 0.9390
today        test accuracy = 0.9230

Why: train_test_split and the random forest both use randomness, and no seed was fixed or written down. And nothing recorded the settings or results.

Real-life example: the cricket scorebook

At a cricket match In ML
The scorer writes every ball while it happens Experiment tracking: log every run automatically
Pitch, toss, playing eleven: the inputs Parameters: model type, C, depth, seed, data hash
Runs scored, and the match ball kept Metrics (accuracy) and artifacts (files, like the model)

MLflow tracking

  • MLflow Tracking: "an API and UI for logging parameters, code versions, metrics, and output files" (MLflow docs).
  • Run: one execution of your training code. with mlflow.start_run(): opens one.
  • Experiment (MLflow): a named group of runs for one task: mlflow.set_experiment("loan-risk").
  • log_param / log_params: record inputs. log_metric: record a number. log_artifact: save a file with the run.
  • Tracking URI: where the records are stored. sqlite:///mlflow.db = a SQLite database file in the current folder.
  • search_runs: query runs as a pandas DataFrame. mlflow ui: a web page listing every run, with charts.
import os; os.environ.update(MLFLOW_DISABLE_AGENT_HINT="1", MLFLOW_LOGGING_LEVEL="ERROR")   # quiet MLflow messages
import hashlib, numpy as np, mlflow; from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression; from sklearn.ensemble import RandomForestClassifier
tries = [("logreg", {"C": 0.01}), ("logreg", {"C": 1}), ("forest", {"max_depth": 3}), ("forest", {"max_depth": 8})]
rng = np.random.default_rng(78)            # made-up loan data: income, loan, credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
data_hash = hashlib.md5(X.tobytes() + y.tobytes()).hexdigest()[:8]     # fingerprint of the data
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=80)
mlflow.set_tracking_uri("sqlite:///mlflow.db"); mlflow.set_experiment("loan-risk")
for kind, params in tries:
    with mlflow.start_run(run_name=f"{kind}-{list(params.values())[0]}"):         # one run = one experiment try
        model = (LogisticRegression(max_iter=1000, **params) if kind == "logreg"
                 else RandomForestClassifier(100, random_state=80, **params)).fit(X_tr, y_tr)
        mlflow.log_params({"model": kind, **params, "seed": 80, "data_md5": data_hash})   # inputs
        mlflow.log_metric("test_accuracy", model.score(X_te, y_te))                       # result
        mlflow.log_dict({"features": ["income", "loan", "credit_score"]}, "card.json")    # a file (artifact)
runs = mlflow.search_runs(order_by=["metrics.test_accuracy DESC"])
print(runs[["tags.mlflow.runName", "params.data_md5", "metrics.test_accuracy"]].to_string(index=False))
tags.mlflow.runName params.data_md5  metrics.test_accuracy
        logreg-0.01        aa96e07d                  0.922
           logreg-1        aa96e07d                  0.921
           forest-8        aa96e07d                  0.914
           forest-3        aa96e07d                  0.897

Each row carries its settings and the same data fingerprint aa96e07d. The environment variables MLFLOW_DISABLE_AGENT_HINT and MLFLOW_LOGGING_LEVEL=ERROR only silence MLflow's informational messages so the output stays readable. mlflow.log_dict saves a small JSON file as an artifact of the run.

Real-life example: the restaurant menu board

In a restaurant In the model registry
Dishes tried in the kitchen; most never reach customers Runs
"Butter chicken", recipe v1, v2: one name, new versions Registered model with versions
A "chef's special" tag, moved to another dish any day Alias: champion → version 2

The model registry

  • Model registry: one shared catalogue of the models a team ships.
  • Registered model: a name, like loan-risk. Model version: each registration under that name (1, 2, 3...), linked to the run that produced it.
  • Stages (old way): None → Staging (being tested) → Production (live) → Archived. Still common in older projects and interviews, but deprecated in MLflow since 2.9.0.
  • Aliases (new way): movable names such as champion (live) and challenger (being tested). Serving code loads models:/loan-risk@champion, never a file path. Shipping = moving the alias; rollback = moving it back.
import os; os.environ.update(MLFLOW_DISABLE_AGENT_HINT="1", MLFLOW_LOGGING_LEVEL="ERROR")   # quiet MLflow messages
import warnings, contextlib, io, numpy as np, mlflow
from mlflow import MlflowClient; from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78)            # made-up loan data: income, loan, credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
mlflow.set_tracking_uri("sqlite:///mlflow.db"); mlflow.set_experiment("loan-risk"); client = MlflowClient()
for C in [1.0, 0.01]:                       # two trainings -> two versions of ONE registered model
    with mlflow.start_run(), contextlib.redirect_stderr(io.StringIO()):     # (hide MLflow's progress text)
        m = LogisticRegression(C=C, max_iter=1000).fit(X, y)
        mlflow.sklearn.log_model(m, name="model", registered_model_name="loan-risk", input_example=X[:1])
for v in client.search_model_versions("name='loan-risk'"): print("version", v.version, "from run", v.run_id[:8])
client.set_registered_model_alias("loan-risk", "champion", version=2)    # production points here
champ = mlflow.sklearn.load_model("models:/loan-risk@champion")
print("champion -> version", client.get_model_version_by_alias("loan-risk", "champion").version)
print("Asha's risk from the champion:", round(champ.predict_proba([[50, 7, 690]])[0, 1], 3))
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always"); client.transition_model_version_stage("loan-risk", 2, "Production")  # old way
    print(w[0].category.__name__ + ":", str(w[0].message).split(". To")[0])

Real output (contextlib.redirect_stderr hides MLflow's progress messages, such as "Created version '1' of model 'loan-risk'."):

version 2 from run 989ba96f
version 1 from run a801421c
champion -> version 2
Asha's risk from the champion: 0.667
FutureWarning: ``mlflow.tracking.client.MlflowClient.transition_model_version_stage`` is deprecated since 2.9.0. Model registry stages will be removed in a future major release

Data versioning with DVC

The data changes silently: someone appends 200 rows to loans.csv. Git is built for code; large data files make a repository slow and huge. DVC (Data Version Control): "a free, open-source tool for data management, ML pipeline automation, and experiment management". How it works:

  • dvc add data/loans.csv computes the file's MD5 hash, moves the data into DVC's cache, and writes a tiny data/loans.csv.dvc file.
  • Git versions the .dvc file; DVC adds the real file to data/.gitignore.
  • Remote storage (dvc push / dvc pull): a shared place for the data, such as a cloud bucket or a network drive.
  • dvc status shows data that changed; dvc checkout restores the data matching the current .dvc file.

The full real session:

$ git init -q && dvc init -q
$ dvc add data/loans.csv

To track the changes with git, run:

    git add data/.gitignore data/loans.csv.dvc

To enable auto staging, run:

    dvc config core.autostage true
$ cat data/loans.csv.dvc
outs:
- md5: 24f21f4922cd0b8308ac9e8c6ae85b4e
  size: 91028
  hash: md5
  path: loans.csv
$ git add data/loans.csv.dvc data/.gitignore && git commit -qm "data v1"
$ # someone appends 200 new loans
$ dvc status
data/loans.csv.dvc:
    changed outs:
        modified:           data/loans.csv
$ dvc add data/loans.csv && git commit -qam "data v2"
$ git log --oneline
7f38476 data v2
2221d6f data v1
$ git checkout -q HEAD~1 -- data/loans.csv.dvc && dvc checkout
M       data/loans.csv
$ wc -l data/loans.csv
5001 data/loans.csv

Before going back, wc -l data/loans.csv printed 5201 data/loans.csv (version 2: 5,000 rows + 200 + the header). After git checkout HEAD~1 -- data/loans.csv.dvc and dvc checkout, it is 5001 again (version 1).

End-to-end reproducibility

Reproducible: anyone can rebuild the same model and get the same numbers. Pin four things and log them in every run:

What Pinned by
Code a Git commit ID
Data the DVC hash (md5)
Parameters + seed the MLflow run
Environment the Docker image tag (Episode 79)

Rebuilding a month-old run from its record alone:

import os; os.environ.update(MLFLOW_DISABLE_AGENT_HINT="1", MLFLOW_LOGGING_LEVEL="ERROR")   # quiet MLflow messages
import hashlib, numpy as np, mlflow; from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
def load_data(extra_rows=0):                 # made-up loan data; extra rows = someone "updated" the file
    rng = np.random.default_rng(78); n = 5000 + extra_rows
    X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1), rng.normal(700, 60, n).clip(300, 900)]
    y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, n) > 20).astype(int)
    return X, y, hashlib.md5(X.tobytes() + y.tobytes()).hexdigest()[:8]
def train(X, y, seed, depth):
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed)
    return RandomForestClassifier(50, max_depth=depth, random_state=seed).fit(X_tr, y_tr).score(X_te, y_te)
mlflow.set_tracking_uri("sqlite:///mlflow.db"); mlflow.set_experiment("loan-risk"); X, y, md5 = load_data()
with mlflow.start_run() as run:              # last month: the run we shipped
    mlflow.log_params({"seed": 80, "max_depth": 6, "data_md5": md5})
    mlflow.log_metric("test_accuracy", train(X, y, 80, 6))
d = mlflow.get_run(run.info.run_id).data; p, ref = d.params, d.metrics["test_accuracy"]   # a month later
for extra in [0, 200]:                       # same data file, then a silently changed one
    X, y, md5 = load_data(extra); acc = train(X, y, int(p["seed"]), int(p["max_depth"]))
    print(f"data {md5} (logged {p['data_md5']}): {acc:.4f} vs {ref:.4f}", ["DIFFERENT", "REPRODUCED"][acc == ref])
data aa96e07d (logged aa96e07d): 0.9160 vs 0.9160 REPRODUCED
data f4b29ff4 (logged aa96e07d): 0.9298 vs 0.9160 DIFFERENT

Same data hash → exactly the same accuracy. Changed data → a different hash, so the reason for the different accuracy is visible immediately.

Common mistakes

Mistake What you see (real) Why Fix
No seed two different accuracies from the same code random split and model fix and log the seed
Results only in a notebook "which settings gave 0.92?" nothing recorded one MLflow run per try
File names as a registry model_FINAL_real.joblib: nobody knows what is live names aren't versions registered model + alias
New code using stages FutureWarning: ... deprecated since 2.9.0. Model registry stages will be removed in a future major release stages are being removed aliases (champion, challenger)
Data not versioned dvc status: modified: data/loans.csv noticed too late, or never data changed silently dvc add + commit the .dvc file
Big data files in Git slow, huge repository Git is built for code DVC cache + remote storage
Logging params but not data can't rebuild the model one of four pins missing log code, data hash, params, image

Mini project: a tracked, versioned loan model

  1. git init, dvc init, export the loan data to data/loans.csv, dvc add it and commit.
  2. A train.py that reads the CSV, computes its MD5, and logs model, its hyperparameters, seed, data_md5 and the Git commit (git rev-parse HEAD) as parameters, plus test_accuracy.
  3. Train several models; register the best as loan-risk and set champion.
  4. Change the data, commit version 2, retrain, set the new version as challenger, and compare.

Practice

1. Log ten random-forest runs with max_depth 1 to 10 and print the best depth.

Solution
for d in range(1, 11):
    with mlflow.start_run(run_name=f"forest-{d}"):
        m = RandomForestClassifier(100, max_depth=d, random_state=80).fit(X_tr, y_tr)
        mlflow.log_params({"model": "forest", "max_depth": d, "seed": 80})
        mlflow.log_metric("test_accuracy", m.score(X_te, y_te))
best = mlflow.search_runs(order_by=["metrics.test_accuracy DESC"]).iloc[0]
print(best["params.max_depth"], best["metrics.test_accuracy"])

2. Set a challenger alias on version 1 and load it.

Solution
client.set_registered_model_alias("loan-risk", "challenger", version=1)
chal = mlflow.sklearn.load_model("models:/loan-risk@challenger")

3. Why does git log show both data versions, but the repository stays small?

Solution

Git only stores the small .dvc files (hash, size, path). The data itself lives in DVC's cache (.dvc/cache) and remote storage.

4. Two runs have the same parameters and seed but different accuracies. What do you check first?

Solution

The logged data hash (did the data change?), then the code commit and the environment (library versions or image tag).

Quick quiz

Version 3 of loan-risk passed every test and the API loads models:/loan-risk@champion. What is the safest way to make version 3 live?

Answer

Move the champion alias to version 3. No code change; moving it back is an instant rollback.

Interview questions

  1. What is the difference between a parameter, a metric and an artifact in MLflow? Parameters are inputs (hyperparameters, seed, data hash), metrics are numeric results (accuracy, loss, possibly per step), artifacts are files (model, plots, model card).
  2. Why use a model registry instead of a folder of model files? One shared source of truth: named models, numbered versions linked to their runs, and aliases for what is live, so shipping and rollback are explicit and auditable.
  3. Stages vs aliases? Stages were fixed labels (Staging, Production, Archived), deprecated in MLflow since 2.9.0. Aliases are custom, movable names; the serving code loads a model by alias.
  4. Why not put datasets in Git? Git keeps every version of every file in full history; large binary data makes the repository slow and huge. DVC keeps a small pointer file in Git and the data elsewhere.
  5. What do you need to reproduce a model exactly? The code version, the exact data, the parameters including seeds, and the environment.
  6. A model's offline accuracy can't be reproduced. How do you debug? Compare the logged data hash, code commit, library versions and seed with the original run; check for unseeded randomness.

Glossary

Term Meaning, with an example
Experiment tracking Recording every training run's inputs and results automatically.
Run One execution of training code. with mlflow.start_run():
Experiment (MLflow) A named group of runs for one task: loan-risk.
Parameter An input to a run: C=0.01, max_depth=8, seed=80, data_md5.
Logging Writing a value into the run's record: mlflow.log_params(...).
Metric A numeric result: test_accuracy=0.922.
Artifact A file saved with a run: card.json, the model itself.
MLflow An open-source platform for the ML lifecycle: tracking, models, registry.
Tracking URI Where MLflow stores records: sqlite:///mlflow.db.
search_runs Query runs as a table, sorted by a metric.
mlflow ui A command that opens a web page with all runs and charts.
Random seed A number that makes random choices repeatable.
MD5 hash A short fingerprint of data: aa96e07d...; changes if any byte changes.
Model registry A shared catalogue of shipped models with versions and aliases.
Registered model A model name in the registry: loan-risk.
Model version One registration under that name: version 1, 2, 3, linked to its run.
Model versioning Giving every shipped model a number so you can compare, ship and roll back.
Stage Old MLflow label: None, Staging, Production, Archived (deprecated since 2.9.0).
Staging The stage for a version being tested before release.
Production The stage (or alias) for the version that is live.
Alias A movable name pointing to a version: champion → 2.
Champion / challenger The live model / the candidate being compared with it.
Rollback Going back to the previous version (move the alias back).
Data versioning Keeping every version of a dataset so you can go back to it.
DVC Data Version Control: versions data with Git-sized pointer files.
.dvc file A small file with the data's hash, size and path, committed to Git.
DVC cache The local folder where DVC stores the actual data versions.
Remote storage A shared place for DVC data (cloud bucket, network drive).
dvc status / dvc checkout Show changed data / restore the data matching the .dvc file.
Git commit A saved snapshot of code; its ID pins the code version.
Reproducibility Rebuilding the same model and getting the same numbers.
End-to-end reproducibility Pinning code, data, parameters + seed and environment together.

Summary

  • Without a seed and a record, the same code gave two different accuracies.
  • MLflow tracks runs: parameters, metrics and artifacts, queryable with search_runs and visible in mlflow ui.
  • The registry gives loan-risk versions linked to runs; aliases like champion decide what is live (stages are deprecated).
  • DVC versions data: the .dvc hash goes in Git, the data in a cache or remote; dvc checkout travels back in time.
  • Reproducibility = code + data + parameters/seed + environment, all logged. Same data hash: REPRODUCED.
  • Next: Episode 81, ML pipelines and orchestration.