Machine Learning Full Course · Episode 80. Every program and command below was run on a 2-core Linux machine with Python 3.13, scikit-learn 1.9.1, MLflow 3.18.0, DVC 3.67.1 and Git, and the output shown is the real output. Run IDs and random results without a seed will differ on your machine. The loan data is made up: random numbers (NumPy seed 78) with a hidden rule.
The big idea: after weeks of experiments, a team must answer "which model is live, how was it made, and can we make it again?" Without records, even the same code gives different results (unfixed randomness). Experiment tracking writes every run down: its parameters (inputs), metrics (numbers) and artifacts (files). MLflow does this in a few lines. A model registry gives each shipped model versions and movable aliases (champion), replacing the older stages (Staging, Production). DVC versions the data: Git stores a tiny .dvc file with the data's hash, DVC stores the data. Reproducibility needs four things pinned: code (Git commit), data (DVC hash), parameters and seed (MLflow run) and environment (Docker image, Episode 79).
Sources: MLflow docs, MLflow Tracking ("an API and UI for logging parameters, code versions, metrics, and output files"; runs are "executions of some piece of data science code"; each run records metadata such as metrics, parameters, start and end times, and artifacts, output files such as model weights and images; "An experiment groups together runs and models for a specific task."). DVC, User Guide ("Data Version Control is a free, open-source tool for data management, ML pipeline automation, and experiment management."). The stage deprecation is the real FutureWarning printed by MLflow 3.18.0 (shown below).
Words you need first
- Experiment (everyday sense): one attempt at a model, with one choice of algorithm and settings.
- Hyperparameter: a setting you choose before training, like
C(Episode 23) ormax_depth(Episode 30). - Random seed: a number that makes "random" choices repeat exactly (Episode 2). Same seed, same split, same forest.
- Hash (MD5): a short fingerprint computed from data. Change even one byte and the hash changes completely.
- Git: the tool that versions code: every commit is a saved snapshot you can go back to.
The story: the lost experiment
A month of experiments, files called model_final.joblib, model_final_v2.joblib, model_FINAL_real.joblib, and the boss asks: which one is live, with which settings and data, and can you make it again? The obvious attempt: run the same code again.
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(78) # made-up loan data: income, loan, credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
for attempt in ["last Monday", "today"]: # "the same experiment", run twice
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2) # no seed
model = RandomForestClassifier(n_estimators=50, max_depth=6).fit(X_tr, y_tr) # no seed
print(f"{attempt:12} test accuracy = {model.score(X_te, y_te):.4f}")
One real run (yours will differ, which is the point):
last Monday test accuracy = 0.9390
today test accuracy = 0.9230
Why: train_test_split and the random forest both use randomness, and no seed was fixed or written down. And nothing recorded the settings or results.
Real-life example: the cricket scorebook
| At a cricket match | In ML |
|---|---|
| The scorer writes every ball while it happens | Experiment tracking: log every run automatically |
| Pitch, toss, playing eleven: the inputs | Parameters: model type, C, depth, seed, data hash |
| Runs scored, and the match ball kept | Metrics (accuracy) and artifacts (files, like the model) |
MLflow tracking
- MLflow Tracking: "an API and UI for logging parameters, code versions, metrics, and output files" (MLflow docs).
- Run: one execution of your training code.
with mlflow.start_run():opens one. - Experiment (MLflow): a named group of runs for one task:
mlflow.set_experiment("loan-risk"). log_param/log_params: record inputs.log_metric: record a number.log_artifact: save a file with the run.- Tracking URI: where the records are stored.
sqlite:///mlflow.db= a SQLite database file in the current folder. search_runs: query runs as a pandas DataFrame.mlflow ui: a web page listing every run, with charts.
import os; os.environ.update(MLFLOW_DISABLE_AGENT_HINT="1", MLFLOW_LOGGING_LEVEL="ERROR") # quiet MLflow messages
import hashlib, numpy as np, mlflow; from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression; from sklearn.ensemble import RandomForestClassifier
tries = [("logreg", {"C": 0.01}), ("logreg", {"C": 1}), ("forest", {"max_depth": 3}), ("forest", {"max_depth": 8})]
rng = np.random.default_rng(78) # made-up loan data: income, loan, credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
data_hash = hashlib.md5(X.tobytes() + y.tobytes()).hexdigest()[:8] # fingerprint of the data
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=80)
mlflow.set_tracking_uri("sqlite:///mlflow.db"); mlflow.set_experiment("loan-risk")
for kind, params in tries:
with mlflow.start_run(run_name=f"{kind}-{list(params.values())[0]}"): # one run = one experiment try
model = (LogisticRegression(max_iter=1000, **params) if kind == "logreg"
else RandomForestClassifier(100, random_state=80, **params)).fit(X_tr, y_tr)
mlflow.log_params({"model": kind, **params, "seed": 80, "data_md5": data_hash}) # inputs
mlflow.log_metric("test_accuracy", model.score(X_te, y_te)) # result
mlflow.log_dict({"features": ["income", "loan", "credit_score"]}, "card.json") # a file (artifact)
runs = mlflow.search_runs(order_by=["metrics.test_accuracy DESC"])
print(runs[["tags.mlflow.runName", "params.data_md5", "metrics.test_accuracy"]].to_string(index=False))
tags.mlflow.runName params.data_md5 metrics.test_accuracy
logreg-0.01 aa96e07d 0.922
logreg-1 aa96e07d 0.921
forest-8 aa96e07d 0.914
forest-3 aa96e07d 0.897
Each row carries its settings and the same data fingerprint aa96e07d. The environment variables MLFLOW_DISABLE_AGENT_HINT and MLFLOW_LOGGING_LEVEL=ERROR only silence MLflow's informational messages so the output stays readable. mlflow.log_dict saves a small JSON file as an artifact of the run.
Real-life example: the restaurant menu board
| In a restaurant | In the model registry |
|---|---|
| Dishes tried in the kitchen; most never reach customers | Runs |
| "Butter chicken", recipe v1, v2: one name, new versions | Registered model with versions |
| A "chef's special" tag, moved to another dish any day | Alias: champion → version 2 |
The model registry
- Model registry: one shared catalogue of the models a team ships.
- Registered model: a name, like
loan-risk. Model version: each registration under that name (1, 2, 3...), linked to the run that produced it. - Stages (old way): None → Staging (being tested) → Production (live) → Archived. Still common in older projects and interviews, but deprecated in MLflow since 2.9.0.
- Aliases (new way): movable names such as
champion(live) andchallenger(being tested). Serving code loadsmodels:/loan-risk@champion, never a file path. Shipping = moving the alias; rollback = moving it back.
import os; os.environ.update(MLFLOW_DISABLE_AGENT_HINT="1", MLFLOW_LOGGING_LEVEL="ERROR") # quiet MLflow messages
import warnings, contextlib, io, numpy as np, mlflow
from mlflow import MlflowClient; from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78) # made-up loan data: income, loan, credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
mlflow.set_tracking_uri("sqlite:///mlflow.db"); mlflow.set_experiment("loan-risk"); client = MlflowClient()
for C in [1.0, 0.01]: # two trainings -> two versions of ONE registered model
with mlflow.start_run(), contextlib.redirect_stderr(io.StringIO()): # (hide MLflow's progress text)
m = LogisticRegression(C=C, max_iter=1000).fit(X, y)
mlflow.sklearn.log_model(m, name="model", registered_model_name="loan-risk", input_example=X[:1])
for v in client.search_model_versions("name='loan-risk'"): print("version", v.version, "from run", v.run_id[:8])
client.set_registered_model_alias("loan-risk", "champion", version=2) # production points here
champ = mlflow.sklearn.load_model("models:/loan-risk@champion")
print("champion -> version", client.get_model_version_by_alias("loan-risk", "champion").version)
print("Asha's risk from the champion:", round(champ.predict_proba([[50, 7, 690]])[0, 1], 3))
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter("always"); client.transition_model_version_stage("loan-risk", 2, "Production") # old way
print(w[0].category.__name__ + ":", str(w[0].message).split(". To")[0])
Real output (contextlib.redirect_stderr hides MLflow's progress messages, such as "Created version '1' of model 'loan-risk'."):
version 2 from run 989ba96f
version 1 from run a801421c
champion -> version 2
Asha's risk from the champion: 0.667
FutureWarning: ``mlflow.tracking.client.MlflowClient.transition_model_version_stage`` is deprecated since 2.9.0. Model registry stages will be removed in a future major release
Data versioning with DVC
The data changes silently: someone appends 200 rows to loans.csv. Git is built for code; large data files make a repository slow and huge. DVC (Data Version Control): "a free, open-source tool for data management, ML pipeline automation, and experiment management". How it works:
dvc add data/loans.csvcomputes the file's MD5 hash, moves the data into DVC's cache, and writes a tinydata/loans.csv.dvcfile.- Git versions the
.dvcfile; DVC adds the real file todata/.gitignore. - Remote storage (
dvc push/dvc pull): a shared place for the data, such as a cloud bucket or a network drive. dvc statusshows data that changed;dvc checkoutrestores the data matching the current.dvcfile.
The full real session:
$ git init -q && dvc init -q
$ dvc add data/loans.csv
To track the changes with git, run:
git add data/.gitignore data/loans.csv.dvc
To enable auto staging, run:
dvc config core.autostage true
$ cat data/loans.csv.dvc
outs:
- md5: 24f21f4922cd0b8308ac9e8c6ae85b4e
size: 91028
hash: md5
path: loans.csv
$ git add data/loans.csv.dvc data/.gitignore && git commit -qm "data v1"
$ # someone appends 200 new loans
$ dvc status
data/loans.csv.dvc:
changed outs:
modified: data/loans.csv
$ dvc add data/loans.csv && git commit -qam "data v2"
$ git log --oneline
7f38476 data v2
2221d6f data v1
$ git checkout -q HEAD~1 -- data/loans.csv.dvc && dvc checkout
M data/loans.csv
$ wc -l data/loans.csv
5001 data/loans.csv
Before going back, wc -l data/loans.csv printed 5201 data/loans.csv (version 2: 5,000 rows + 200 + the header). After git checkout HEAD~1 -- data/loans.csv.dvc and dvc checkout, it is 5001 again (version 1).
End-to-end reproducibility
Reproducible: anyone can rebuild the same model and get the same numbers. Pin four things and log them in every run:
| What | Pinned by |
|---|---|
| Code | a Git commit ID |
| Data | the DVC hash (md5) |
| Parameters + seed | the MLflow run |
| Environment | the Docker image tag (Episode 79) |
Rebuilding a month-old run from its record alone:
import os; os.environ.update(MLFLOW_DISABLE_AGENT_HINT="1", MLFLOW_LOGGING_LEVEL="ERROR") # quiet MLflow messages
import hashlib, numpy as np, mlflow; from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
def load_data(extra_rows=0): # made-up loan data; extra rows = someone "updated" the file
rng = np.random.default_rng(78); n = 5000 + extra_rows
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1), rng.normal(700, 60, n).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, n) > 20).astype(int)
return X, y, hashlib.md5(X.tobytes() + y.tobytes()).hexdigest()[:8]
def train(X, y, seed, depth):
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed)
return RandomForestClassifier(50, max_depth=depth, random_state=seed).fit(X_tr, y_tr).score(X_te, y_te)
mlflow.set_tracking_uri("sqlite:///mlflow.db"); mlflow.set_experiment("loan-risk"); X, y, md5 = load_data()
with mlflow.start_run() as run: # last month: the run we shipped
mlflow.log_params({"seed": 80, "max_depth": 6, "data_md5": md5})
mlflow.log_metric("test_accuracy", train(X, y, 80, 6))
d = mlflow.get_run(run.info.run_id).data; p, ref = d.params, d.metrics["test_accuracy"] # a month later
for extra in [0, 200]: # same data file, then a silently changed one
X, y, md5 = load_data(extra); acc = train(X, y, int(p["seed"]), int(p["max_depth"]))
print(f"data {md5} (logged {p['data_md5']}): {acc:.4f} vs {ref:.4f}", ["DIFFERENT", "REPRODUCED"][acc == ref])
data aa96e07d (logged aa96e07d): 0.9160 vs 0.9160 REPRODUCED
data f4b29ff4 (logged aa96e07d): 0.9298 vs 0.9160 DIFFERENT
Same data hash → exactly the same accuracy. Changed data → a different hash, so the reason for the different accuracy is visible immediately.
Common mistakes
| Mistake | What you see (real) | Why | Fix |
|---|---|---|---|
| No seed | two different accuracies from the same code | random split and model | fix and log the seed |
| Results only in a notebook | "which settings gave 0.92?" | nothing recorded | one MLflow run per try |
| File names as a registry | model_FINAL_real.joblib: nobody knows what is live |
names aren't versions | registered model + alias |
| New code using stages | FutureWarning: ... deprecated since 2.9.0. Model registry stages will be removed in a future major release |
stages are being removed | aliases (champion, challenger) |
| Data not versioned | dvc status: modified: data/loans.csv noticed too late, or never |
data changed silently | dvc add + commit the .dvc file |
| Big data files in Git | slow, huge repository | Git is built for code | DVC cache + remote storage |
| Logging params but not data | can't rebuild the model | one of four pins missing | log code, data hash, params, image |
Mini project: a tracked, versioned loan model
git init,dvc init, export the loan data todata/loans.csv,dvc addit and commit.- A
train.pythat reads the CSV, computes its MD5, and logsmodel, its hyperparameters,seed,data_md5and the Git commit (git rev-parse HEAD) as parameters, plustest_accuracy. - Train several models; register the best as
loan-riskand setchampion. - Change the data, commit version 2, retrain, set the new version as
challenger, and compare.
Practice
1. Log ten random-forest runs with max_depth 1 to 10 and print the best depth.
Solution
for d in range(1, 11):
with mlflow.start_run(run_name=f"forest-{d}"):
m = RandomForestClassifier(100, max_depth=d, random_state=80).fit(X_tr, y_tr)
mlflow.log_params({"model": "forest", "max_depth": d, "seed": 80})
mlflow.log_metric("test_accuracy", m.score(X_te, y_te))
best = mlflow.search_runs(order_by=["metrics.test_accuracy DESC"]).iloc[0]
print(best["params.max_depth"], best["metrics.test_accuracy"])
2. Set a challenger alias on version 1 and load it.
Solution
client.set_registered_model_alias("loan-risk", "challenger", version=1)
chal = mlflow.sklearn.load_model("models:/loan-risk@challenger")
3. Why does git log show both data versions, but the repository stays small?
Solution
Git only stores the small .dvc files (hash, size, path). The data itself lives in DVC's cache (.dvc/cache) and remote storage.
4. Two runs have the same parameters and seed but different accuracies. What do you check first?
Solution
The logged data hash (did the data change?), then the code commit and the environment (library versions or image tag).
Quick quiz
Version 3 of loan-risk passed every test and the API loads models:/loan-risk@champion. What is the safest way to make version 3 live?
Answer
Move the champion alias to version 3. No code change; moving it back is an instant rollback.
Interview questions
- What is the difference between a parameter, a metric and an artifact in MLflow? Parameters are inputs (hyperparameters, seed, data hash), metrics are numeric results (accuracy, loss, possibly per step), artifacts are files (model, plots, model card).
- Why use a model registry instead of a folder of model files? One shared source of truth: named models, numbered versions linked to their runs, and aliases for what is live, so shipping and rollback are explicit and auditable.
- Stages vs aliases? Stages were fixed labels (Staging, Production, Archived), deprecated in MLflow since 2.9.0. Aliases are custom, movable names; the serving code loads a model by alias.
- Why not put datasets in Git? Git keeps every version of every file in full history; large binary data makes the repository slow and huge. DVC keeps a small pointer file in Git and the data elsewhere.
- What do you need to reproduce a model exactly? The code version, the exact data, the parameters including seeds, and the environment.
- A model's offline accuracy can't be reproduced. How do you debug? Compare the logged data hash, code commit, library versions and seed with the original run; check for unseeded randomness.
Glossary
| Term | Meaning, with an example |
|---|---|
| Experiment tracking | Recording every training run's inputs and results automatically. |
| Run | One execution of training code. with mlflow.start_run(): |
| Experiment (MLflow) | A named group of runs for one task: loan-risk. |
| Parameter | An input to a run: C=0.01, max_depth=8, seed=80, data_md5. |
| Logging | Writing a value into the run's record: mlflow.log_params(...). |
| Metric | A numeric result: test_accuracy=0.922. |
| Artifact | A file saved with a run: card.json, the model itself. |
| MLflow | An open-source platform for the ML lifecycle: tracking, models, registry. |
| Tracking URI | Where MLflow stores records: sqlite:///mlflow.db. |
| search_runs | Query runs as a table, sorted by a metric. |
| mlflow ui | A command that opens a web page with all runs and charts. |
| Random seed | A number that makes random choices repeatable. |
| MD5 hash | A short fingerprint of data: aa96e07d...; changes if any byte changes. |
| Model registry | A shared catalogue of shipped models with versions and aliases. |
| Registered model | A model name in the registry: loan-risk. |
| Model version | One registration under that name: version 1, 2, 3, linked to its run. |
| Model versioning | Giving every shipped model a number so you can compare, ship and roll back. |
| Stage | Old MLflow label: None, Staging, Production, Archived (deprecated since 2.9.0). |
| Staging | The stage for a version being tested before release. |
| Production | The stage (or alias) for the version that is live. |
| Alias | A movable name pointing to a version: champion → 2. |
| Champion / challenger | The live model / the candidate being compared with it. |
| Rollback | Going back to the previous version (move the alias back). |
| Data versioning | Keeping every version of a dataset so you can go back to it. |
| DVC | Data Version Control: versions data with Git-sized pointer files. |
| .dvc file | A small file with the data's hash, size and path, committed to Git. |
| DVC cache | The local folder where DVC stores the actual data versions. |
| Remote storage | A shared place for DVC data (cloud bucket, network drive). |
| dvc status / dvc checkout | Show changed data / restore the data matching the .dvc file. |
| Git commit | A saved snapshot of code; its ID pins the code version. |
| Reproducibility | Rebuilding the same model and getting the same numbers. |
| End-to-end reproducibility | Pinning code, data, parameters + seed and environment together. |
Summary
- Without a seed and a record, the same code gave two different accuracies.
- MLflow tracks runs: parameters, metrics and artifacts, queryable with
search_runsand visible inmlflow ui. - The registry gives
loan-riskversions linked to runs; aliases likechampiondecide what is live (stages are deprecated). - DVC versions data: the
.dvchash goes in Git, the data in a cache or remote;dvc checkouttravels back in time. - Reproducibility = code + data + parameters/seed + environment, all logged. Same data hash: REPRODUCED.
- Next: Episode 81, ML pipelines and orchestration.