SyntaxPilot
Courses/Machine Learning/Lesson 81
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 81 of 90

ML pipelines and orchestration

MLOpsML pipelinefeature pipelinetraining pipelineinference pipelineorchestration (Airflow, Prefect, Kubeflow)DAGfeature storetraining-serving skewscheduled retrainingonline learning

Machine Learning Full Course · Episode 81. Every program and command below was run on a 2-core Linux machine with Python 3.13, scikit-learn 1.9.1, Prefect 3.9.2 and Apache Airflow 3.4.0, and the output shown is the real output (long orchestrator logs are trimmed, and marked as trimmed). The loan data is made up: random numbers with a hidden rule.

The big idea: a production ML system is not one script. It is a set of pipelines (feature, training, inference) made of tasks with dependencies, drawn as a DAG (Directed Acyclic Graph). An orchestrator (Airflow, Prefect, Kubeflow Pipelines) runs the DAG in the right order, on a schedule, with retries, alerts and history. The quietest bug in these systems is training-serving skew: the same feature computed differently in training and serving. A feature store prevents it by defining features once. Models stay fresh by scheduled retraining or online learning.

Sources: Apache Airflow docs, DAGs ("A DAG (Directed Acyclic Graph) is the core concept of Airflow, collecting Tasks together, organized with dependencies and relationships to say how they should run."). Kubeflow, Pipelines overview ("a platform for building and deploying portable and scalable machine learning (ML) workflows", which run in containers on Kubernetes). Prefect: the package's own summary, "Workflow orchestration and management." (prefect 3.9.2). Google, Rules of Machine Learning ("Training-serving skew is a difference between performance during training and performance during serving."; causes include "a discrepancy between how you handle data in the training and serving pipelines" and "a change in the data between when you train and when you serve"; Rule #29: save the features used at serving time and use them at training time). Feast ("an open-source feature store that helps teams operate production ML systems at scale"; an offline store for training data, an online store for low-latency serving, point-in-time correct feature sets that avoid data leakage).

Words you need first

  • Task: one step of work, like "fetch today's loans" or "train the model".
  • Dependency: "task B needs task A's output first". Features need the data; training needs the features.
  • Batch scoring: predicting for many rows at once on a schedule (Episode 78).
  • Alias / registry: the champion pointer in the model registry (Episode 80).
  • Kubernetes / container: a system that runs many containers across machines (Episode 79).

The story: the 2 a.m. script

Every night the bank wants to fetch the day's new loans, build features, train a fresh model and score all 2 lakh customers. Keeping jobs like this running reliably in production is MLOps (machine learning operations). The obvious attempt: one big script, started every night by cron (a Linux tool that runs a command on a schedule).

def fetch_new_loans():
    print("fetching today's loans...")
    raise ConnectionError("bank database timed out at 02:00")   # a short network glitch
def build_features(): print("building features (from which data?)")
def train_model(): print("training")
def score_all_customers(): print("scoring 2 lakh customers")
print("02:00 cron starts: python do_everything.py")            # one big script, every night
build_features()                                               # oops: runs before the new data arrives
fetch_new_loans()
train_model()
score_all_customers()
print("done")
02:00 cron starts: python do_everything.py
building features (from which data?)
fetching today's loans...
Traceback (most recent call last):
ConnectionError: bank database timed out at 02:00

Two problems, and no warning about either: features were built before the new data arrived (wrong line order), and one short network glitch lost the whole night (no retry, no alert). The script doesn't know which step needs which.

Real-life example: making dosa

In the kitchen In ML
Soak rice → grind → ferment → cook; each step needs the one before Pipeline of tasks with dependencies
You can't soak again after cooking: arrows only go forward DAG: no step loops back
The head cook re-grinds a failed batch, without re-soaking Orchestrator: retries one task, alerts, keeps history

Three pipelines

  • ML pipeline: the chain of steps from raw data to predictions, written as separate tasks.
  • Feature pipeline: raw data → clean features (e.g. "loan as years of income").
  • Training pipeline: features → trained, evaluated and registered model.
  • Inference pipeline: new inputs (+ features) → predictions, in batch or online.

DAG: Directed Acyclic Graph

  • Graph: dots (tasks) joined by lines (dependencies).
  • Directed: every line is an arrow, from the task that is needed to the task that needs it.
  • Acyclic: no loops, so there is always at least one order that runs every task after the tasks it needs (a topological order).
  • Orchestration: running the DAG for you: in order, on a schedule, with retries, alerts and a history of runs.

Orchestration from scratch

Python's built-in graphlib.TopologicalSorter finds a valid order from the dependencies:

from graphlib import TopologicalSorter            # Python's built-in DAG helper
result = {"fetch": "5000 new loans", "features": "features table", "train": "model, accuracy 0.922",
          "evaluate": "better than champion 0.921", "register": "registered version 3",
          "score": "2 lakh customers scored"}
dag = {"features": {"fetch"}, "train": {"features"}, "evaluate": {"train"},     # task: {tasks it needs}
       "register": {"evaluate"}, "score": {"register"}}
def run_task(task, retries=3):              # each task stands in for real work
    for attempt in range(1, retries + 1):
        try:
            if task == "fetch" and attempt == 1: raise ConnectionError("timed out")   # 2 a.m. glitch, first try only
            return result[task]
        except ConnectionError as e:
            print(f"  {task}: attempt {attempt} failed ({e}), retrying")
    raise RuntimeError(f"{task} failed {retries} times: alert the on-call person")
order = list(TopologicalSorter(dag).static_order())     # an order that respects every arrow
print("run order:", order)
for task in order:
    print(f"{task:9} -> {run_task(task)}")
run order: ['fetch', 'features', 'train', 'evaluate', 'register', 'score']
  fetch: attempt 1 failed (timed out), retrying
fetch     -> 5000 new loans
features  -> features table
train     -> model, accuracy 0.922
evaluate  -> better than champion 0.921
register  -> registered version 3
score     -> 2 lakh customers scored

The order comes from the arrows, not from where the lines were typed; fetch failed once and was retried.

Mistake: a loop in the DAG

Adding one line, "fetch needs score" (fetch again after scoring):

dag["fetch"] = {"score"}                      # mistake: "fetch again after scoring" makes a loop
Traceback (most recent call last):
graphlib.CycleError: ('nodes are in a cycle', ['features', 'train', 'evaluate', 'register', 'score', 'fetch', 'features'])

With a cycle no task can run first. To repeat the work tomorrow, run the whole DAG again on a schedule, never with a loop inside the DAG.

Real orchestrators

  • Apache Airflow: DAGs written in Python, a scheduler that starts them, and a web UI with every run's history.
  • Prefect: "Workflow orchestration and management": decorate plain Python functions with @task and @flow.
  • Kubeflow Pipelines: "a platform for building and deploying portable and scalable machine learning (ML) workflows", where every step runs in its own container on Kubernetes.

Prefect (real run)

from prefect import flow, task
calls = {"fetch": 0}
@task(retries=2, retry_delay_seconds=5)
def fetch_new_loans():
    calls["fetch"] += 1
    if calls["fetch"] == 1:                       # the 2 a.m. glitch
        raise ConnectionError("bank database timed out")
    return 5000
@task
def build_features(rows): return rows
@task
def train_model(rows): return 0.922
@task
def register_if_better(acc, champion=0.921):
    return "registered new champion" if acc > champion else "kept old champion"
@flow(log_prints=True)
def nightly_training():
    rows = fetch_new_loans()
    feats = build_features(rows)
    acc = train_model(feats)
    print(register_if_better(acc))
if __name__ == "__main__":
    nightly_training()
$ python3 nightly_flow.py
INFO | prefect - Starting temporary server on
       http://127.0.0.1:8505
INFO | Flow run 'devout-mastiff' - Beginning flow run
       'devout-mastiff' for flow 'nightly-training'
INFO | Task run 'fetch_new_loans-65c' - Task run failed
       with exception: ConnectionError('bank database
       timed out') - Retry 1/2 will start 5 second(s)
       from now
INFO | Task run 'fetch_new_loans-65c' - Finished in
       state Completed()
INFO | Task run 'build_features-cf7' - Completed()
INFO | Task run 'train_model-01d' - Completed()
INFO | Task run 'register_if_better-7a6' - Completed()
INFO | Flow run 'devout-mastiff' - registered new champion
INFO | Flow run 'devout-mastiff' - Finished in state
       Completed()

(Prefect's log, trimmed and wrapped; the run name devout-mastiff is random.) retries=2, retry_delay_seconds=5 made Prefect retry the failed fetch after 5 seconds.

Airflow (real run)

from datetime import datetime
from airflow.sdk import dag, task

@dag(schedule="0 2 * * *", start_date=datetime(2026, 11, 1), catchup=False)
def nightly_training():
    @task(retries=2)
    def fetch_new_loans(): return 5000
    @task
    def build_features(rows): return rows
    @task
    def train_model(rows): return 0.922
    @task
    def register_if_better(acc): return "registered" if acc > 0.921 else "kept champion"
    register_if_better(train_model(build_features(fetch_new_loans())))

nightly_training()
$ airflow dags test nightly_training 2026-11-13
setting next dagrun info  next_dagrun='2026-11-13 02:00:00'
Created dag run.
[DAG TEST] starting task_id=fetch_new_loans
  Done. Returned value was: 5000
[DAG TEST] starting task_id=build_features
  Done. Returned value was: 5000
[DAG TEST] starting task_id=train_model
  Done. Returned value was: 0.922
[DAG TEST] starting task_id=register_if_better
  Done. Returned value was: registered
Marking run <DagRun nightly_training ...> successful
DagRun Finished: dag_id=nightly_training

(Trimmed from Airflow 3.4.0's log.) airflow dags test runs the DAG once, immediately, without the scheduler. Each task's return value is passed to the next one (Airflow calls this XCom).

Scheduled retraining

A cron expression has five fields: minute, hour, day of month, month, day of week. * means "every".

Expression Meaning
0 2 * * * 02:00 every day
30 6 * * 1 06:30 every Monday
0 */6 * * * every 6 hours, on the hour
  • Scheduled retraining: retrain on fresh data at fixed times (nightly, weekly).
  • Triggered retraining: retrain when something happens: a lot of new data, or accuracy dropping (Episode 82).
  • Never ship automatically without checks: evaluate, compare with the champion, then move the alias.

Training-serving skew

Real-life example: the head cook tests dal with "1 cup of water"; the night cook reads "1 glass". Same dish name, watery dal.

Training-serving skew (Google): "a difference between performance during training and performance during serving". Causes: training and serving pipelines handle data differently; the data changes between training and serving; feedback loops. There is no error message: predictions just get worse.

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78)            # made-up loan data: income (thousand ₹/month), loan (lakh ₹), score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
# TRAINING pipeline (data team, a notebook): loan as years of income
def train_features(income, loan, score): return [loan * 100 / (income * 12), score]
model = LogisticRegression(max_iter=1000).fit([train_features(*r) for r in X], y)
# SERVING pipeline (app team, rewritten in the API): loan as MONTHS of income
def serve_features(income, loan, score): return [loan * 100 / income, score]
asha = (50, 7, 690)
print("training-time features :", np.round(train_features(*asha), 2), "-> risk",
      round(model.predict_proba([train_features(*asha)])[0, 1], 3))
print("serving-time features  :", np.round(serve_features(*asha), 2), "-> risk",
      round(model.predict_proba([serve_features(*asha)])[0, 1], 3))
training-time features : [  1.17 690.  ] -> risk 0.756
serving-time features  : [ 14. 690.] -> risk 1.0

The same feature name, computed as years of income in training (1.17) and months of income in serving (14). The fix: one definition used by both pipelines.

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78)            # made-up loan data: income (thousand ₹/month), loan (lakh ₹), score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
# TRAINING pipeline (data team, a notebook): loan as years of income
def train_features(income, loan, score): return [loan * 100 / (income * 12), score]
model = LogisticRegression(max_iter=1000).fit([train_features(*r) for r in X], y)
# SERVING pipeline: no rewrite. Both read the SAME feature definition (the feature-store idea)
serve_features = train_features
asha = (50, 7, 690)
print("training-time features :", np.round(train_features(*asha), 2), "-> risk",
      round(model.predict_proba([train_features(*asha)])[0, 1], 3))
print("serving-time features  :", np.round(serve_features(*asha), 2), "-> risk",
      round(model.predict_proba([serve_features(*asha)])[0, 1], 3))
training-time features : [  1.17 690.  ] -> risk 0.756
serving-time features  : [  1.17 690.  ] -> risk 0.756

Feature stores

  • Feature store: one place where features are defined, computed and stored for every team and model.
  • Offline store: the history of feature values, used to build training data.
  • Online store: the latest feature values, returned to the API in milliseconds.
  • Point-in-time correctness: each training row only sees feature values that were known at that moment, which prevents data leakage (Episode 15).
  • Feast: "an open-source feature store that helps teams operate production ML systems at scale".
  • Google's Rule #29: log the features actually used at serving time and train on them, to keep training and serving consistent.

Online learning

  • Batch (offline) learning: train on all the data at once, then freeze the model until the next retraining.
  • Online learning: update the model a little with each new batch of labelled data. In scikit-learn: partial_fit(X, y) on models such as SGDClassifier (the first call needs classes=[0, 1]).
  • Risk: bad or noisy data goes straight into the live model, so validate data, monitor, and keep a rollback.
import numpy as np
from sklearn.linear_model import SGDClassifier
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(81)
def day_of_loans(day, n=400):                # made-up; from day 4 interest rates jump: big loans default more
    X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1), rng.normal(700, 60, n).clip(300, 900)]
    cut = 20 if day < 4 else 0                # the world changes on day 4
    y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, n) > cut).astype(int)
    return X, y
scaler = StandardScaler().fit(day_of_loans(0)[0])
frozen = SGDClassifier(loss="log_loss", random_state=81)        # trained once, never updated
online = SGDClassifier(loss="log_loss", random_state=81)        # learns a little every day
X0, y0 = day_of_loans(0)
frozen.fit(scaler.transform(X0), y0); online.partial_fit(scaler.transform(X0), y0, classes=[0, 1])
for day in range(1, 8):
    X, y = day_of_loans(day); Xs = scaler.transform(X)
    print(f"day {day}: frozen {frozen.score(Xs, y):.2f}   online {online.score(Xs, y):.2f}")
    online.partial_fit(Xs, y)                                    # then learn from today's answers
day 1: frozen 0.93   online 0.90
day 2: frozen 0.93   online 0.86
day 3: frozen 0.94   online 0.90
day 4: frozen 0.72   online 0.81
day 5: frozen 0.78   online 0.94
day 6: frozen 0.72   online 0.94
day 7: frozen 0.73   online 0.92

Before day 4 the frozen model is a little better (online updates are noisier). On day 4 the rule behind the data changes: the frozen model falls to about 0.72 and stays there; the online model is back above 0.9 by day 5.

Common mistakes

Mistake What you see (real) Why Fix
One big script under cron ConnectionError: bank database timed out at 02:00, whole night lost no tasks, no retries, no alerts a DAG in an orchestrator
Order by line position features built from old data, no warning dependencies not written down declare what each task needs
A loop in the pipeline graphlib.CycleError: ('nodes are in a cycle', ...) no task can run first repeat with a schedule
Feature code rewritten for serving risk 1.0 instead of 0.756 training-serving skew one shared definition / feature store
Auto-shipping every retrained model a worse model goes live no comparison evaluate vs champion, then move alias
Online learning, unwatched bad data poisons the model updates go straight live validate, monitor, keep rollback

Mini project: a nightly loan pipeline

  1. Write five Prefect tasks: fetch_new_loans, build_features (using one shared features.py), train_model, evaluate (against the current champion's accuracy) and register_if_better (MLflow, Episode 80).
  2. Give fetch_new_loans retries=3.
  3. Add an inference flow that loads models:/loan-risk@champion and scores every customer, using the same features.py.
  4. Schedule the training flow for 0 2 * * * and the scoring flow for 30 3 * * *.

Practice

1. Change the DAG so that score needs both register and features. Is the order still valid?

Solution

Write "score": {"register", "features"}. The order is still valid (features already runs before register); TopologicalSorter guarantees every task comes after all the tasks it needs.

2. Write the cron expression for every Monday at 6:30 a.m.

Solution

30 6 * * 1 (minute 30, hour 6, any day of month, any month, Monday = 1).

3. Make the Prefect fetch task fail twice with retries=2. What happens on the third call?

Solution

Prefect logs "Retry 1/2" and "Retry 2/2"; the third attempt (the second retry) succeeds and the flow completes. With three failures the task would end in a Failed state and the flow would fail.

4. Your offline accuracy is 0.93 but live accuracy is 0.61; the API computes "age of account" in days, training used months. Name and fix it.

Solution

Training-serving skew. Put the feature computation in one shared module (or a feature store) used by both the training and the serving pipelines.

Quick quiz

Offline accuracy 0.93, live 0.61, and the same feature is computed in days in the API but in months in training. What is it?

Answer

Training-serving skew. Define the feature once and use it in both pipelines.

Interview questions

  1. What is a DAG and why must it be acyclic? A graph of tasks connected by directed dependency arrows with no loops; without loops there is always a valid execution order.
  2. What does an orchestrator give you that cron doesn't? Dependencies between tasks, per-task retries, alerts, run history and logs, backfills, and a UI.
  3. Airflow vs Prefect vs Kubeflow Pipelines? Airflow: mature, DAG-and-scheduler centred. Prefect: Python-first decorators on ordinary functions. Kubeflow Pipelines: each step in a container on Kubernetes, aimed at ML workloads.
  4. What is training-serving skew? Give two causes. Worse performance in serving than in training; caused by different feature code in the two pipelines, or data that changed between training and serving.
  5. What problems does a feature store solve? One feature definition for training and serving (no skew), an offline store for historical training data, an online store for low-latency serving, and point-in-time correct joins (no leakage).
  6. Scheduled retraining vs online learning? Scheduled: retrain on a fixed timetable, then evaluate and ship; stable and easy to audit. Online: update continuously with partial_fit; adapts quickly but noisier and riskier.

Glossary

Term Meaning, with an example
MLOps Practices that keep ML systems running reliably in production.
cron A Linux tool that runs commands on a schedule: 0 2 * * * = 2 a.m. daily.
Cron expression Five fields: minute, hour, day of month, month, day of week.
ML pipeline The chain of tasks from raw data to predictions.
Feature pipeline Raw data → clean features.
Training pipeline Features → trained, evaluated, registered model.
Inference pipeline New inputs → predictions, batch or online.
Task One step of a pipeline: fetch_new_loans.
Dependency "B needs A's output first": train needs features.
Graph Dots (tasks) joined by lines (dependencies).
Directed Each line is an arrow with a direction.
Acyclic No loops.
DAG Directed Acyclic Graph: tasks + arrows, no loops.
Topological order An order where every task comes after all the tasks it needs.
graphlib Python's built-in module for topological sorting; raises CycleError on loops.
Orchestration Running a DAG in order, on a schedule, with retries, alerts and history.
Orchestrator The tool that does it: Airflow, Prefect, Kubeflow Pipelines.
Retry Running a failed task again automatically (retries=2).
Apache Airflow An orchestrator with Python DAGs, a scheduler and a web UI.
Prefect A Python orchestrator using @flow and @task decorators.
Flow (Prefect) The pipeline function, marked @flow.
Kubeflow Pipelines An ML workflow platform where each step runs in a container on Kubernetes.
Scheduled retraining Retraining at fixed times.
Triggered retraining Retraining when a condition is met (new data, accuracy drop).
Training-serving skew Worse performance in serving than in training, e.g. a feature computed differently.
Feature store One place to define, compute and serve features for training and serving.
Offline store Historical feature values for training.
Online store Latest feature values for low-latency serving.
Point-in-time correctness Training rows see only feature values known at that time.
Feast An open-source feature store.
Batch (offline) learning Train once on all data, then freeze.
Online learning Update the model continuously with new data (partial_fit).
partial_fit scikit-learn method that updates a model with one more batch.
SGDClassifier A linear classifier trained by stochastic gradient descent (Episode 18), supports partial_fit.

Summary

  • One big cron script ran steps in the wrong order and lost the whole night to one glitch.
  • Pipelines split work into tasks; a DAG records what needs what; an orchestrator runs it with order, retries, schedules and alerts.
  • We built one with graphlib (and saw CycleError for loops), then ran real Prefect and Airflow DAGs.
  • Training-serving skew turned a 0.756 risk into 1.0; one shared feature definition (a feature store) fixed it.
  • Retrain on a schedule or learn online; when the world changed on day 4, the frozen model fell to 0.72.
  • Next: Episode 82, Monitoring and drift.