Machine Learning Full Course · Episode 79. Every command and program below was run on a 2-core Linux machine: Python 3.13 on the "laptop", Python 3.12.3 inside the Docker image, scikit-learn 1.9.1, FastAPI 0.150.0, uvicorn 0.54.0, Docker 29.9.3, pytest. The output shown is the real output (long build logs are trimmed, and marked as trimmed). The loan data is made up: random numbers (NumPy seed 78) with a hidden rule.
One honest note: Docker Hub could not be reached from the machine we recorded on, so we built our own base image, ubuntu-python:24.04, from Ubuntu 24.04 packages (it has Python 3.12.3). On your computer you would normally start from an official image such as python:3.12-slim; everything else is the same.
The big idea: a program is its code plus its environment (OS, Python version, every library and its exact version). Copying only the code to another computer breaks it. Docker ships the environment with the code: a Dockerfile is the recipe, docker build makes an image (a sealed, read-only package built in layers), and docker run starts containers (isolated running copies of the image). Inside a container, a server must listen on 0.0.0.0, and the Dockerfile should install libraries before copying code so rebuilds stay fast. At scale, model servers batch requests, the cloud runs your containers (VMs, managed container services, Kubernetes), serverless platforms run them per request but can start cold, and CI/CD tests every change, including the model itself, before it ships.
Sources: Docker docs, What is a container? ("containers are isolated processes for each of your app's components"; "an isolated process with all of the files it needs to run"). Docker docs, What is an image? ("a standardized package that includes all of the files, binaries, libraries, and configurations to run a container"; "Images are immutable"; images are "composed of layers", each layer a set of file system changes). TorchServe (notice "Limited Maintenance": "This project is no longer actively maintained", no planned updates, bug fixes, new features or security patches). BentoML ("a Unified Model Serving Framework for building model inference APIs and multi-model serving systems"). AWS, Lambda quotas (timeout up to 900 seconds = 15 minutes; memory 128 MB to 10,240 MB; container image up to 10 GB uncompressed).
Words you need first
- Deploy: put your program where its real users can reach it (a server, the cloud, a phone).
- Server: a computer (or program) that waits for requests and answers them. Our server answers
POST /predict. - API endpoint: a fixed address a program can call, like
/predict(Episode 78). - Process: one running program. Two copies of the same program are two processes.
- Port: a numbered "door" on a computer for network traffic. Our API listens on port 8000.
- Library / package: code other people wrote that you install, like scikit-learn or FastAPI.
- Version: which release of a library, like scikit-learn
1.9.1. Different versions can behave differently.
The story: it works on my machine
In Episode 78 we built app.py: it loads loan_model.joblib once and answers POST /predict. On our laptop, Asha (income ₹50k a month, a ₹7 lakh loan, credit score 690) gets a default risk of 0.702.
# app.py
import joblib
from fastapi import FastAPI
from pydantic import BaseModel, Field
model = joblib.load("loan_model.joblib") # loaded ONCE, when the server starts
app = FastAPI(title="Loan risk API")
class Applicant(BaseModel):
income: float = Field(gt=0) # thousand rupees a month
loan: float = Field(gt=0) # lakh rupees
credit_score: int = Field(ge=300, le=900)
@app.get("/health")
def health(): return {"status": "ok"}
@app.post("/predict")
def predict(a: Applicant):
risk = model.predict_proba([[a.income, a.loan, a.credit_score]])[0, 1]
return {"default_risk": round(float(risk), 3), "approve": bool(risk < 0.5)}
The bank wants it on their server. The obvious attempt: copy the folder and start it. We copied it into a fresh Python environment, which is what a new server looks like:
$ python3 -m venv server-env
$ server-env/bin/python -c "import app"
Traceback (most recent call last):
File "/opt/demo79/loan-api/app.py", line 1, in <module>
import joblib
ModuleNotFoundError: No module named 'joblib'
Someone then installs "what looks right", and gets an older scikit-learn:
$ server-env/bin/pip install scikit-learn==1.6.1 joblib fastapi
$ server-env/bin/python -c "import app; print(app.model.predict_proba([[50,7,690]])[0,1])"
InconsistentVersionWarning: Trying to unpickle estimator LogisticRegression from version 1.9.1 when using version 1.6.1. This might lead to breaking code or invalid results. Use at your own risk.
Traceback (most recent call last):
AttributeError: 'LogisticRegression' object has no attribute 'multi_class'
Why it failed: the code was fine. The environment was different. The model file was written by scikit-learn 1.9.1 and the server had 1.6.1.
The environment is part of the program
- Environment: everything around your code that it needs to run: the operating system (OS), the Python version, every library and its exact version, and system files.
- Environment parity: the environment where the model runs in production is the same as the one where you built and tested it. Without parity, your tests prove nothing about production.
- Pinning: writing exact versions, so everyone installs the same ones:
# requirements.txt
scikit-learn==1.9.1
joblib==1.6.0
numpy==2.5.3
fastapi==0.150.0
uvicorn==0.54.0
Pinning helps, but the OS and Python itself can still differ. The real fix is to ship the whole environment together with the code.
Real-life example: a sealed ready-meal pack
| Meal company | Docker |
|---|---|
| The recipe card: exact ingredients and steps | Dockerfile: exact steps to build the environment |
| A sealed meal pack: same taste in Delhi or Chennai, never changed after sealing | Image: code + libraries + OS files, read-only |
| A heated plate being served; many plates from the same pack type | Container: a running copy of an image; many from one image |
Docker words, in plain English
- Docker: a tool that builds images and runs containers.
- Image: "a standardized package that includes all of the files, binaries, libraries, and configurations to run a container" (Docker docs). Example:
loan-api:1.0. - Immutable: an image never changes after it is built. A new version is a new image (
loan-api:1.0.1). - Layer: each build step adds one layer of file changes; layers stack to make the image and are reused (cached) when nothing changed.
- Container: "an isolated process with all of the files it needs to run" (Docker docs). Isolated means it has its own files and network, and cannot see other programs on the machine.
- Base image: the image you start from (an OS with Python inside).
- Registry: an online store of images that you
docker pushto anddocker pullfrom. Docker Hub is the best-known public one. - Tag: the name after the colon, usually a version:
loan-api:1.0.
Writing a Dockerfile
# 1. Start from a base image: an OS with Python already inside
FROM ubuntu-python:24.04
# 2. Work inside /app in the image
WORKDIR /app
# 3. Libraries first (this layer is cached until requirements.txt changes)
COPY requirements.txt .
RUN python3 -m venv /venv && \
/venv/bin/pip install --no-cache-dir -r requirements.txt
# 4. Then our code and the model artifact
COPY app.py loan_model.joblib model_card.json ./
# 5. Document the port, and the command a container runs
EXPOSE 8000
CMD ["/venv/bin/uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
| Instruction | What it does |
|---|---|
FROM |
the base image to start from |
WORKDIR |
the folder inside the image where the next steps run (created if missing) |
COPY src dest |
copy files from your folder into the image |
RUN cmd |
run a command while building (install libraries); its file changes become a layer |
EXPOSE 8000 |
documents which port the app listens on (it does not open it by itself) |
CMD [...] |
the command a container runs when it starts |
A virtual environment (python3 -m venv /venv) is a private folder of libraries for one project. A .dockerignore file lists files that should never be copied into the image (caches, notebooks, raw data):
__pycache__/
*.ipynb
data/
Building the image
$ docker build -t loan-api:1.0 .
[1/5] FROM ubuntu-python:24.04
[2/5] WORKDIR /app
[3/5] COPY requirements.txt .
[4/5] RUN python3 -m venv /venv && pip install ...
Successfully installed ... fastapi-0.150.0 joblib-1.6.0 numpy-2.5.3 scikit-learn-1.9.1 ...
DONE 19.6s
[5/5] COPY app.py loan_model.joblib model_card.json ./
naming to docker.io/library/loan-api:1.0 done
real 0m30.936s
(Trimmed. -t gives the image a name and tag; the final . means "use the Dockerfile and files in this folder". We also passed the machine's proxy settings, which are left out here.)
$ docker images loan-api
IMAGE ID DISK USAGE CONTENT SIZE
loan-api:1.0 3e9013bf044f 745MB 174MB
$ docker history loan-api:1.0
CREATED BY SIZE
CMD ["/venv/bin/uvicorn" ... 0B
EXPOSE [8000/tcp] 0B
COPY app.py loan_model.joblib ... 20.5kB
RUN python3 -m venv /venv && ... 312MB
COPY requirements.txt . 12.3kB
WORKDIR /app 8.19kB
Each instruction is a layer. The libraries layer is 312 MB; our code and model are only 20.5 kB.
Running containers
$ docker run -d --name loan1 -p 8000:8000 loan-api:1.0
4a8952ddf3f6...
$ docker run -d --name loan2 -p 8001:8000 loan-api:1.0
059a2f605140...
$ curl -X POST localhost:8000/predict -H "Content-Type: application/json" \
-d '{"income": 50, "loan": 7, "credit_score": 690}'
{"default_risk":0.702,"approve":false}
$ curl localhost:8001/health
{"status":"ok"}
$ docker ps
NAMES IMAGE STATUS PORTS
loan2 loan-api:1.0 Up 4 seconds 0.0.0.0:8001->8000/tcp
loan1 loan-api:1.0 Up 4 seconds 0.0.0.0:8000->8000/tcp
$ docker exec loan1 /venv/bin/python -c "import sklearn,sys; print(sys.version.split()[0], sklearn.__version__)"
3.12.3 1.9.1
-d(detached): run in the background.--name: a friendly name for the container.-p 8001:8000(port mapping): the host's port 8001 leads to port 8000 inside the container. Two containers can both use 8000 inside because each is isolated.docker pslists running containers;docker logs <name>shows a container's output;docker execruns a command inside a running container;docker stop/docker rmstop and delete it.
Inside the container: Python 3.12.3 and scikit-learn 1.9.1, on any computer that runs it. That is environment parity, shipped.
Image vs container
| Image | Container | |
|---|---|---|
| What | a read-only package on disk | a running, isolated process |
| Made by | docker build |
docker run |
| Changes? | never (immutable) | can be stopped, restarted, deleted |
| How many | one loan-api:1.0 |
many from one image |
| Python analogy | a class | an object of that class |
Mistake 1: the server listens only to itself
$ docker run -d --name bad -p 8002:8000 loan-api:1.0 /venv/bin/uvicorn app:app --port 8000
$ curl localhost:8002/health
curl: (56) Recv failure: Connection reset by peer
$ docker logs bad
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
127.0.0.1 (also called localhost) means "this computer". Inside a container, "this computer" is the container itself, so the server ignores traffic arriving through Docker's port mapping. --host 0.0.0.0 means "listen on every network interface", which includes the mapped one.
Mistake 2: layer order and the build cache
Docker reuses (caches) a layer only if that instruction and everything before it are unchanged. We changed one comment in app.py and rebuilt both versions:
| Dockerfile | Order | Rebuild after a one-line code change |
|---|---|---|
Dockerfile.slow |
COPY . . then RUN pip install ... |
30.988 s (pip install ran again: 19.8 s) |
Dockerfile |
COPY requirements.txt, RUN pip install, then COPY app.py ... |
0.658 s (libraries layer CACHED) |
# Dockerfile.slow: copying everything first breaks the cache on every code change
FROM ubuntu-python:24.04
WORKDIR /app
COPY . .
RUN python3 -m venv /venv && /venv/bin/pip install --no-cache-dir -r requirements.txt
EXPOSE 8000
CMD ["/venv/bin/uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Rule: put what changes least at the top (base image, libraries) and what changes most at the bottom (your code).
Model servers and batching
A model server is a program built only to serve models: many models and versions behind one API, request batching, and measurements such as latency. Examples: BentoML ("a Unified Model Serving Framework for building model inference APIs and multi-model serving systems") and TorchServe for PyTorch, which now carries a "Limited Maintenance" notice: "This project is no longer actively maintained." Check that a tool is maintained before you choose it.
Dynamic batching: requests that arrive within a few milliseconds are grouped into one call to the model. Why it helps:
import numpy as np, time
from sklearn.ensemble import RandomForestClassifier
rng = np.random.default_rng(79) # made-up loan data, as in episode 78
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
model = RandomForestClassifier(100, random_state=79).fit(X, y)
requests = X[:256] # 256 applicants arrive within a second
for size in [1, 16, 64, 256]: # how many requests the server groups together
t = time.perf_counter()
for i in range(0, len(requests), size):
model.predict_proba(requests[i:i + size])
s = time.perf_counter() - t
print(f"batch size {size:3}: {s * 1000:6.0f} ms for 256 requests = {s / 256 * 1000:5.2f} ms each")
batch size 1: 979 ms for 256 requests = 3.82 ms each
batch size 16: 62 ms for 256 requests = 0.24 ms each
batch size 64: 17 ms for 256 requests = 0.07 ms each
batch size 256: 6 ms for 256 requests = 0.02 ms each
Every call to the model has a fixed cost; batching pays it once for many requests. The price: a request may wait a few milliseconds for its batch to fill, so model servers let you set the maximum wait and the maximum batch size. (Exact milliseconds depend on the machine and how busy it is: in the video's recording, with the machine also busy recording, the numbers were 1578 ms vs 9 ms. The gap is always large.)
Cloud deployment
- Cloud: renting someone else's computers over the internet, paid by the hour or second.
- Step 1:
docker pushyour image to a registry. Step 2: the cloud pulls it and runs containers. - Virtual machine (VM): a rented computer where you install Docker and run containers yourself.
- Managed container service: you hand over an image, the provider runs it and gives you a web address.
- Kubernetes: a system that runs many containers across many machines, restarts crashed ones and spreads traffic among them.
- Autoscaling: starting more containers when traffic is high and stopping them when it is quiet.
Serverless inference and cold starts
Real-life example: owning a car (pay every month, even when parked) vs booking a cab per ride (pay per ride, but you may wait for it to arrive).
- Serverless: you give the platform a function (or an image) and it runs it for each request. There are still servers; you just never manage them. It scales to zero when idle, and you pay mostly for the time your code runs.
- Cold start: the first request on a fresh machine must start everything and load the model before answering. Later requests on that warm machine are fast.
- Limits (AWS Lambda docs): one call can run at most 900 seconds (15 minutes), with 128 MB to 10,240 MB of memory, and a container image of up to 10 GB. Fine for small models, not for very large ones.
import time, joblib, numpy as np
from sklearn.ensemble import RandomForestClassifier
rng = np.random.default_rng(79) # build a model file to deploy (made-up data)
X = rng.normal(size=(5000, 3)); y = (X.sum(axis=1) > 0).astype(int)
joblib.dump(RandomForestClassifier(300, random_state=79).fit(X, y), "model.joblib")
model = None # nothing loaded: a brand-new, cold machine
def handler(event, context=None): # what a serverless platform calls per request
global model
if model is None: # cold start: first request on this machine
model = joblib.load("model.joblib")
return {"risk": round(float(model.predict_proba([event["features"]])[0, 1]), 3)}
for n in range(1, 5):
t = time.perf_counter()
out = handler({"features": [0.5, -0.2, 0.1]})
print(f"request {n}: {out} took {(time.perf_counter() - t) * 1000:6.1f} ms")
request 1: {'risk': 0.957} took 52.1 ms
request 2: {'risk': 0.957} took 10.4 ms
request 3: {'risk': 0.957} took 10.1 ms
request 4: {'risk': 0.957} took 10.0 ms
Timings vary from run to run (98.5 ms then about 17-38 ms in the video's recording), but request 1 is always the slow one. On a real platform the machine and container must also start, so cold starts are usually longer than this. Fixes: small models, load once outside the per-request code, keep some machines warm, or use an always-on server when every request must be fast.
CI/CD for ML
- CI (continuous integration): every change pushed to the shared code is tested automatically.
- CD (continuous delivery / deployment): changes that pass are packaged and shipped automatically (delivery: ready to release with one click; deployment: released automatically).
- For ML, test the model too: an accuracy gate (block a model below a minimum score on held-out data), known answers (cases whose output you understand), and input validation.
A teammate's "small refactor" swapped two features. No error, no crash:
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78) # same made-up loan data as the API
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
model = LogisticRegression(max_iter=1000).fit(X[:4000], y[:4000])
def to_row(a): return [a["loan"], a["income"], a["credit_score"]] # a teammate's "small refactor"
def test_accuracy_gate(): # the model must stay good enough to ship
acc = model.score(X[4000:], y[4000:]); assert acc >= 0.90, f"accuracy {acc:.3f} < 0.90"
def test_known_applicant(): # a case whose answer we already know
risk = model.predict_proba([to_row({"income": 50, "loan": 7, "credit_score": 690})])[0, 1]
assert 0.6 < risk < 0.8, f"Asha's risk {risk:.3f}, expected about 0.70"
for test in [test_accuracy_gate, test_known_applicant]:
try: test(); print("PASS", test.__name__)
except AssertionError as e: print("FAIL", test.__name__, "->", e)
PASS test_accuracy_gate
FAIL test_known_applicant -> Asha's risk 1.000, expected about 0.70
After fixing to_row to [a["income"], a["loan"], a["credit_score"]] (the order in model_card.json):
PASS test_accuracy_gate
PASS test_known_applicant
The accuracy gate alone would have missed it. Known-answer tests catch silent bugs.
A GitHub Actions workflow
GitHub Actions runs a workflow (a YAML file in .github/workflows/) on every push and pull request. YAML is a simple text format of names and values.
# .github/workflows/ml-ci.yml
name: ml-ci
on: [push, pull_request]
jobs:
test-and-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: "3.12"}
- run: pip install -r requirements.txt pytest httpx
- run: python train.py
- run: pytest -q tests
- run: docker build -t loan-api:${{ github.sha }} .
The tests (tests/test_api.py), using FastAPI's TestClient:
from fastapi.testclient import TestClient
from app import app
client = TestClient(app)
def test_health():
assert client.get("/health").json() == {"status": "ok"}
def test_known_applicant():
r = client.post("/predict", json={"income": 50, "loan": 7, "credit_score": 690})
assert r.status_code == 200 and 0.6 < r.json()["default_risk"] < 0.8
def test_bad_input_rejected():
r = client.post("/predict", json={"income": "fifty", "loan": 7, "credit_score": 1200})
assert r.status_code == 422
The same steps, run on our machine:
$ python3 train.py
saved loan_model.joblib, accuracy 0.932
$ pytest -q tests
... [100%]
3 passed in 1.20s
$ docker build -t loan-api:1.0.2 .
real 0m0.658s
The image tag ${{ github.sha }} is the commit's ID, so every image can be traced back to the exact code that built it.
Common mistakes
| Mistake | What you see (real) | Why | Fix |
|---|---|---|---|
| Copying code to a new server | ModuleNotFoundError: No module named 'joblib' |
different environment | ship an image |
| Unpinned versions | InconsistentVersionWarning, then AttributeError: 'LogisticRegression' object has no attribute 'multi_class' |
model file ≠ library version | requirements.txt with == versions |
| uvicorn on 127.0.0.1 in a container | curl: (56) Recv failure: Connection reset by peer |
listens only to itself | --host 0.0.0.0 |
COPY . . before pip install |
30.988 s rebuild per code change | cache broken early | libraries first, code last |
| Loading the model per request | every request slow | repeated loading | load once, outside the handler |
| No model tests in CI | risk 1.000 instead of 0.702, silently | nothing checks the model | accuracy gate + known answers |
| Choosing an unmaintained server | no security fixes | project in limited maintenance | check the project's status |
Mini project: the loan API, end to end
train.pytrains the model and writesloan_model.joblibandmodel_card.json.app.py(above) with/healthand/predict.requirements.txtwith pinned versions, the Dockerfile above, and a.dockerignore.docker build -t loan-api:1.0 ., thendocker run -d -p 8000:8000 loan-api:1.0andcurlit.tests/test_api.pyand the GitHub Actions workflow.
# train.py
import json, joblib, numpy as np, sklearn
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
model = LogisticRegression(max_iter=1000).fit(X, y)
joblib.dump(model, "loan_model.joblib")
json.dump({"name": "loan-risk", "version": "1.0.0", "features": ["income", "loan", "credit_score"],
"train_accuracy": round(model.score(X, y), 3), "sklearn": sklearn.__version__}, open("model_card.json", "w"), indent=1)
print("saved loan_model.joblib, accuracy", round(model.score(X, y), 3))
Output: saved loan_model.joblib, accuracy 0.932
Practice
1. Start three containers from loan-api:1.0 on ports 8000, 8001 and 8002, and query /health on each.
Solution
docker run -d --name a -p 8000:8000 loan-api:1.0
docker run -d --name b -p 8001:8000 loan-api:1.0
docker run -d --name c -p 8002:8000 loan-api:1.0
curl localhost:8000/health; curl localhost:8001/health; curl localhost:8002/health
docker rm -f a b c
Each prints {"status":"ok"}. One image, three containers; inside, all three use port 8000.
2. Add a test that sends loan = 0 and expects status 422.
Solution
def test_zero_loan_rejected():
r = client.post("/predict", json={"income": 50, "loan": 0, "credit_score": 690})
assert r.status_code == 422 # Field(gt=0): loan must be greater than 0
3. Your container exits immediately and docker logs shows ModuleNotFoundError: No module named 'fastapi'. What is wrong?
Solution
fastapi is missing from requirements.txt, so it was never installed in the image. Add it with a pinned version and rebuild. Your laptop had it installed, which is exactly the "works on my machine" problem.
4. Rewrite the cold-start handler so the model is loaded when the module is imported, not inside the handler. What changes for request 1?
Solution
model = joblib.load("model.joblib") # loaded during the platform's start-up phase
def handler(event, context=None):
return {"risk": round(float(model.predict_proba([event["features"]])[0, 1]), 3)}
Loading now happens once when the module is imported, so request 1 is about as fast as the rest; the start-up itself still takes the same time.
Quick quiz
You ran docker run -d -p 8000:8000 loan-api:1.0 and docker run -d -p 8001:8000 loan-api:1.0. How many images and containers do you have?
Answer
One image (loan-api:1.0) and two containers. Each docker run starts a new container from the same image.
Interview questions
- What is the difference between a Docker image and a container? An image is a read-only, layered package (code, libraries, OS files); a container is a running, isolated process started from an image. Many containers can run from one image.
- Why should a Dockerfile install requirements before copying the code? Docker caches layers until an instruction or anything before it changes. Code changes often and libraries rarely, so libraries-first keeps the expensive install layer cached.
- Your API works on your laptop but not in Docker:
Connection reset by peer. First thing to check? That the server listens on0.0.0.0, not127.0.0.1, and that the port is mapped with-p. - What is a cold start and how do you reduce it? The first request on a fresh machine must start the runtime and load the model. Reduce with smaller models and images, loading once at start-up, and keeping instances warm.
- What does a model server add over a plain web app? Multiple models and versions, dynamic batching, monitoring metrics and standard APIs.
- What should CI test in an ML project beyond unit tests? A model quality gate on held-out data, known-answer cases, input validation, and that the image builds.
- Serverless or an always-on server for a fraud model that must answer in 50 ms all day? Usually always-on (or serverless with warm instances), because cold starts would break the latency target and traffic is steady.
Glossary
| Term | Meaning, with an example |
|---|---|
| Deploy | Put a program where its users can reach it. Our loan API on the bank's server. |
| Environment | Everything around the code it needs: OS, Python version, libraries and versions. |
| Environment parity | Production's environment = the tested one. Python 3.12.3 + scikit-learn 1.9.1 everywhere. |
| "Works on my machine" | Code that runs only where it was written, because the environment differs elsewhere. |
| Pinning | Writing exact versions: scikit-learn==1.9.1 in requirements.txt. |
| requirements.txt | A text file listing the libraries (and versions) to install with pip install -r. |
| Virtual environment (venv) | A private folder of libraries for one project: python3 -m venv /venv. |
| Docker | A tool that builds images and runs containers. |
| Dockerfile | The recipe: step-by-step instructions to build an image (FROM, COPY, RUN, CMD...). |
| Image | A read-only package of files, libraries and settings to run a container. loan-api:1.0. |
| Immutable | Cannot be changed after creation. A new image version is a new image. |
| Layer | One instruction's file changes; layers stack to form an image and are cached. |
| Build cache | Reuse of unchanged layers on rebuild (CACHED), 0.658 s instead of 30.988 s. |
| Base image | The starting image, e.g. an OS with Python (FROM ubuntu-python:24.04). |
| Tag | The version label after the colon: loan-api:1.0. |
| Container | A running, isolated process started from an image. loan1, loan2. |
| Isolated | Has its own files and network; other programs can't see inside. |
| Registry | An online store for images (docker push / docker pull), e.g. Docker Hub. |
| .dockerignore | Files never copied into the image (caches, notebooks, raw data). |
| Port | A numbered network door on a computer. Our API listens on 8000. |
| Port mapping | -p 8001:8000: host port 8001 leads to container port 8000. |
| localhost / 127.0.0.1 | "This computer". Inside a container, the container itself. |
| 0.0.0.0 | "Listen on every network interface", needed inside containers. |
| uvicorn | The web server that runs a FastAPI app. |
| Model server | A program built to serve models: many models, versions, batching, metrics (BentoML, TorchServe). |
| Dynamic batching | Grouping requests that arrive close together into one model call. 979 ms → 6 ms for 256 requests. |
| Cloud | Renting computers over the internet, paid by time. |
| Virtual machine (VM) | A rented computer that behaves like a real one, where you run Docker yourself. |
| Managed container service | A cloud service that runs your image and gives you a web address. |
| Kubernetes | A system that runs many containers across many machines, restarts crashed ones and spreads traffic. |
| Autoscaling | Automatically adding containers when busy and removing them when quiet. |
| Serverless | The platform runs your function or image per request; no servers to manage; scales to zero. |
| Scale to zero | No instances (and no cost for them) when there is no traffic. |
| Cold start | Extra delay on the first request to a fresh instance (start-up + model loading). 52 ms vs 10 ms here. |
| CI (continuous integration) | Every change is tested automatically. |
| CD (continuous delivery/deployment) | Passing changes are packaged and shipped automatically. |
| Accuracy gate | A test that blocks a model scoring below a threshold (here 0.90). |
| Known-answer test | A test on a case whose correct output you know (Asha ≈ 0.70). |
| GitHub Actions | GitHub's CI/CD service, configured by YAML workflow files. |
| YAML | A simple text format of names and values used for configuration. |
| pytest | A Python tool that finds and runs test functions (3 passed). |
Summary
- A program is code plus environment. Copying code alone gave
ModuleNotFoundErrorand, with another scikit-learn version,InconsistentVersionWarningand anAttributeError. - Docker ships the environment: a Dockerfile builds an immutable, layered image; containers are its running, isolated copies.
- Inside containers listen on
0.0.0.0; put libraries before code in the Dockerfile (0.66 s vs 31 s rebuilds). - Model servers batch requests (979 ms → 6 ms for 256 requests); the cloud runs containers on VMs, managed services or Kubernetes.
- Serverless runs per request and scales to zero, but has limits and cold starts.
- CI/CD tests every change, including the model: an accuracy gate and known answers caught a silent feature swap.
- Next: Episode 80, Experiment tracking and model registry.