Machine Learning Full Course · Episode 78. Every code sample below was run with Python 3.13, scikit-learn 1.9.1, PyTorch 2.15.0, FastAPI 0.162.5, Pydantic 2.13.5, skl2onnx 1.20.0 and onnxruntime 1.29.0 on a 2-core CPU, and the output shown is the real output. The loan data is made up: random numbers (NumPy seed 78) with a hidden rule, so that every program can run anywhere.
The big idea: a trained model is an object in memory (RAM) and disappears when the Python process ends. Serialisation turns it into bytes in a file (pickle, joblib), and deserialisation (loading) rebuilds it. Pickle files can run code while loading, so only load files you trust. ONNX stores the model as a graph of standard maths operators that ONNX Runtime can run from many languages, without Python. For PyTorch, TorchScript is now deprecated in favour of torch.export. A model artifact is the file plus a label (a model card). To let apps use it, we put it behind a REST API built with FastAPI, with Pydantic checking every request. We measure inference latency with p50 and p95, load the model once, and choose batch or online inference.
Sources: Python documentation, pickle ("The pickle module is not secure. Only unpickle data you trust. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling."). scikit-learn, Model persistence (joblib is more efficient than pickle for large NumPy arrays and supports memory mapping; pickle, joblib and cloudpickle can run arbitrary code when loading; ONNX needs no Python and is the most secure for serving; loading a model saved with a different scikit-learn version is unsupported and raises InconsistentVersionWarning; save metadata such as data, code, dependency versions and scores with the model; skops.io as a safer format). ONNX ("an open format built to represent machine learning models", a common set of operators and a common file format). PyTorch documentation, TorchScript ("TorchScript is deprecated, please use torch.export instead"). FastAPI ("a modern, fast (high-performance), web framework for building APIs with Python based on standard Python type hints"; automatic interactive docs at /docs; automatic, clear errors when data is invalid).
Words you need first
- Model: the learned function, here a
LogisticRegression(Episode 22) or aRandomForestClassifier(Episode 31), with its learned numbers (weights). - Process: one running program. When it ends, everything it kept in memory (RAM) is gone.
- File / bytes: what can survive on disk after the process ends.
- Probability of default: the model's estimate that a person will fail to repay a loan (0 to 1).
- Pipeline (Episode 17): preprocessing steps and a model chained into one object.
The story: the model that vanished
A bank wants your loan-risk model inside its mobile app: income (thousand rupees a month), loan amount (lakh rupees) and credit score in, risk of default out, for every applicant, all day.
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
model = LogisticRegression(max_iter=1000).fit(X, y)
print("trained! accuracy:", round(model.score(X, y), 3))
print("Asha (income 50k, loan 7 lakh, score 690) default risk:",
round(model.predict_proba([[50, 7, 690]])[0, 1], 3))
trained! accuracy: 0.932
Asha (income 50k, loan 7 lakh, score 690) default risk: 0.702
The next morning a new Python process starts (a server restarts every day) and the app asks again:
# Next morning. A new Python process starts, and the app asks for a prediction.
print(model.predict_proba([[50, 7, 690]]))
Traceback (most recent call last):
File "main.py", line 2, in <module>
print(model.predict_proba([[50, 7, 690]]))
^^^^^
NameError: name 'model' is not defined
Why it failed: model lived only in the memory of yesterday's process. Retraining on every start costs one second here, but our tiny GPT took 5 minutes (Episode 75) and real models take days.
Real-life example: cook once, pack a tiffin
| In the kitchen | With a model |
|---|---|
| Cooking the dal: slow, needs the whole kitchen | Training: slow, needs data and compute |
| Packing it into a tiffin box | Serialisation: object in memory → bytes in a file |
| Opening it at lunch, anywhere | Deserialisation (loading): bytes → the same object |
Serialisation with pickle
pickle is Python's built-in serialisation module. pickle.dump(obj, f) writes, pickle.load(f) reads. Files are opened in "wb" / "rb" mode: write and read bytes, not text.
import numpy as np, pickle, os
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
model = LogisticRegression(max_iter=1000).fit(X, y)
with open("loan_model.pkl", "wb") as f: # serialise: object -> bytes on disk
pickle.dump(model, f)
print("file size:", os.path.getsize("loan_model.pkl"), "bytes")
print("first bytes:", open("loan_model.pkl", "rb").read(40))
del model # pretend the process ended
with open("loan_model.pkl", "rb") as f: # deserialise: bytes -> object again
loaded = pickle.load(f)
print(type(loaded).__name__, "weights:", loaded.coef_.round(3))
print("Asha's risk from the loaded model:", round(loaded.predict_proba([[50, 7, 690]])[0, 1], 3))
file size: 729 bytes
first bytes: b'\x80\x04\x95\xce\x02\x00\x00\x00\x00\x00\x00\x8c\x1esklearn.linear_model._logis'
LogisticRegression weights: [[-0.12 1.849 -0.024]]
Asha's risk from the loaded model: 0.702
The whole model is 729 bytes (three weights and a bias plus bookkeeping). Notice the first bytes: the file stores where to find the class (sklearn.linear_model._logistic...), not the class's code. That detail causes two problems below.
Is loading safe? Why pickle can run code
A pickle is a list of instructions for rebuilding an object, and pickle.load follows them. An object can define __reduce__ to say "to rebuild me, call this function with these arguments". Here is a harmless demonstration:
import pickle
class Surprise: # what an attacker could put in a .pkl file
def __reduce__(self): # tells pickle "call THIS function when loading"
return (print, ("!!! This code ran just because you LOADED the file !!!",))
evil_bytes = pickle.dumps(Surprise())
print("downloaded 'model.pkl':", len(evil_bytes), "bytes. Loading it...")
model = pickle.loads(evil_bytes) # an attacker would call os.system here instead
print("pickle.load returned:", model)
downloaded 'model.pkl': 94 bytes. Loading it...
!!! This code ran just because you LOADED the file !!!
pickle.load returned: None
The message printed during loading; no method was called. An attacker would put os.system there instead of print. The Python docs warn: "The pickle module is not secure. Only unpickle data you trust." For files from strangers, scikit-learn's docs suggest the safer skops.io format, or ONNX for serving.
joblib: pickle for big NumPy arrays
joblib.dump(model, "model.joblib") and joblib.load("model.joblib"). scikit-learn's docs: more efficient than pickle for large NumPy arrays, supports memory mapping (mmap_mode="r", several processes share one copy in memory) and compression. It is built on pickle, so the same trust rule applies.
import numpy as np, pickle, joblib, os, time
from sklearn.ensemble import RandomForestClassifier
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
forest = RandomForestClassifier(n_estimators=200, random_state=78).fit(X, y)
t = time.perf_counter(); pickle.dump(forest, open("forest.pkl", "wb")); tp = time.perf_counter() - t
t = time.perf_counter(); joblib.dump(forest, "forest.joblib"); tj = time.perf_counter() - t
joblib.dump(forest, "forest_small.joblib", compress=3)
for f in ["forest.pkl", "forest.joblib", "forest_small.joblib"]:
print(f"{f:20} {os.path.getsize(f) / 1e6:6.1f} MB")
print(f"save time: pickle {tp:.2f} s, joblib {tj:.2f} s")
again = joblib.load("forest_small.joblib")
print("same predictions after loading:", (again.predict(X) == forest.predict(X)).all())
forest.pkl 11.2 MB
forest.joblib 11.3 MB
forest_small.joblib 2.4 MB
save time: pickle 0.02 s, joblib 0.04 s
same predictions after loading: True
Compression level 3 shrinks 11.2 MB to 2.4 MB. Save times are tiny on this forest (and vary from run to run); measure on your own model.
Mistake: the custom class is missing at load time
import numpy as np, pickle
from sklearn.base import BaseEstimator, TransformerMixin
class EmiFeature(BaseEstimator, TransformerMixin): # our own custom step
def fit(self, X, y=None): return self
def transform(self, X): return np.c_[X, X[:, 1] / X[:, 0]] # loan / income
data = pickle.dumps(EmiFeature())
print("saved", len(data), "bytes")
del EmiFeature # the serving script never defined or imported the class
pickle.loads(data)
saved 44 bytes
Traceback (most recent call last):
File "main.py", line 9, in <module>
pickle.loads(data)
~~~~~~~~~~~~^^^^^^
AttributeError: Can't get attribute 'EmiFeature' on <module '__main__' from 'main.py'>
The pickle stored only the class name and location (__main__.EmiFeature). The loading script must be able to import the same class. Fix: put custom code in a module (features.py) imported by both the training and the serving code, and keep library versions the same: scikit-learn says loading a model saved with another version is unsupported and warns with InconsistentVersionWarning.
Real-life example: a PDF for models
| Sending a document | Sending a model |
|---|---|
| A Word file with special fonts looks broken on another computer | A pickle needs Python and the same libraries |
| Export a PDF: it opens the same on any device | Export ONNX: a standard model file any language can run |
ONNX and ONNX Runtime
ONNX (Open Neural Network Exchange) is "an open format built to represent machine learning models": the model becomes a graph of standard operators (scale, matrix multiply, add, ...) in a common file format. ONNX Runtime executes that graph from Python, C++, C#, Java, JavaScript and more. Converters: skl2onnx for scikit-learn, torch.onnx.export for PyTorch. The file holds only maths, not Python code, so it is much safer to load.
import numpy as np, os
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from skl2onnx import to_onnx
import onnxruntime as ort
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
pipe = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
onx = to_onnx(pipe, X[:1].astype(np.float32), options={"zipmap": False}) # sklearn pipeline -> ONNX graph
open("loan.onnx", "wb").write(onx.SerializeToString())
print("loan.onnx:", os.path.getsize("loan.onnx"), "bytes; operators:", [n.op_type for n in onx.graph.node])
sess = ort.InferenceSession("loan.onnx", providers=["CPUExecutionProvider"])
asha = np.array([[50, 7, 690]], dtype=np.float32)
label, probs = sess.run(None, {"X": asha})
print("ONNX Runtime :", label, probs.round(4))
print("scikit-learn :", pipe.predict(asha), pipe.predict_proba(asha).round(4))
loan.onnx: 484 bytes; operators: ['Scaler', 'LinearClassifier']
ONNX Runtime : [1] [[0.3087 0.6913]]
scikit-learn : [1] [[0.3087 0.6913]]
options={"zipmap": False} makes the classifier return a plain probability array instead of a list of dictionaries. The ONNX graph has two operators and gives exactly the same answers as scikit-learn.
PyTorch: state_dict, TorchScript and torch.export
torch.save(model.state_dict(), "w.pt")stores only the weights; you still need the class code to rebuild the model (the same problem asEmiFeature).- TorchScript (
torch.jit.script) compiled a model into a program that could run without Python, e.g. from C++. The PyTorch docs now say: "TorchScript is deprecated, please use torch.export instead." - torch.export captures the model's operations as a graph and saves it as a
.pt2file.
import torch, torch.nn as nn, warnings
warnings.simplefilter("once")
torch.manual_seed(78)
net = nn.Sequential(nn.Linear(3, 16), nn.ReLU(), nn.Linear(16, 1), nn.Sigmoid()) # a small PyTorch model
x = torch.tensor([[0.5, 0.7, 0.69]])
with warnings.catch_warnings(record=True) as w:
torch.jit.script(net).save("net_ts.pt") # TorchScript: the model saved as a program
print("TorchScript reloaded :", round(torch.jit.load("net_ts.pt")(x).item(), 6))
for m in w: print(" warning:", m.message)
ep = torch.export.export(net, (x,)) # the replacement: torch.export
torch.export.save(ep, "net.pt2")
print("torch.export reloaded:", round(torch.export.load("net.pt2").module()(x).item(), 6))
print("original model :", round(net(x).item(), 6))
TorchScript reloaded : 0.472512
warning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
warning: `torch.jit.load` is deprecated. Please switch to `torch.export`.
torch.export reloaded: 0.472512
original model : 0.472512
PyTorch 2.15 prints the deprecation warnings itself. For new projects use torch.export or ONNX.
The model artifact: the box and its label
A model artifact is everything needed to use the model later, saved together: the model file plus a label (often called a model card): name, version, training date, metrics, the feature names in order, and library versions.
import numpy as np, joblib, json, sklearn, datetime
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
model = LogisticRegression(max_iter=1000).fit(X, y)
joblib.dump(model, "loan_model.joblib") # the model file
json.dump({"name": "loan-risk", "version": "1.0.0", # the label on the box
"features": ["income", "loan", "credit_score"],
"trained_on": str(datetime.date.today()), "train_accuracy": round(model.score(X, y), 3),
"sklearn": sklearn.__version__}, open("model_card.json", "w"), indent=1)
print(open("model_card.json").read())
{
"name": "loan-risk",
"version": "1.0.0",
"features": [
"income",
"loan",
"credit_score"
],
"trained_on": "2026-11-13",
"train_accuracy": 0.932,
"sklearn": "1.9.1"
}
Sending [loan, income, score] to a model trained on [income, loan, score] gives silently wrong answers, which is why the order belongs on the label.
Real-life example: the restaurant counter
| At a restaurant | Serving a model |
|---|---|
| Customers don't walk into the kitchen | Apps never touch the model file; it stays on a server |
| They order from a menu at the counter | An API endpoint, e.g. POST /predict, with a fixed format |
| A strange order is refused at the counter | Input validation: bad requests get a clear error |
REST API, FastAPI and Pydantic
- API (application programming interface): a fixed way for one program to ask another for something.
- REST API: an API over HTTP, the protocol of the web. Each endpoint is a path plus a method:
GETto read,POSTto send data. - JSON: a text format of names and values,
{"income": 50, "loan": 7}. - Status code:
200OK,422the input failed validation,500the server crashed. - FastAPI: a Python web framework for APIs, based on type hints. It validates input, returns clear errors, and generates interactive docs at
/docs. - Pydantic
BaseModel: a class that declares the fields of a request and their types;Field(gt=0, ge=300, le=900)adds limits (greater than, greater or equal, less or equal).
import numpy as np, sklearn.linear_model as lm
from fastapi import FastAPI, testclient
from pydantic import BaseModel, Field
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
model = lm.LogisticRegression(max_iter=1000).fit(X, y) # real life: joblib.load(...), ONCE at startup
app = FastAPI()
class Applicant(BaseModel): # what a request must look like
income: float = Field(gt=0) # thousand rupees a month
loan: float = Field(gt=0) # lakh rupees
credit_score: int = Field(ge=300, le=900)
@app.post("/predict")
def predict(a: Applicant):
risk = model.predict_proba([[a.income, a.loan, a.credit_score]])[0, 1]
return {"default_risk": round(float(risk), 3), "approve": bool(risk < 0.5)}
client = testclient.TestClient(app) # sends real HTTP requests to our app
for body in [dict(income=50, loan=7, credit_score=690), dict(income="fifty", loan=7, credit_score=1200)]:
r = client.post("/predict", json=body); print(r.status_code, r.json())
200 {'default_risk': 0.702, 'approve': False}
422 {'detail': [{'type': 'float_parsing', 'loc': ['body', 'income'], 'msg': 'Input should be a valid number, unable to parse string as a number', 'input': 'fifty'}, {'type': 'less_than_equal', 'loc': ['body', 'credit_score'], 'msg': 'Input should be less than or equal to 900', 'input': 1200, 'ctx': {'le': 900}}]}
TestClient sends real HTTP requests to the app inside one script. The bad request ("fifty" and a score of 1200) never reached the model.
A real server: uvicorn and curl
app.py:
import joblib
from fastapi import FastAPI
from pydantic import BaseModel, Field
model = joblib.load("loan_model.joblib") # loaded ONCE, when the server starts
app = FastAPI(title="Loan risk API")
class Applicant(BaseModel):
income: float = Field(gt=0) # thousand rupees a month
loan: float = Field(gt=0) # lakh rupees
credit_score: int = Field(ge=300, le=900)
@app.post("/predict")
def predict(a: Applicant):
risk = model.predict_proba([[a.income, a.loan, a.credit_score]])[0, 1]
return {"default_risk": round(float(risk), 3), "approve": bool(risk < 0.5)}
$ python3 train.py # saves loan_model.joblib + model_card.json
$ uvicorn app:app --port 8000
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
$ curl -X POST localhost:8000/predict -H "Content-Type: application/json" \
-d '{"income": 50, "loan": 7, "credit_score": 690}'
{"default_risk":0.702,"approve":false}
$ curl -X POST localhost:8000/predict -H "Content-Type: application/json" \
-d '{"income": 120, "loan": 3, "credit_score": 780}'
{"default_risk":0.0,"approve":true}
server log:
INFO: 127.0.0.1:55800 - "POST /predict HTTP/1.1" 200 OK
INFO: 127.0.0.1:55810 - "POST /predict HTTP/1.1" 200 OK
INFO: 127.0.0.1:55826 - "GET /docs HTTP/1.1" 200 OK
uvicorn is a web server that runs FastAPI apps; curl is a command-line tool that sends HTTP requests. Open http://127.0.0.1:8000/docs in a browser to try the API by clicking.
Latency: how fast does it answer?
- Inference latency: time from a request to its answer, for one prediction.
- p50 (median): half the requests are faster. p95: 95 out of 100 are faster. Users feel the slow ones, so watch p95.
import numpy as np, joblib, time
from sklearn.ensemble import RandomForestClassifier
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
joblib.dump(RandomForestClassifier(100, random_state=78).fit(X, y), "model.joblib")
def handle_slow(row): # mistake: load the model inside every request
return joblib.load("model.joblib").predict_proba([row])[0, 1]
model = joblib.load("model.joblib") # fix: load ONCE when the server starts
def handle(row): return model.predict_proba([row])[0, 1]
def latency(fn, n):
times = []
for row in X[:n]:
t = time.perf_counter(); fn(row); times.append((time.perf_counter() - t) * 1000)
return np.percentile(times, 50), np.percentile(times, 95)
print("load per request : p50 %.1f ms, p95 %.1f ms" % latency(handle_slow, 50))
print("load once : p50 %.1f ms, p95 %.1f ms" % latency(handle, 200))
t = time.perf_counter(); model.predict_proba(X); bt = time.perf_counter() - t
print(f"batch of {len(X)}: {bt * 1000:.0f} ms total = {bt / len(X) * 1000:.3f} ms per person")
load per request : p50 20.9 ms, p95 28.0 ms
load once : p50 3.9 ms, p95 5.9 ms
batch of 5000: 33 ms total = 0.007 ms per person
(A second run of the same program on the same machine, while other work was running. Exact milliseconds change from run to run and machine to machine; the gap does not.) Loading the file on every request makes each answer several times slower. Load once at startup. Scoring all 5,000 people in one call costs far less than a millisecond each.
Batch vs online inference
| Batch inference | Online (real-time) inference | |
|---|---|---|
| What | Score many rows at once, on a schedule | Score one request right now |
| Example | Every night, refresh pre-approved offers for all customers | Asha taps "Apply" and waits for a decision |
| Pros | Cheapest per prediction; no server always running | Fresh answer using the newest input |
| Cons | Answers can be hours old | A server must always run; latency is felt |
| You watch | Throughput (rows per second) | Latency (p50, p95) |
Many companies use both.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Model only in memory | NameError: name 'model' is not defined |
the process ended | joblib.dump / ONNX right after training |
pickle.load on a stranger's file |
code runs while loading | pickle stores instructions | load only trusted files; ONNX or skops.io |
| Custom class not importable | AttributeError: Can't get attribute 'EmiFeature' on <module '__main__' ...> |
pickle stores the class name, not its code | a shared module imported by training and serving |
| Different library version when loading | InconsistentVersionWarning (scikit-learn) |
unsupported across versions | pin versions; save them in the model card |
| New TorchScript code | `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. |
replaced in PyTorch 2 | torch.export or ONNX |
| Loading the model per request | p50 several times higher | disk read + rebuild on every call | load once at startup |
| No input validation | garbage in, garbage out | anyone can send anything | Pydantic fields with limits (422) |
| Wrong feature order | silently wrong predictions | columns mean different things | feature names in the model card; named fields in the API |
Mini project: a complete loan-risk service
Files: train.py (trains, saves loan_model.joblib and model_card.json, shown above) and app.py (loads the model once, validates with Pydantic, serves POST /predict). Run python3 train.py, then uvicorn app:app --port 8000, then call it with curl or from /docs. The real run is shown in the section A real server above.
Practice
1. Save our 200-tree forest with joblib in one script and load it in a different script.
Hint
Script A: train and joblib.dump(forest, "forest.joblib", compress=3). Script B (a new process): forest = joblib.load("forest.joblib") and predict for Asha. If you used a custom class, script B must import it.
2. Convert a random forest to ONNX and check it agrees with scikit-learn on all 5,000 rows.
Solution
import numpy as np, os
from sklearn.ensemble import RandomForestClassifier
from skl2onnx import to_onnx
import onnxruntime as ort
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
forest = RandomForestClassifier(n_estimators=50, random_state=78).fit(X, y)
onx = to_onnx(forest, X[:1].astype(np.float32), options={"zipmap": False})
open("forest.onnx", "wb").write(onx.SerializeToString())
sess = ort.InferenceSession("forest.onnx", providers=["CPUExecutionProvider"])
labels, probs = sess.run(None, {"X": X.astype(np.float32)})
print("forest.onnx:", round(os.path.getsize("forest.onnx") / 1e6, 1), "MB")
print("same labels:", (labels == forest.predict(X)).mean())
print("max probability difference:", np.abs(probs - forest.predict_proba(X)).max().round(6))
forest.onnx: 1.4 MB
same labels: 1.0
max probability difference: 0.0
3 and 4. Add a GET /health endpoint, then measure p50 and p95 of 1,000 API calls.
Solution
import numpy as np, time
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pydantic import BaseModel, Field
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(78) # made-up: income (thousand ₹/month), loan (lakh ₹), credit score
X = np.c_[rng.normal(60, 20, 5000).clip(10), rng.normal(8, 4, 5000).clip(1), rng.normal(700, 60, 5000).clip(300, 900)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 - (X[:, 2] - 700) * 0.08 + rng.normal(0, 6, 5000) > 20).astype(int)
model = LogisticRegression(max_iter=1000).fit(X, y)
app = FastAPI()
class Applicant(BaseModel):
income: float = Field(gt=0); loan: float = Field(gt=0); credit_score: int = Field(ge=300, le=900)
@app.get("/health")
def health(): return {"status": "ok", "model": "loan-risk 1.0.0"}
@app.post("/predict")
def predict(a: Applicant):
risk = model.predict_proba([[a.income, a.loan, a.credit_score]])[0, 1]
return {"default_risk": round(float(risk), 3), "approve": bool(risk < 0.5)}
client = TestClient(app)
print(client.get("/health").json())
times = []
for income, loan, score in X[:1000]:
t = time.perf_counter()
client.post("/predict", json={"income": income, "loan": loan, "credit_score": int(score)})
times.append((time.perf_counter() - t) * 1000)
print(f"1,000 calls: p50 {np.percentile(times, 50):.2f} ms, p95 {np.percentile(times, 95):.2f} ms, max {max(times):.1f} ms")
{'status': 'ok', 'model': 'loan-risk 1.0.0'}
1,000 calls: p50 2.23 ms, p95 7.10 ms, max 14.0 ms
This includes FastAPI's request handling and Pydantic validation. Note how p95 is about three times the median: a few slow calls hide behind a good average.
5. Why is ONNX safer to load than a pickle? Answer
An ONNX file contains only a graph of standard maths operators and numbers; loading it does not run arbitrary Python code. A pickle contains instructions that can call any function while loading.
Quick quiz
Your scikit-learn model must run inside an Android app written in Java, on a phone without Python. How do you ship it? (a) pickle (b) joblib with compress=3 (c) ONNX + ONNX Runtime (d) retrain it on the phone
Answer
(c). pickle and joblib need Python and the same libraries. ONNX is a standard graph that ONNX Runtime runs from Java.
Interview questions
- What is model serialisation, and which formats do you know? Turning a trained model into bytes on disk: pickle, joblib, skops.io, ONNX, PyTorch
state_dict,torch.export(.pt2), formerly TorchScript. - Why is unpickling untrusted files dangerous? Pickle can call arbitrary functions while loading (
__reduce__), so a file can run code on your machine. - When would you choose ONNX? When the serving side needs no Python, another language or hardware runtime, a lean deployment, or safer loading.
- What goes into a model artifact? The model file, version, training data reference and date, metrics, feature names and order, library versions.
- Why load the model at startup, not per request? Loading reads and rebuilds the model; per request it multiplies latency for no benefit.
- p50 vs p95 latency? The median vs the 95th percentile; the tail is what unhappy users experience.
- Batch vs online inference? Scheduled bulk scoring (cheap, possibly stale) vs scoring each request now (fresh, needs an always-on low-latency service).
- What does Pydantic add to an API? Declared types and limits; invalid requests are rejected with a 422 and a clear reason before reaching the model.
Glossary
| Term | Meaning, with an example |
|---|---|
| Model serialisation | Turning a model object in memory into bytes in a file: joblib.dump(model, "m.joblib") |
| Deserialisation (loading) | Turning the bytes back into the same object: joblib.load("m.joblib") |
| RAM / process memory | Where objects live while a program runs; wiped when it ends (NameError next morning) |
| pickle | Python's built-in serialisation module; pickle.dump / pickle.load |
__reduce__ |
A method telling pickle how to rebuild an object; can name any function, which is why loading can run code |
| joblib | A pickle-based library efficient for big NumPy arrays; compression, memory mapping |
| Compression | Storing the same data in fewer bytes: 11.2 MB → 2.4 MB with compress=3 |
| Memory mapping | Reading a file's data directly from disk so several processes share one copy |
| skops.io | A safer scikit-learn format that only loads trusted types |
InconsistentVersionWarning |
scikit-learn's warning when a model is loaded with a different version than it was saved with |
| ONNX | Open Neural Network Exchange: an open file format for models as graphs of standard operators |
| Operator | One standard maths step in a graph, e.g. Scaler, LinearClassifier, MatMul |
| ONNX Runtime | The engine that runs ONNX graphs from many languages |
| skl2onnx | The converter from scikit-learn to ONNX: to_onnx(pipe, X[:1]) |
state_dict |
A PyTorch model's weights as a dictionary; needs the class code to rebuild |
| TorchScript | PyTorch's older way to compile a model into a program (torch.jit.script); deprecated |
| torch.export | PyTorch's current way to capture a model as a graph, saved as .pt2 |
| Model artifact | Everything needed to use the model later: the file plus its label |
| Model card | The label: name, version, date, metrics, features, library versions (model_card.json) |
| API | Application programming interface: a fixed way for programs to ask each other for things |
| REST API | An API over HTTP with endpoints like POST /predict |
| HTTP | The protocol of the web; requests have a method (GET, POST) and a path |
| Endpoint | One address of an API, e.g. POST /predict |
| JSON | A text format of names and values: {"income": 50} |
| Status code | A number saying how a request went: 200 OK, 422 invalid input |
| FastAPI | A Python web framework for building APIs from type hints |
| Pydantic | A library that validates data against declared types: class Applicant(BaseModel) |
Field(gt=0, le=900) |
Pydantic limits: greater than 0, less than or equal to 900 |
| Input validation | Checking requests before using them; bad ones get a clear error |
| uvicorn | A web server that runs FastAPI apps: uvicorn app:app --port 8000 |
| curl | A command-line tool that sends HTTP requests |
| TestClient | FastAPI's tool to send requests to an app inside a Python script |
| Serving | Answering prediction requests from other programs |
| Inference | Using a trained model to make predictions |
| Inference latency | Time from a request to its answer, for one prediction |
| p50 / p95 | 50th / 95th percentile of latency: half / 95% of requests are faster |
| Batch inference | Scoring many rows at once on a schedule (nightly offers) |
| Online (real-time) inference | Scoring one request immediately (Asha taps "Apply") |
| Throughput | How many predictions per second a system can make |
Summary
- A model in memory vanishes when the process ends: serialise it right after training.
- pickle and joblib save almost anything, but loading can run code: only trusted files. Custom classes and library versions must match.
- ONNX turns a model into a portable, safer graph; ONNX Runtime gave the same 0.6913 as scikit-learn. For PyTorch, use torch.export (TorchScript is deprecated).
- A model artifact = file + model card (version, features in order, metrics, library versions).
- FastAPI + Pydantic serve predictions over HTTP and reject bad input with 422; load the model once; watch p50 and p95; choose batch or online inference.
Next: Episode 79, containerising and deploying models: Docker, Dockerfiles, images vs containers, model servers, cloud and serverless inference, CI/CD for ML.