SyntaxPilot
Courses/Machine Learning/Lesson 2
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 2 of 90

Python toolkit for ML

Jupyter notebookNumPy arrayndarray shape and dtypevectorisationbroadcastingpandas Series and DataFramereading CSVindexing and filteringgroupbymatplotlib plotsscikit-learn overviewrandom seed and reproducibility

Machine Learning Full Course · Episode 2. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5, matplotlib 3.11.2 and scikit-learn 1.9.1; the output shown is the real output.

A library is ready-made code someone else wrote, which you import and use. Four libraries do almost all classic ML work in Python, and each one exists to fix a real pain.

The problem: add 18% GST to a price list

Raju's chai stall has prices 120, 80, 45 and 300 rupees and must show them with 18% GST (goods and services tax), i.e. multiply every price by 1.18. Later a supermarket chain wants the same for 5 million prices. In ML, millions of numbers is normal, so the method must be correct and fast.

Python lists fail at maths

prices = [120, 80, 45, 300]
print(prices * 1.18)
TypeError: can't multiply sequence by non-int of type 'float'

Worse, multiplying by an integer gives no error and a wrong result:

prices = [120, 80, 45, 300]
print(prices * 2)
[120, 80, 45, 300, 120, 80, 45, 300]

For a list, * 2 means repeat the list twice, not double each item. A silent wrong answer is more dangerous than a crash.

A loop works, but it is slow on big data:

import time
import numpy as np

n = 5_000_000
prices = list(range(n))
arr = np.arange(n)

start = time.time()
with_gst = []
for p in prices:
    with_gst.append(p * 1.18)
print("python loop:", round(time.time() - start, 2), "seconds")

start = time.time()
with_gst = arr * 1.18
print("numpy      :", round(time.time() - start, 2), "seconds")

One real run:

python loop: 0.46 seconds
numpy      : 0.09 seconds

Your timings will differ (they depend on the computer), but NumPy wins every time. Why the loop is slow: a Python list stores each number as a separate Python object scattered in memory, and the loop processes them one at a time, re-checking each one's type.

NumPy arrays: ndarray, shape and dtype

Real-life example: a bag of loose items (anything, anywhere, handled one by one) is a Python list; an egg tray (same kind of item, fixed slots, packed in rows) is a NumPy array.

  • NumPy (import numpy as np) is Python's library for fast numeric arrays.
  • ndarray ("n-dimensional array") is NumPy's array type: a grid of items that all share one type, stored packed together. It can be 1-D (a row), 2-D (a table) or more.
  • shape = how many items along each direction (axis). (4,) = 1-D with 4 items; (2, 3) = 2 rows × 3 columns.
  • dtype (data type) = the kind of value in every slot. int64 = whole numbers in 64 bits; float64 = decimals in 64 bits.
import numpy as np

prices = np.array([120, 80, 45, 300])
print(prices)
print(prices * 1.18)
print(prices.shape, prices.dtype)

week = np.array([[120, 80, 45],
                 [130, 75, 50]])
print(week.shape, week.dtype)

Output:

[120  80  45 300]
[141.6  94.4  53.1 354. ]
(4,) int64
(2, 3) int64

More array basics:

import numpy as np
print(np.array([1, 2, 3]) + 10)
print(np.array([[1, 2, 3], [4, 5, 6]]).sum(axis=0), np.array([[1, 2, 3], [4, 5, 6]]).sum(axis=1))
print(np.array([1, 2.5, 3]).dtype, np.array([1, 2, 3]).mean())
a = np.arange(6).reshape(2, 3); print(a, a.shape)
[11 12 13]
[5 7 9] [ 6 15]
float64 2.0
[[0 1 2]
 [3 4 5]] (2, 3)
  • axis=0 works down the rows (one result per column); axis=1 works across the columns (one result per row).
  • Mixing 1 and 2.5 makes the whole array float64: one array, one dtype.
  • np.arange(6) makes 0..5; reshape(2, 3) re-arranges the same 6 items into 2 rows × 3 columns.

Vectorisation

Vectorisation = writing one operation on a whole array instead of a Python loop (arr * 1.18). NumPy runs the loop for you in fast compiled code. Real-life example: rinsing the whole egg tray at once instead of egg by egg.

Broadcasting

Real-life example: Raju holds the same price card next to each day's row of the sales register.

Broadcasting = when two arrays have different shapes, NumPy stretches the smaller one to match the bigger one (without copying data), so element-by-element maths still works.

import numpy as np

sales = np.array([[10, 4, 7],     # Mon: chai, samosa, biscuit
                  [12, 6, 5],     # Tue
                  [ 9, 3, 8]])    # Wed
price = np.array([15, 20, 10])    # one price per item

print(sales * price)                # broadcasting
print((sales * price).sum(axis=1))  # money per day

Output:

[[150  80  70]
 [180 120  50]
 [135  60  80]]
[300 350 275]

The rule: compare shapes from the right; each pair of sizes must be equal, or one of them must be 1. (3, 3) with (3,) works. (3, 3) with (2,) doesn't:

ValueError: operands could not be broadcast together with shapes (3,3) (2,)

When you get a shape error, print the .shape of every array involved.

pandas: reading a CSV file into a DataFrame

CSV (comma-separated values) is a plain-text table: one line per row, commas between columns. Every spreadsheet app can save one. Raju's sales.csv:

day,item,cups,price
Mon,chai,120,15
Mon,coffee,40,25
Tue,chai,135,15
Tue,coffee,38,25
Wed,chai,98,15
Wed,coffee,52,25

The by-hand way fails, because everything read from a text file is a string:

total = 0
for line in open("sales.csv").readlines()[1:]:
    day, item, cups, price = line.strip().split(",")
    total = total + cups
print(total)
TypeError: unsupported operand type(s) for +: 'int' and 'str'

Start with total = "" and the "numbers" are glued together as text: 12040135389852.

pandas (import pandas as pd) gives Python a table type:

Term Meaning
DataFrame a whole table: rows plus named columns (like a spreadsheet)
Series one column: values plus a label for each row
index the row labels (0, 1, 2 … by default)
column dtypes each column has its own dtype, so text and numbers live side by side
import pandas as pd

df = pd.read_csv("sales.csv")
print(df)
print(df.shape)
print(df.dtypes)

Output:

   day    item  cups  price
0  Mon    chai   120     15
1  Mon  coffee    40     25
2  Tue    chai   135     15
3  Tue  coffee    38     25
4  Wed    chai    98     15
5  Wed  coffee    52     25
(6, 4)
day        str
item       str
cups     int64
price    int64
dtype: object

read_csv detected that cups and price are numbers (int64) and day, item are text (str).

Indexing and filtering

cups = df["cups"]            # one column = a Series
print(type(cups).__name__, cups.sum())

print(df[df["cups"] > 100])  # filter rows
print(df.loc[2, "item"], df.iloc[0, 2])

Output:

Series 483
   day  item  cups  price
0  Mon  chai   120     15
2  Tue  chai   135     15
chai 120
  • df["cups"] selects a column (a Series).
  • Filtering: df[condition] keeps only rows where the condition is True. Combine conditions with & (and), | (or), with brackets around each: df[(df["item"] == "chai") & (df["day"] != "Mon")].
  • loc selects by labels (row label 2, column name "item"); iloc selects by positions (row 0, column number 2, counting from 0).

groupby

groupby = split the rows into groups by a column's value, compute something inside each group, and combine the results ("split, apply, combine").

df["revenue"] = df["cups"] * df["price"]
print(df.groupby("item")["revenue"].sum())

Output:

item
chai      5295
coffee    3250
Name: revenue, dtype: int64

The new revenue column is also vectorised: cups * price for every row at once.

matplotlib: plotting

matplotlib is Python's standard plotting library; matplotlib.pyplot (imported as plt) draws charts.

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("sales.csv")
chai = df[df["item"] == "chai"]

plt.bar(chai["day"], chai["cups"])
plt.title("Chai sold per day")
plt.xlabel("Day")
plt.ylabel("Cups")
plt.savefig("chai.png")
print("saved chai.png")

Output: saved chai.png, a bar chart with Mon 120, Tue 135, Wed 98. The x-axis is horizontal (days), the y-axis vertical (cups). In a Jupyter notebook, plt.show() (or just running the cell) draws the chart under the cell; in scripts and online compilers, save it with plt.savefig. Plotting first shows patterns, like the Wednesday dip, that a table hides.

Other everyday plots: plt.plot(x, y) (line), plt.scatter(x, y) (points), plt.hist(values) (histogram, covered in episode 13).

scikit-learn: one API for every model

An API (application programming interface) is the set of commands a library offers you. scikit-learn's API is the same for every model:

Command Meaning
model.fit(X, y) train (learn from features X and labels y)
model.predict(X_new) infer (answer for new examples)
transformer.transform(X) change data (e.g. scale it); fit_transform does both
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor

X = [[12], [16], [20], [24], [28], [32]]   # temperature
y = [182, 165, 141, 120, 104, 79]          # cups sold

for model in [LinearRegression(), DecisionTreeRegressor(random_state=0)]:
    model.fit(X, y)
    print(type(model).__name__, model.predict([[30]]))

Output:

LinearRegression [90.74761905]
DecisionTreeRegressor [104.]

A linear regression fits a straight line; a decision tree asks yes/no questions ("is it hotter than 22 °C?"). Different models, different answers, identical code. scikit-learn also holds preprocessing tools, model-selection tools (splitting data, cross-validation, tuning) and metrics; we use them all in this course.

Random seeds and reproducibility

ML uses random numbers to shuffle data, split it and start training. Computers generate pseudo-random numbers from a formula that starts from a value called the seed. Same seed → same sequence. Reproducibility = being able to get exactly the same results again (you, a teammate, or a reviewer).

import numpy as np

rng = np.random.default_rng()
print(rng.integers(1, 100, size=5))

Two real runs: [33 79 16 98 45], then [66 86 38 81 95]: different every time.

import numpy as np

rng = np.random.default_rng(42)
print(rng.integers(1, 100, size=5))

Every run: [ 9 77 65 44 43]. That's why ML code is full of random_state=0 or random_state=42. Any number works; what matters is fixing it. (The exact numbers can change between NumPy versions, so record your library versions too.)

Jupyter notebooks

A Jupyter notebook is a page of cells. You type code in a cell and run it with Shift + Enter; its output (text, tables, charts) appears right under it. Text cells hold notes. Variables stay in memory between cells (the running Python process behind the notebook is called the kernel).

Trap: you can run cells in any order, so a notebook can reach a confusing state. If results look odd, use Restart and Run All.

Project Jupyter grew out of IPython in 2014; its name comes from the languages Julia, Python and R. Everything in this course also runs in the SyntaxPilot compiler.

Where the libraries came from

Library Origin
NumPy created by Travis Oliphant in 2005 by merging the older Numeric and Numarray libraries; first release 2006
pandas development began at AQR Capital Management in 2008; open source by the end of 2009
matplotlib written by John D. Hunter, first released 2003; name from MATLAB + plot + library
scikit-learn started in 2007 as a Google Summer of Code project by David Cournapeau; first public release 1 February 2010, by an INRIA team

Common mistakes (and the real error messages)

Mistake Error Why Fix
[120, 80] * 1.18 TypeError: can't multiply sequence by non-int of type 'float' lists don't do maths np.array([120, 80]) * 1.18
[120, 80] * 2 no error, [120, 80, 120, 80] * repeats a list use an array
(3,3) array * (2,) array ValueError: operands could not be broadcast together with shapes (3,3) (2,) trailing sizes must match or be 1 one value per column: shape (3,)
adding values read from a file TypeError: unsupported operand type(s) for +: 'int' and 'str' file values are strings pd.read_csv detects types
df["Cups"] KeyError: 'Cups' column names are case sensitive df["cups"]; check df.columns
df[df["item"] == "chai" & df["day"] != "Mon"] TypeError: unsupported operand type(s) for &: 'str' and 'StringArray' & binds tighter than == brackets: (df["item"] == "chai") & (df["day"] != "Mon")
no seed different results every run pseudo-random numbers start from a random seed np.random.default_rng(42), random_state=0

Mini project: Raju's best day

import pandas as pd
import matplotlib.pyplot as plt

open("sales.csv", "w").write("""day,item,cups,price
Mon,chai,120,15
Mon,coffee,40,25
Mon,samosa,60,20
Tue,chai,135,15
Tue,coffee,38,25
Tue,samosa,72,20
Wed,chai,98,15
Wed,coffee,52,25
Wed,samosa,45,20
""")

df = pd.read_csv("sales.csv")
df["revenue"] = df["cups"] * df["price"]

per_day = df.groupby("day")["revenue"].sum()
print(per_day)
print("best day:", per_day.idxmax())

plt.bar(per_day.index, per_day.values)
plt.title("Revenue per day")
plt.ylabel("Rs")
plt.savefig("revenue.png")
print("saved revenue.png")

Output:

day
Mon    4000
Tue    4415
Wed    3670
Name: revenue, dtype: int64
best day: Tue
saved revenue.png

idxmax() returns the index label of the largest value.

Practice

  1. What are the shape and dtype of np.array([[1.5, 2], [3, 4], [5, 6]])?
Solution

Shape (3, 2) (3 rows, 2 columns); dtype float64 (one decimal makes the whole array float).

  1. With sales (shape (3, 3)), how do you get total items sold per item (per column)?
Solution

sales.sum(axis=0) → [31 13 20] (down the rows, one total per column).

  1. Show only chai rows from Tuesday onwards, sorted by cups (highest first).
Solution
print(df[(df["item"] == "chai") & (df["day"] != "Mon")])
print(df.sort_values("cups", ascending=False).head(2))

With the 9-row file from the mini project:

   day  item  cups  price
3  Tue  chai   135     15
6  Wed  chai    98     15
   day  item  cups  price
3  Tue  chai   135     15
0  Mon  chai   120     15
  1. Run the seed demo with seed 7 twice. Same numbers?
Solution

Yes: [94 62 68 89 58] both times (NumPy 2.5.3).

Quick quiz

import numpy as np

sales = np.array([[2, 4], [6, 8]])
print(sales.shape, sales * 10)

A) (4,) [20 40 60 80] B) (2, 2) [[20 40] [60 80]] C) the list repeated 10 times D) TypeError

Answer

B. Two rows and two columns → shape (2, 2); an array multiplies every item (vectorisation). Only a list would repeat.

Interview questions

  1. Why is NumPy faster than Python lists? Arrays store one dtype packed contiguously in memory and run operations in compiled code (vectorisation); lists hold separate Python objects processed one at a time by the interpreter.
  2. What is broadcasting? NumPy's rule for element-wise operations on arrays of different shapes: shapes are compared from the right; sizes must be equal or 1, and size-1 (or missing) dimensions are stretched.
  3. DataFrame vs Series? A DataFrame is a 2-D table with named columns; a Series is one column (1-D values plus an index).
  4. loc vs iloc? loc selects by labels; iloc by integer positions.
  5. What does groupby do? Split rows into groups by key, apply an aggregation (sum, mean, count) per group, combine the results.
  6. Why set a random seed? Reproducibility: the same code gives the same splits, initial values and results, so changes can be compared fairly.
  7. What is the scikit-learn estimator API? Every model has fit(X, y) and predict(X); transformers have transform / fit_transform.

Glossary

Term Meaning Example
Library Ready-made code you import and use import numpy as np
GST India's goods and services tax (here 18%) price × 1.18
NumPy Python library for fast numeric arrays np.array([1, 2, 3])
ndarray NumPy's n-dimensional array: one dtype, a shape np.array([[1, 2], [3, 4]])
Shape Number of items along each axis (2, 3) = 2 rows, 3 columns
Axis One direction of an array; axis 0 = down rows, axis 1 = across columns sum(axis=1) = per row
dtype The data type of every item in an array int64, float64
int64 / float64 Whole numbers / decimals stored in 64 bits 120 / 141.6
Vectorisation One operation on a whole array, no Python loop arr * 1.18
Broadcasting Stretching a smaller array to match a bigger one's shape (3, 3) * (3,)
CSV Comma-separated values: a plain-text table sales.csv
String Text value in Python "120" (not the number 120)
pandas Python library for tables import pandas as pd
DataFrame A pandas table: rows + named columns df = pd.read_csv(...)
Series One column of a DataFrame df["cups"]
Index Row labels of a DataFrame/Series 0 … 5
read_csv Reads a CSV file into a DataFrame, detecting types pd.read_csv("sales.csv")
Indexing Selecting columns, rows or cells df["cups"], df.loc[2, "item"]
Filtering Keeping rows that pass a condition df[df["cups"] > 100]
loc / iloc Select by labels / by positions df.loc[2, "item"], df.iloc[0, 2]
groupby Split rows into groups, aggregate each, combine df.groupby("item")["revenue"].sum()
matplotlib / pyplot Plotting library / its charting module (plt) plt.bar(x, y)
x-axis / y-axis Horizontal / vertical axis of a chart days / cups
savefig Saves a chart to an image file plt.savefig("chai.png")
scikit-learn Python library of ML models and tools from sklearn...
API The set of commands a library offers fit, predict, transform
Decision tree A model that predicts by asking yes/no questions about features "hotter than 22 °C?"
Pseudo-random numbers Random-looking numbers from a formula rng.integers(1, 100)
Random seed The starting value of a pseudo-random sequence default_rng(42)
Reproducibility Getting exactly the same results again same seed → same numbers
random_state scikit-learn's seed parameter DecisionTreeRegressor(random_state=0)
Jupyter notebook A page of code and text cells with output under each .ipynb file
Cell One box of code or text in a notebook run with Shift + Enter
Kernel The running Python process behind a notebook restart it to clear state

Summary

  • Python lists can't do element-wise maths (TypeError, or silent repetition) and loops are slow on big data.
  • NumPy arrays have one dtype and a shape, support vectorisation and broadcasting.
  • pandas reads CSV files into DataFrames (tables) made of Series (columns); filter rows, select with loc/iloc, aggregate with groupby.
  • matplotlib draws charts (plt.bar, plt.savefig).
  • scikit-learn gives every model the same fit / predict API.
  • A random seed makes results reproducible; Jupyter notebooks run code cell by cell with output underneath.

Next episode: Maths for ML 1, vectors: scalars, vectors, norms, distance, the dot product and cosine similarity.

Sources for the facts above: NumPy (Wikipedia, "NumPy"); pandas (pandas.pydata.org, "About pandas"); matplotlib (Wikipedia, "Matplotlib"); scikit-learn (scikit-learn.org, "About us", History); Project Jupyter (Wikipedia, "Project Jupyter").