SyntaxPilot
Courses/Machine Learning/Lesson 6

Machine Learning · Lesson 6 of 90 · 13 min

Maths 4: derivatives

functionsloperate of changederivativederivative rules (power, sum, product)chain ruleminimum and maximumsetting derivative to zeroconvex vs non-convex functionnumerical derivative in Python

Machine Learning Full Course · Episode 6. Every code sample below was run with Python 3.13 in plain Python (no libraries); the output shown is the real output. No calculus background is assumed: only arithmetic and powers like x² and x³.

The problem: the best price for a cup of chai

Each cup costs Raju Rs 5 to make. At Rs 10 a cup, 150 people buy but he keeps only Rs 5 each; at Rs 40 nobody buys. Every extra rupee loses 5 customers. Somewhere in between is the price with the most profit. Machine learning has the same problem upside down: find the settings that give the least error.

Functions

A function is a rule that takes an input and always gives exactly one output, like a sugarcane juice machine (cane in, juice out). Written f(x), "f of x".

  • cups(p) = 200 − 5p
  • profit(p) = (p − 5) × cups(p)
def cups(p):                 # cups sold at price p
    return 200 - 5 * p

def profit(p):               # Rs 5 cost per cup
    return (p - 5) * cups(p)

for p in [10, 20, 30, 40]:
    print(p, cups(p), profit(p))

Output:

10 150 750
20 100 1500
30 50 1250
40 0 0

Profit rises, then falls: the best price is somewhere between 20 and 30.

The obvious way: try every value (and why it fails)

import time

def profit(p):
    return (p - 5) * (200 - 5 * p)

best = max(range(5, 41), key=profit)
print("best price:", best, profit(best))

t = time.perf_counter()
for _ in range(1_000_000):
    profit(22)
per_try = (time.perf_counter() - t) / 1_000_000

tries = 100 ** 10            # 10 knobs, 100 values each
years = tries * per_try / (3600 * 24 * 365)
print(f"{tries:.0e} tries = {years:.0e} years")

Output (one run; the second number depends on your computer):

best price: 22 1530
1e+20 tries = 4e+05 years
  • max(range(5, 41), key=profit) tries every whole price and keeps the one with the biggest profit: Rs 22.
  • A model doesn't have one knob (a number we can tune) but thousands or millions. Each new knob multiplies the combinations by 100, so 10 knobs already need 10²⁰ tries: hundreds of thousands of years. This blow-up is called exponential growth.

Why it fails: blind search never uses the shape of the function. We need to stand at one point and know which way to move.

Slope and rate of change

Idea Real life Maths
Slope a ghat road sign "1 in 10": 1 m up per 10 m along rise / run = 1/10 = 0.1
Rate of change a speedometer: 60 km per hour, right now change in output per unit change of input
Sign walking uphill, downhill, on flat ground + up · − down · 0 flat

For Raju, a positive slope of profit means raising the price raises the profit.

The derivative

A curve's steepness changes from place to place, so measure the slope over a small step h: (f(x + h) − f(x)) / h, and shrink h.

def profit(p):
    return (p - 5) * (200 - 5 * p)

def slope(f, x, h):          # rise over run, over a small step h
    return (f(x + h) - f(x)) / h

for h in [1, 0.1, 0.001]:
    print(h, slope(profit, 10, h))
print("tiny h:", slope(profit, 10, 1e-15))

Output:

1 120.0
0.1 124.49999999999932
0.001 124.99499999989894
tiny h: 227.37367544323203

The values close in on 125. The value the slope approaches as h shrinks towards 0 (its limit) is the derivative: the slope at exactly one point. At Rs 10, profit′ = 125: each extra rupee adds about Rs 125 of profit.

The tiny-h trap: h = 10⁻¹⁵ gives 227, nearly double. Computers store decimals with about 16 significant digits (floating point). f(x + h) − f(x) subtracts two almost equal numbers, leaving only rounding noise, and dividing by a tiny h magnifies it. Too big an h is inaccurate; too small is noise.

Numerical derivative: the central difference

def profit(p):
    return (p - 5) * (200 - 5 * p)

def slope(f, x, h=1e-5):     # central difference
    return (f(x + h) - f(x - h)) / (2 * h)

print(slope(profit, 10))

Output:

125.00000000272847

Stepping to both sides and dividing by 2h (the central difference) is far more accurate; h ≈ 10⁻⁵ is a good default. A derivative computed this way is a numerical derivative. Use it to check every derivative you work out by hand (ML libraries call this gradient checking).

Derivative rules

Notation: f′(x) ("f prime of x") or d/dx f(x) both mean the derivative of f.

Rule Formula Example
Power rule (xⁿ)′ = n·xⁿ⁻¹ (x⁴)′ = 4x³; (x²)′ = 2x; (x)′ = 1
Constant (c)′ = 0 (7)′ = 0
Constant multiple (c·f)′ = c·f′ (7x)′ = 7
Sum rule (f + g)′ = f′ + g′ (x² + 7x)′ = 2x + 7
Product rule (f·g)′ = f′·g + f·g′ (x²(x + 1))′ = 2x(x + 1) + x²
def slope(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

def power(t): return t ** 4
def total(t): return t ** 2 + 7 * t
def prod(t):  return t ** 2 * (t + 1)

x = 3
print("power:  ", round(slope(power, x), 3), 4 * x ** 3)
print("sum:    ", round(slope(total, x), 3), 2 * x + 7)
print("product:", round(slope(prod, x), 3), 2 * x * (x + 1) + x ** 2 * 1)

Output:

power:   108.0 108
sum:     13.0 13
product: 33.0 33

Raju's profit by the product rule: f = p − 5 (f′ = 1), g = 200 − 5p (g′ = −5): profit′ = 1·(200 − 5p) + (p − 5)·(−5) = 200 − 5p − 5p + 25 = −10p + 225. At p = 10: 125, matching the measurement.

The chain rule

A function inside a function: f(x) = (3x + 1)². Think of cycle gears: the pedal gear turns the middle gear 3× as fast (inner function u = 3x + 1, rate du/dx = 3), and the middle gear drives the wheel (outer function u², rate 2u). Pedal-to-wheel rate: multiply the rates.

Chain rule: (f(g(x)))′ = f′(g(x)) · g′(x), "outer rate × inner rate". In Leibniz notation: dy/dx = dy/du · du/dx.

def f(x):
    return (3 * x + 1) ** 2

def slope(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

x = 2
print("power rule only:", 2 * (3 * x + 1))
print("chain rule:", 2 * (3 * x + 1) * 3)
print("numerical:", round(slope(f, x), 4))

Output:

power rule only: 14
chain rule: 42
numerical: 42.0

Forgetting the inner rate is the most common derivative mistake; here it is off by exactly the factor 3. Neural networks are hundreds of functions nested inside each other, and training them (backpropagation, later in the course) is the chain rule applied over and over.

Maximum, minimum and setting the derivative to zero

Walking over a hill: before the top the slope is +, exactly at the top it is 0, after it is −. The highest point is the maximum; the lowest point of a valley is the minimum. At both, the derivative is 0 (a stationary point or flat spot). To find them: set the derivative to zero and solve.

For Raju: −10p + 225 = 0 ⇒ p = 22.5.

def profit(p):
    return (p - 5) * (200 - 5 * p)

def slope(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

def d_profit(p):             # from the rules: -10p + 225
    return -10 * p + 225

for p in [10, 22.5, 30]:
    print(p, round(slope(profit, p), 4), d_profit(p))
print("best:", 225 / 10, profit(22.5))

Output:

10 125.0 125
22.5 0.0 0.0
30 -75.0 -75
best: 22.5 1531.25

+125 (go up), 0 (the top), −75 (too far). Best price Rs 22.50, profit Rs 1,531.25, slightly better than brute force's whole-rupee answer (Rs 22, 1,530), found with one equation instead of a loop.

How to tell a maximum from a minimum: the slope goes + → 0 → − at a maximum and − → 0 → + at a minimum (the second derivative, the slope of the slope, is negative at a maximum and positive at a minimum; more in episode 7).

Convex vs non-convex functions

Convex Non-convex
Picture a round bowl: a marble always rolls to the one bottom a mountain range: the marble stops in the nearest valley
Formal test a straight line between any two points of the curve never goes below the curve some such line dips below
Flat spots one, and it is the best (the global minimum) several: local minima, maxima
In ML linear and logistic regression losses neural network losses

A local minimum is the lowest point only in its neighbourhood; the global minimum is the lowest of all.

def bowl(x):
    return (x - 3) ** 2 + 1

def hills(x):
    return x ** 4 - 4 * x ** 2 + x

def slope(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

for name, f in [("bowl", bowl), ("hills", hills)]:
    xs = [i / 100 + 0.005 for i in range(-300, 500)]
    flat = [round(x, 1) for x in xs if slope(f, x) * slope(f, x + 0.01) < 0]
    print(name, "slope = 0 near:", [(x, round(f(x), 2)) for x in flat])

Output:

bowl slope = 0 near: [(3.0, 1.0)]
hills slope = 0 near: [(-1.5, -5.44), (0.1, 0.06), (1.3, -2.6)]

The loop walks along x from −3 to 5 in steps of 0.01 and records where the slope changes sign (product of neighbouring slopes < 0). The bowl has one flat spot (its minimum, 1.0 at x = 3). The hills have three: the global minimum (−5.44 at x ≈ −1.5), a local maximum (x ≈ 0.1) and a local minimum (−2.6 at x ≈ 1.3) that is not the lowest. Rolling downhill from x = 1 gets stuck at 1.3.

Common mistakes

Mistake Result Why Fix
try every value of every knob 1e+20 tries = 100,000+ years combinations multiply with each knob follow the derivative
slope(profit, 10, 1e-15) 227.37367544323203 instead of 125 floating-point rounding noise ÷ tiny h central difference, h ≈ 1e-5
((3x + 1)²)′ = 2(3x + 1) 14 instead of 42 at x = 2 forgot the inner rate chain rule: × 3
(x²)′ = x wrong slope everywhere power rule: bring the power down AND subtract 1 (x²)′ = 2x
(f·g)′ = f′·g′ wrong for products the product rule has two terms f′g + fg′
slope 0 ⇒ the best point stuck at x = 1.3 (−2.6) non-convex: local minimum or even a peak check the shape; compare candidates

Mini project: let the slope find the price

Instead of solving the equation, start anywhere and keep stepping a little in the uphill direction:

def profit(p):
    return (p - 5) * (200 - 5 * p)

def slope(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

p = 10.0                       # start anywhere
for step in range(1, 31):
    p = p + 0.05 * slope(profit, p)   # move uphill, a little
    if step % 5 == 0:
        print(step, round(p, 4), round(profit(p), 2))

Output:

5 22.1094 1530.49
10 22.4878 1531.25
15 22.4996 1531.25
20 22.5 1531.25
25 22.5 1531.25
30 22.5 1531.25

Big slope, big step; as the slope flattens near the top, the steps shrink. Flip the sign (step downhill) and minimise an error instead, and you have gradient descent, which we build properly in episode 8. Try a step size of 0.3 instead of 0.05: each step overshoots further (47.5, −27.5, 122.5, …) and it blows up; the step size matters (the learning rate, episode 8).

Practice

  1. Derivative of 4x³ − 2x + 9; check it numerically at x = 2.
Solution

Power, constant-multiple, sum and constant rules: 12x² − 2. At x = 2: 46. round(slope(lambda x: 4*x**3 - 2*x + 9, 2), 3) → 46.0. (lambda x: ... is Python's way to write a one-line function without def.)

  1. Raju's cost rises to Rs 7 a cup. New best price?
Solution

profit = (p − 7)(200 − 5p); profit′ = (200 − 5p) − 5(p − 7) = −10p + 235 = 0 ⇒ p = 23.5, profit = Rs 1,361.25. The numerical slope at 23.5 prints 0.0.

  1. Derivative of (2x − 5)³ by the chain rule; check at x = 4.
Solution

Outer u³ → 3u², inner 2x − 5 → 2: 3(2x − 5)²·2 = 6(2x − 5)². At x = 4: 6 × 9 = 54; numerically 54.0.

  1. Find the flat spots of x³ − 3x.
Solution

3x² − 3 = 0 ⇒ x = ±1. Numerical slope at both is 0.0. x = −1 is a local maximum (f = 2), x = 1 a local minimum (f = −2). The function isn't convex.

Quick quiz

def f(x):
    return x ** 3 - 12 * x

def df(x):
    return 3 * x ** 2 - 12

print(df(2), df(3))

A) 0 15 B) -4 9 C) 0 -3 D) 12 15

Answer

A) 0 15. df(2) = 12 − 12 = 0, a flat spot (here a local minimum); df(3) = 27 − 12 = 15, going up.

Interview questions

  1. What is a derivative, intuitively? The instantaneous rate of change: the slope of the function at one point.
  2. Why does ML need derivatives? Training minimises a loss; the derivative (gradient) says which direction lowers the loss and by how much, without blind search.
  3. State the chain rule and why it matters for neural networks. (f(g(x)))′ = f′(g(x))·g′(x); backpropagation applies it layer by layer to get every weight's gradient.
  4. Forward vs central difference? Forward (f(x+h) − f(x))/h has error proportional to h; central (f(x+h) − f(x−h))/2h has error proportional to h², so it is much more accurate for the same h.
  5. Why not use a tiny h like 1e-15? Floating-point cancellation: rounding error divided by h dominates.
  6. What is a convex function and why is it nice? Any chord lies above the curve; every local minimum is global, so gradient methods find the best answer.
  7. Does a zero derivative guarantee a minimum? No: it can be a maximum, a local minimum or a saddle/inflection point (like x³ at 0).

Glossary

Term Meaning Example
Function A rule giving exactly one output for each input profit(p)
Input / output What goes in / what comes out of a function price → profit
Knob (parameter) A number we can tune the price; a model's weights
Brute force (blind search) Trying every possible value max(range(5, 41), key=profit)
Exponential growth Multiplying by a fixed factor each step 100¹⁰ = 10²⁰
Slope Rise ÷ run "1 in 10" = 0.1
Rate of change Change in output per unit change of input 60 km per hour
Limit The value something approaches as a step shrinks to 0 slope → 125
Derivative (f′, d/dx) The slope at one exact point profit′(10) = 125
Floating point How computers store decimals, about 16 digits tiny h gives 227
Forward difference (f(x + h) − f(x)) / h first slope function
Central difference (f(x + h) − f(x − h)) / 2h accurate numerical slope
Numerical derivative A derivative measured by computing slope(profit, 10)
Constant A plain number that never changes 7
Power rule (xⁿ)′ = n·xⁿ⁻¹ (x⁴)′ = 4x³
Sum rule (f + g)′ = f′ + g′ (x² + 7x)′ = 2x + 7
Constant multiple rule (c·f)′ = c·f′ (7x)′ = 7
Product rule (f·g)′ = f′g + fg′ profit′ = −10p + 225
Inner / outer function In f(g(x)), g is inner and f is outer 3x + 1 inside u²
Chain rule Derivative of nested functions = outer rate × inner rate ((3x + 1)²)′ = 6(3x + 1)
Backpropagation Training a neural network by the chain rule later in the course
Maximum / minimum Highest / lowest point Rs 22.50 is the maximum of profit
Stationary point Where the derivative is 0 x = 22.5
Convex function Bowl-shaped; any chord lies above the curve; one minimum (x − 3)² + 1
Non-convex function Has several valleys or peaks x⁴ − 4x² + x
Local minimum Lowest only in its neighbourhood x ≈ 1.3
Global minimum Lowest of all x ≈ −1.5
Gradient descent Repeatedly stepping downhill using the slope episode 8
lambda A one-line Python function without a name lambda x: x ** 2

Summary

  • A function maps each input to one output; ML wants the input (weights) with the smallest output (error).
  • Brute force explodes with every extra knob; we need to know which way to move.
  • The derivative is the slope at a point, the rate of change; measure it with the central difference (h ≈ 1e-5, never tiny).
  • Power, sum, constant-multiple and product rules give derivatives on paper; the chain rule multiplies rates through nested functions.
  • Maxima and minima have zero slope: set f′ = 0 and solve (Raju: Rs 22.50).
  • Convex functions have one bottom; non-convex ones have local minima where a zero slope isn't the best.

Next episode: Maths for ML 5, partial derivatives and gradients: many knobs at once, the gradient vector that points uphill, and the chain rule on a computational graph.