SyntaxPilot
Courses/Machine Learning/Lesson 7

Machine Learning · Lesson 7 of 90 · 12 min

Maths 5: partial derivatives and gradients

multivariable functionpartial derivativegradient vectordirection of steepest ascentJacobian (intuition)Hessian and curvature (intuition)computational graphchain rule on a graph

Machine Learning Full Course · Episode 7. Every code sample below was run with Python 3.13 in plain Python (no libraries); the output shown is the real output. You only need episode 6 (derivatives: slope, power/sum/product rules, chain rule) and episode 3 (vectors, dot product).

The problem: two knobs at once

Raju sells chai. Last episode he had one knob, the price. Now he adds a second knob: spoons of sugar per cup.

  • A higher price gives more margin (what he keeps per cup) but fewer buyers.
  • A spoon or two of sugar brings more buyers; too much drives them away.
  • Each spoon costs him Rs 1, so sugar also eats into the margin.

The two knobs pull against each other. Which pair (price, sugar) gives the most profit? A machine learning model has the same problem with millions of knobs (its weights), and wants the lowest error instead of the highest profit.

Multivariable functions

A variable is an input that can change. A multivariable function is a rule with more than one input and (here) one output.

  • cups(p, s) = 200 − 5p + 20s − 5s²
  • profit(p, s) = (p − 5 − s) × cups(p, s)

The −5s² term is what makes too much sugar backfire. Picture the function as a landscape: walking east raises the price, walking north raises the sugar, and the height of the ground is the profit.

def profit(p, s):            # price p, spoons of sugar s
    cups = 200 - 5 * p + 20 * s - 5 * s ** 2
    return (p - 5 - s) * cups

for p, s in [(20, 0), (20, 1), (20, 2), (25, 1)]:
    print(p, s, profit(p, s))

Output:

20 0 1500
20 1 1610
20 2 1560
25 1 1710

Both knobs change the profit: one spoon beats zero and two; a higher price helps too (from here).

The obvious attempt fails

Reuse last episode's slope function:

def profit(p, s):            # price p, spoons of sugar s
    cups = 200 - 5 * p + 20 * s - 5 * s ** 2
    return (p - 5 - s) * cups

def slope(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

print(profit(20, 1))
print(slope(profit, 20))

Output (Python 3.13, saved as main.py):

1610
Traceback (most recent call last):
  File "main.py", line 9, in <module>
    print(slope(profit, 20))
          ~~~~~^^^^^^^^^^^^
  File "main.py", line 6, in slope
    return (f(x + h) - f(x - h)) / (2 * h)
            ~^^^^^^^
TypeError: profit() missing 1 required positional argument: 's'

(The traceback shows the full path of your file; older Python versions leave out the ~~^^ marker lines. The last line is always the same.)

Why it fails: a one-input derivative nudges the input. With two inputs, which one do we nudge, and what is the other one doing meanwhile?

Partial derivatives

Real-life example: a sound mixing desk. To learn what the bass slider does, push only the bass slider and leave every other slider exactly where it is, then measure how much the sound changed per notch.

A partial derivative does exactly that: freeze every input except one, nudge that one, measure the slope. It is written with a curly d (said "del"):

  • ∂profit/∂p: the slope along p, with s frozen
  • ∂profit/∂s: the slope along s, with p frozen
def profit(p, s):
    cups = 200 - 5 * p + 20 * s - 5 * s ** 2
    return (p - 5 - s) * cups

h = 1e-5
p, s = 20, 1
dp = (profit(p + h, s) - profit(p - h, s)) / (2 * h)   # s frozen
ds = (profit(p, s + h) - profit(p, s - h)) / (2 * h)   # p frozen
print("d profit / d p:", round(dp, 4))
print("d profit / d s:", round(ds, 4))

Output:

d profit / d p: 45.0
d profit / d s: 25.0

At (20, 1): one more rupee adds about Rs 45 profit; one more spoon adds about Rs 25.

On paper: treat every other letter as a plain number and use episode 6's rules. With margin m = p − 5 − s and cups c:

  • ∂profit/∂p = 1·c + m·(−5) = 115 − 70 = 45 (product rule; ∂m/∂p = 1, ∂c/∂p = −5)
  • ∂profit/∂s = (−1)·c + m·(20 − 10s) = −115 + 14 × 10 = 25

The gradient

The gradient collects all the partial derivatives into one vector (an arrow: a list of numbers with a direction and a length). It is written ∇f, said "nabla f" or "grad f".

∇profit(20, 1) = (45, 25)

  • Direction: how to move all the knobs together (more price than sugar, roughly 2 : 1).
  • Length: √(45² + 25²) ≈ 51.5, how steep the ground is.

Direction of steepest ascent (measured)

Take one step of the same length (0.01) in five directions and measure the change in profit:

import math

def profit(p, s):
    cups = 200 - 5 * p + 20 * s - 5 * s ** 2
    return (p - 5 - s) * cups

grad = (45, 25)                       # (d/dp, d/ds) at (20, 1)
norm = math.hypot(*grad)
step = 0.01                           # same step length every time
dirs = {"p only": (1, 0), "s only": (0, 1), "both equally": (0.7071, 0.7071),
        "gradient": (grad[0] / norm, grad[1] / norm), "minus gradient": (-grad[0] / norm, -grad[1] / norm)}
for name, (a, b) in dirs.items():
    gain = profit(20 + step * a, 1 + step * b) - profit(20, 1)
    print(f"{name:15} {gain:+.4f}")

Output:

p only          +0.4495
s only          +0.2420
both equally    +0.4915
gradient        +0.5131
minus gradient  -0.5164
  • math.hypot(*grad) is the length √(45² + 25²); dividing by it makes a unit vector (length 1), so every direction gets the same step length.
  • f"{name:15} {gain:+.4f}" pads the name to 15 characters and prints the gain with a sign and 4 decimals.

The gradient is the direction of steepest ascent; minus the gradient is the direction of steepest descent. Why: for a small step u, the change is ≈ ∇f · u (a dot product, episode 3), and a dot product with a fixed-length u is largest when u points the same way as ∇f.

The Jacobian (intuition)

When a function has several outputs, put each output's partial derivatives in a row: rows = outputs, columns = inputs. That grid is the Jacobian matrix. Real-life example: a school report card (rows = students, columns = subjects).

The Hessian and curvature (intuition)

A second derivative is the slope of the slope: how fast the slope itself changes (when a car brakes, the change in speed changes too). It describes curvature, how the ground bends:

  • negative: curving down, like the top of a drum or a dome (near a peak)
  • positive: curving up, like a bowl (near a valley)

With several inputs, the second partial derivatives for every pair of inputs form the Hessian matrix.

def cups(p, s):
    return 200 - 5 * p + 20 * s - 5 * s ** 2

def profit(p, s):
    return (p - 5 - s) * cups(p, s)

def partial(f, x, i, h=1e-5):         # nudge input i only
    up, dn = list(x), list(x)
    up[i] += h; dn[i] -= h
    return (f(*up) - f(*dn)) / (2 * h)

x = [20, 1]
J = [[round(partial(f, x, i), 3) for i in range(2)] for f in (cups, profit)]
print("Jacobian:", J)
dp = lambda p, s: partial(profit, [p, s], 0)
ds = lambda p, s: partial(profit, [p, s], 1)
H = [[round(partial(g, x, i, 1e-3), 2) for i in range(2)] for g in (dp, ds)]
print("Hessian: ", H)

Output:

Jacobian: [[-5.0, 10.0], [45.0, 25.0]]
Hessian:  [[-10.0, 15.0], [15.0, -160.0]]
  • f(*up) unpacks the list [p, s] into two arguments.
  • lambda p, s: ... is a one-line function without a name.
  • Jacobian: the cups row (−5, 10): each rupee loses 5 cups, each spoon adds 10 (20 − 10s at s = 1). The profit row is the gradient (45, 25). The gradient is one row of a Jacobian.
  • Hessian: negative diagonal (−10, −160): the ground curves down in both directions, like a dome. It is symmetric (15 twice): for smooth functions the order of nudging doesn't matter.

By hand: ∂²profit/∂p² = −10, ∂²profit/∂s² = −2(20 − 10s) − 10m = −20 − 140 = −160, ∂²profit/∂p∂s = (20 − 10s) + 5 = 15.

Computational graphs

A computational graph breaks a formula into small steps: boxes (nodes) for the steps, arrows for the data. Real-life example: a dabbawala route, where each tiffin passes through several hands.

p, s ──► m = p − 5 − s ─────────────┐
   │                                 ├──► profit = m × c
   └───► c = 200 − 5p + 20s − 5s² ──┘
  • Forward pass: inputs → output, computing values.
  • Backward pass: output → inputs, computing slopes.

The chain rule on a graph

Along one path, multiply the local slopes (the chain rule from episode 6). When an input reaches the output by several paths, add the paths.

p, s = 20, 1
# forward pass: one small step per node
m = p - 5 - s                          # margin per cup
c = 200 - 5 * p + 20 * s - 5 * s ** 2  # cups sold
profit = m * c
print("forward:", m, c, profit)
# backward pass: local slopes, multiplied along each path
d_m, d_c = c, m                        # profit = m * c
d_p = d_m * 1 + d_c * (-5)             # p reaches profit via m AND via c
d_s = d_m * (-1) + d_c * (20 - 10 * s)
print("backward:", d_p, d_s)

Output:

forward: 14 115 1610
backward: 45 25

Exactly the gradient, with no nudging. Why it matters: nudging needs two function runs per knob (two million runs for a million knobs); the backward pass gets every partial derivative in roughly one extra pass. Deep learning libraries do this automatically; it is called backpropagation.

Follow the gradient

def profit(p, s):
    cups = 200 - 5 * p + 20 * s - 5 * s ** 2
    return (p - 5 - s) * cups

p, s = 20, 1
for step in range(4):
    gp = -5 * (p - 5 - s) + (200 - 5 * p + 20 * s - 5 * s ** 2)
    gs = -(200 - 5 * p + 20 * s - 5 * s ** 2) + (p - 5 - s) * (20 - 10 * s)
    print(step, round(p, 3), round(s, 3), round(profit(p, s), 2))
    p, s = p + 0.01 * gp, s + 0.01 * gs

Output:

0 20 1 1610
1 20.45 1.25 1632.11
2 20.889 1.166 1650.12
3 21.274 1.273 1664.92

Each small step along the gradient climbs. Step against it on a model's error and you have gradient descent (episode 8).

Common mistakes

Mistake Result Why Fix
slope(profit, 20) on a two-input function TypeError: profit() missing 1 required positional argument: 's' a one-input slope can't know which input to nudge partial derivative: nudge one, freeze the rest
nudging p and s together a mix of both slopes not one input at a time one partial per input
tuning one knob at a time +0.4495 instead of +0.5131 per step ignores the other knobs step along the full gradient
backward pass through one path only 115 instead of 45 for ∂/∂p p reaches profit through m and c multiply along each path, then add the paths
treating a frozen input as 0 wrong partials frozen means "held at its value", not zero keep s = 1 while nudging p
comparing directions with different step lengths unfair comparison longer steps always change more use unit vectors (divide by the length)

Mini project: a gradient for any function

A numerical gradient that works for any number of inputs. Raju adds a third knob: millilitres of milk per cup (m), each ml costs 5 paise and a little milk brings customers.

def numerical_gradient(f, x, h=1e-5):
    grad = []
    for i in range(len(x)):
        up, dn = list(x), list(x)
        up[i] += h; dn[i] -= h
        grad.append((f(*up) - f(*dn)) / (2 * h))
    return grad

def bill(p, s, m):                 # price, sugar, millilitres of milk per cup
    cups = 200 - 5 * p + 20 * s - 5 * s ** 2 + 0.5 * m - 0.01 * m ** 2
    cost = 5 + s + 0.05 * m
    return (p - cost) * cups

x = [20, 1, 20]
print("value:", round(bill(*x), 2))
print("gradient:", [round(g, 4) for g in numerical_gradient(bill, x)])

Output:

value: 1573.0
gradient: [56.0, 9.0, -4.75]

Raise the price, add a little sugar, and use less milk (negative partial). Check by hand: cups = 121, margin = 13; ∂/∂m = −0.05 × 121 + 13 × (0.5 − 0.02 × 20) = −6.05 + 1.3 = −4.75.

Practice

  1. Gradient of f(x, y) = x²y + 3y at (1, 4); check numerically.
Solution

∂f/∂x = 2xy = 8, ∂f/∂y = x² + 3 = 4, so ∇f(1, 4) = (8, 4). With the partial function above: [round(partial(f, [1, 4], i), 4) for i in range(2)] → [8.0, 4.0].

  1. Show that minus the gradient is the steepest way down from (20, 1).
Solution

In the steepest-ascent program, the "minus gradient" row prints -0.5164, the most negative change of all the directions; any other unit direction loses less.

  1. Hessian of g(x, y) = x² + 3xy + y².
Solution

∂²g/∂x² = 2, ∂²g/∂y² = 2, ∂²g/∂x∂y = 3: H = [[2, 3], [3, 2]] everywhere. Numerically at (1, 2): [[2.0, 3.0], [3.0, 2.0]].

  1. Draw the graph of f = (x + y)·x and do the backward pass at (2, 3).
Solution

u = x + y, f = u·x. Forward: u = 5, f = 10. Backward: ∂f/∂u = x = 2, and x reaches f directly (slope u = 5) and through u (slope 2 × 1), so ∂f/∂x = 5 + 2 = 7 (= 2x + y); ∂f/∂y = 2 × 1 = 2. The code prints forward: 5 10 backward: 7 2.

x, y = 2, 3
u = x + y            # forward
f = u * x
d_u, d_x_direct = x, u          # f = u * x
d_x = d_u * 1 + d_x_direct      # x reaches f by two paths
d_y = d_u * 1
print("forward:", u, f, " backward:", d_x, d_y)

Quick quiz

def f(x, y):
    return x ** 2 * y + 3 * y

# partial derivatives on paper
dfdx = lambda x, y: 2 * x * y
dfdy = lambda x, y: x ** 2 + 3

print(dfdx(2, 5), dfdy(2, 5))

A) 20 7 B) 4 7 C) 20 28 D) 10 7

Answer

A) 20 7. Freeze y: 2xy = 20. Freeze x: x² + 3 = 7. The gradient at (2, 5) is (20, 7).

Interview questions

  1. What is a partial derivative? The rate of change of a multivariable function with respect to one input while all other inputs are held fixed.
  2. What is the gradient and what does it tell you? The vector of all partial derivatives; it points in the direction of steepest increase, and its length is the steepness.
  3. Why does gradient descent move along the negative gradient? The directional change ≈ ∇f · u is most negative when u points opposite to ∇f.
  4. Gradient vs Jacobian vs Hessian? Gradient: first partials of one output (a vector). Jacobian: first partials of several outputs (a matrix, one row per output). Hessian: second partials of one output (a square, symmetric matrix of curvature).
  5. What does the Hessian tell you at a flat point? Curvature: all-positive curvature means a minimum, all-negative a maximum, mixed signs a saddle point.
  6. What is a computational graph and why do frameworks use one? A formula broken into nodes; the backward pass applies the chain rule node by node and gets all gradients in about one extra pass (backpropagation).
  7. How do you handle a variable used in two places in the graph? Add the gradient contributions from every path.
  8. How can you check a hand-written gradient? Gradient checking: compare it with a central-difference numerical gradient.

Glossary

Term Meaning Example
Variable An input that can change p, s
Multivariable function A function with more than one input profit(p, s)
Margin What is kept from each sale after costs p − 5 − s
Landscape (surface) A picture of a 2-input function: inputs = position, output = height p east, s north, profit = height
Partial derivative (∂f/∂x) Slope along one input with all the others frozen ∂profit/∂p = 45
Gradient (∇f) Vector of all partial derivatives (45, 25)
Vector A list of numbers; an arrow with direction and length (45, 25)
Length (norm) of a vector √(sum of squares) √(45² + 25²) ≈ 51.5
Unit vector A vector of length 1, a pure direction (0.7071, 0.7071)
Dot product Multiply matching entries and add ∇f · u
Steepest ascent / descent The direction of fastest increase / decrease: ∇f / −∇f +0.5131 / −0.5164
Jacobian Matrix of first partials: rows = outputs, columns = inputs [[−5, 10], [45, 25]]
Second derivative The slope of the slope ∂²profit/∂p² = −10
Curvature How the function bends: up (bowl) or down (dome) negative = dome
Hessian Matrix of second partials for every pair of inputs [[−10, 15], [15, −160]]
Symmetric matrix Equal to its transpose (mirror across the diagonal) 15 appears twice
Computational graph A formula as nodes (steps) and arrows (data) m, c → profit
Node One small step of the computation m = p − 5 − s
Forward pass Computing values from inputs to output 14, 115, 1610
Backward pass Computing slopes from output back to inputs 45, 25
Chain rule on a graph Multiply local slopes along a path; add over paths d_p = d_m·1 + d_c·(−5)
Backpropagation The backward pass used to train neural networks later in the course
Gradient checking Comparing an exact gradient with a numerical one numerical_gradient
TypeError Python error: a function got the wrong number or kind of arguments missing argument 's'
*args unpacking f(*[20, 1]) is f(20, 1) f(*up)
lambda A one-line function without a name lambda p, s: ...

Summary

  • A multivariable function has many inputs; a model's error is one, with millions of inputs (weights).
  • A one-input derivative can't handle it (TypeError); a partial derivative nudges one input and freezes the rest.
  • The gradient collects all partials; it points the steepest way up, and −gradient the steepest way down (measured: +0.5131 vs −0.5164).
  • The Jacobian is the grid of partials for many outputs; the Hessian is the grid of second partials, the curvature.
  • A computational graph splits a formula into nodes; the backward pass (multiply along paths, add the paths) gives every partial at once: backpropagation.

Next episode: Maths for ML 6, gradient descent: loss functions, the learning rate, convergence and divergence, local minima and saddle points, built from scratch.