SyntaxPilot
Courses/Data Analytics with Python/Lesson 10

Data Analytics with Python · Lesson 10 of 90 · 16 min

NumPy 3: broadcasting, random numbers and NaN

broadcastingbroadcasting rulesnp.random.default_rngseedreproducibilityrng.normalrng.integersrng.choicesimulationnp.nanNaN propagationnan-functions (np.nanmean)copy vs view

These are the notes for Data Analytics with Python, episode 10, the last NumPy episode before pandas. In episode 9 we picked cells out of Ramesh's month of chai sales, filtered with masks and totalled along an axis. Here we learn broadcasting (maths between arrays of different shapes), random numbers with seeds for reproducible simulations, NaN (the value that means missing) and the copy vs view trap. Every term is defined the first time it appears and collected in the Glossary. Every program below was run for real (Python 3.13, NumPy 2.5); outputs are unedited. Where the output is random on purpose, the notes say so.

The problem: what did each day earn?

Ramesh's month from episode 9: 4 weeks (rows) x 7 days (columns, Mon to Sun), cups of chai sold. Its shape is (4, 7).

import numpy as np
cups = np.array([[320, 280, 410, 300, 350, 390, 450],
                 [310, 295, 380, 330, 360, 420, 470],
                 [290, 260, 400, 310, 340, 405, 440],
                 [330, 300, 430, 320, 370, 415, 480]])

He wants money, not cups. Every cup costs 10 rupees. Then he changes his mind: weekends cost 12. Now there are 7 prices and 28 cells: two arrays of different shapes. Can NumPy multiply them?

Broadcasting

Element-wise (episode 8) means an operation is applied to every element, one by one. A scalar is a single number, like 10.

print(cups * 10)
[[3200 2800 4100 3000 3500 3900 4500]
 [3100 2950 3800 3300 3600 4200 4700]
 [2900 2600 4000 3100 3400 4050 4400]
 [3300 3000 4300 3200 3700 4150 4800]]

NumPy behaved as if 10 were a whole 4 x 7 grid of tens. That stretching of a smaller array to fit a bigger one is called broadcasting.

Now 7 prices, one per weekday:

price = np.array([10, 10, 10, 10, 10, 12, 12])
print(price.shape, cups.shape)
print(cups * price)
(7,) (4, 7)
[[3200 2800 4100 3000 3500 4680 5400]
 [3100 2950 3800 3300 3600 5040 5640]
 [2900 2600 4000 3100 3400 4860 5280]
 [3300 3000 4300 3200 3700 4980 5760]]

The row of 7 prices was reused for every week. The first Saturday's 390 cups became 4,680 rupees (390 x 12).

Real-life example: a rangoli stamp. One dot stamped on every tile = a scalar on a grid. A 7-tile strip stamped on every row of a 7-tile-wide floor = a (7,) row on a (4, 7) grid. A 4-tile strip on a 7-tile row doesn't fit. Broadcasting does not really copy the small array in memory; it reuses it, so it stays fast.

When it fails

Ramesh gives a bonus per week: 0, 20, 40 and 60 extra cups. Four numbers, one per row.

bonus = np.array([0, 20, 40, 60])
print(cups + bonus)
ValueError: operands could not be broadcast together with shapes (4,7) (4,) 

Operands are the things an operator works on (the two sides of +). We meant "one number per row", but NumPy lines up shapes from the right: the grid's last size is 7, the bonus has 4, and 7 is not 4.

The broadcasting rules

  1. Write the two shapes one under the other, lined up on the right.
  2. Each pair of sizes must be equal, or one of them must be 1.
  3. A size of 1 (or a missing size) is stretched to match the other.
Shapes Right-most pair Next pair Result
(4, 7) and (7,) 7 = 7 4 vs missing → stretch (4, 7)
(4, 7) and (4,) 7 vs 4 ValueError
(4, 7) and (4, 1) 7 vs 1 → stretch 4 = 4 (4, 7)
(3, 1) and (4,) 1 vs 4 → stretch 3 vs missing → stretch (3, 4)

The fix is to make the bonus a column (4 rows, 1 column) with reshape from episode 9:

bonus = np.array([0, 20, 40, 60])
col = bonus.reshape(4, 1)
print(col.shape)
print(cups + col)
(4, 1)
[[320 280 410 300 350 390 450]
 [330 315 400 350 380 440 490]
 [330 300 440 350 380 445 480]
 [390 360 490 380 430 475 540]]

Rule of thumb: a row has shape (1, n) or (n,), a column has shape (n, 1). (bonus[:, np.newaxis] and bonus[:, None] also make a column.)

Both sides can stretch at once, which builds a whole table in one line:

import numpy as np
a = np.array([1, 2, 3]).reshape(3, 1)
b = np.array([10, 20, 30, 40])
print(a + b)
[[11 21 31 41]
 [12 22 32 42]
 [13 23 33 43]]

A real use: above or below the usual level?

Subtract each weekday's average (shape (7,), from axis=0) from the whole grid:

day_avg = cups.mean(axis=0)
print((cups - day_avg).round(1))
[[  7.5  -3.8   5.  -15.   -5.  -17.5 -10. ]
 [ -2.5  11.2 -25.   15.    5.   12.5  10. ]
 [-22.5 -23.8  -5.   -5.  -15.   -2.5 -20. ]
 [ 17.5  16.2  25.    5.   15.    7.5  20. ]]

Positive = better than usual for that weekday. Week 4 is positive everywhere (a strong week); week 3 is negative everywhere. Subtracting the mean like this is called centring the data.

Random numbers

New problem: Ramesh buys milk the night before. Too little and chai runs out; too much and it spoils. Next month hasn't happened, so he can't measure it. Instead we simulate: imagine thousands of possible days with random numbers and count how often he runs out. A simulation is imagining many possible outcomes with random numbers.

np.random.default_rng() creates a random number generator (called rng by convention): an object that produces random numbers. rng.integers(low, high, size=...) gives random whole numbers from low up to but not including high (like range). A dice:

import numpy as np
rng = np.random.default_rng()
print(rng.integers(1, 7, size=5))
[3 2 6 5 6]

Run it again and you get different numbers (one real second run gave [4 2 6 1 6]); your output will differ too. That is what random means, but it is a problem for analysis: if your report says 667 bad days today and 671 tomorrow, nobody can check it.

Seeds and reproducibility

Computer random numbers are pseudo-random: they come from a fixed recipe that starts from a number called the seed. With no seed, NumPy takes a fresh one from the operating system every run. With a seed, the sequence is the same every time, on every computer.

Real-life example: shuffling cards by a written recipe. Recipe #42 always gives the same order, and a friend using recipe #42 gets the same order too. Being able to repeat a result exactly is called reproducibility.

import numpy as np
rng = np.random.default_rng(42)
print(rng.integers(1, 7, size=5))
rng = np.random.default_rng(42)
print(rng.integers(1, 7, size=5))
[1 5 4 3 3]
[1 5 4 3 3]

Tips:

  • Create one generator at the top of a notebook with a seed and use it everywhere.
  • The number itself doesn't matter (42, 7, 2026...); writing it down does.
  • The older style np.random.seed(42) + np.random.randint(...) still works but NumPy recommends default_rng for new code.

rng.normal: the bell curve

Dice give every value an equal chance. Real sales don't: most days are near the usual level and very quiet or very busy days are rare. A distribution describes which values happen and how often. The normal distribution (also called the Gaussian or "bell curve") piles values up in the middle and thins out symmetrically on both sides.

rng.normal(mean, std, size=n) draws n values from a normal distribution with that mean (centre) and standard deviation (spread, episode 9). For Ramesh: mean 360 cups, std 60.

import numpy as np
rng = np.random.default_rng(7)
sim = rng.normal(360, 60, size=30).round()
print(sim[:6])
print(sim.mean().round(1), sim.std().round(1))
[360. 378. 344. 307. 333. 301.]
335.2 49.3

30 simulated days averaged 335.2 with spread 49.3, not exactly 360 and 60. Small samples wobble; more samples land closer to the true values (the statistics episodes call this the law of large numbers).

rng.choice: picking with probabilities

A probability is a chance between 0 (never) and 1 (always); 0.6 means 60%. rng.choice(items, size=n, p=[...]) picks n items, each with its probability (the p values must add up to 1).

import numpy as np
rng = np.random.default_rng(1)
snacks = np.array(["chai", "samosa", "bun"])
orders = rng.choice(snacks, size=10,
                    p=[0.6, 0.3, 0.1])
print(orders[:5])
names, counts = np.unique(orders, return_counts=True)
print(names, counts)
['chai' 'bun' 'chai' 'bun' 'chai']
['bun' 'chai' 'samosa'] [2 7 1]

Mostly chai, but not exactly 6/3/1: 10 orders is a tiny sample. rng.choice(10, size=3, replace=False) picks 3 different numbers from 0 to 9 (no repeats), e.g. a lucky-draw.

A simulation: how often does demand pass 450?

import numpy as np
rng = np.random.default_rng(42)
days = rng.normal(360, 60, size=10_000)
print((days > 450).sum(), "of", days.size)
print((days > 450).mean())
667 of 10000
0.0667

10_000 is just ten thousand; Python lets you put underscores in numbers for readability. The mean of a True/False mask is the fraction of Trues (True = 1, False = 0): about 6.7% of days need more than 450 cups, roughly 2 days a month. Repeating a random experiment many times to estimate a chance like this is called a Monte Carlo simulation.

NaN: missing values

New problem: one Wednesday wasn't written down. Writing 0 would be a lie (0 means "sold nothing") and drags the average down. We need a value that means unknown.

First try: Python's None (episode 4, "no value").

import numpy as np
week = np.array([320, 280, None, 300, 350, 390, 450])
print(week.dtype)
print(week.mean())
object
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

The dtype (data type) became object: a slow array of general Python objects instead of fast numbers, and int + None is impossible.

The fix is np.nan, short for Not a Number: a special float value that means missing or undefined.

import numpy as np
week = np.array([320, 280, np.nan, 300, 350, 390, 450])
print(week)
print(week.dtype)
print(week.sum(), week.mean())
print(week * 2)
[320. 280.  nan 300. 350. 390. 450.]
float64
nan nan
[640. 560.  nan 600. 700. 780. 900.]

The array is float64 again (fast numbers), but any maths with NaN gives NaN. This is NaN propagation: one missing value spreads into every result. It is honest: NumPy refuses to pretend it knows the total.

Finding NaN: never with ==

print(np.nan == np.nan)
print(week == np.nan)
print(np.isnan(week))
print(np.isnan(week).sum(), "missing")
False
[False False False False False False False]
[False False  True False False False False]
1 missing

NaN is not equal to anything, not even itself, so week == np.nan silently finds nothing. Use np.isnan: True exactly where a value is missing; its .sum() counts the gaps.

nan-functions

nan-functions are versions of the aggregations that ignore NaN: np.nansum, np.nanmean, np.nanmax, np.nanmin, np.nanstd, np.nanmedian.

print(np.nansum(week), np.nanmean(week).round(1))
print(np.nanmax(week), np.nanmin(week))
print(week[~np.isnan(week)])
2090.0 348.3
450.0 280.0
[320. 280. 300. 350. 390. 450.]

The mean is from the six known days. ~np.isnan(week) (episode 9's ~ = not) keeps only the real values.

Integers can't hold NaN

import numpy as np
w = np.array([320, 280, 410, 300])
w[2] = np.nan
ValueError: cannot convert float NaN to integer

NaN exists only as a float (type(np.nan) is <class 'float'>). That is why a column of whole numbers with a gap becomes float, in NumPy and in pandas. Fix: w = w.astype(float) first.

Filling a gap

import numpy as np
week = np.array([320, 280, np.nan, 300, 350, 390, 450])
avg = np.nanmean(week)
filled = np.where(np.isnan(week), avg, week)
print(filled.round(1))
[320.  280.  348.3 300.  350.  390.  450. ]

Filling missing values with a reasonable guess is called imputation. Whether the mean is a good guess is a big question; episode 19 (Cleaning 1: missing values) covers it properly.

Copy vs view

Real-life example: a view is a window onto Ramesh's register: write on the window and you write in the register. A copy is a photocopy: scribble freely, the original is safe.

Basic slicing (cups[0], cups[1:3, :]) gives a view: a new array object that shares memory (the same stored numbers) with the original.

import numpy as np
cups = np.arange(1, 13).reshape(3, 4)
first = cups[0]
first[0] = 99
print(cups)
print(np.shares_memory(cups, first))
[[99  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]
True

The original changed! NumPy avoids copying to stay fast and memory-light. To change values safely, take a copy with .copy():

import numpy as np
cups = np.arange(1, 13).reshape(3, 4)
first = cups[0].copy()
first[0] = 99
print(cups[0], first)
print(np.shares_memory(cups, first))
[1 2 3 4] [99  2  3  4]
False

Boolean masks and fancy indexing always give a copy:

import numpy as np
cups = np.arange(1, 13).reshape(3, 4)
big = cups[cups > 10]
big[0] = 0
print(cups[2], big)
[ 9 10 11 12] [ 0 12]
Operation Gives Changing the result changes the original?
Slice a[0], a[1:3], a[:, 0] view yes
a.reshape(...), a.T (usually) view yes
Boolean mask a[a > 10] copy no
Fancy indexing a[[0, 2]] copy no
a.copy(), a.flatten() copy no

pandas has its own version of this question; episode 14 covers it (pandas 3 uses Copy-on-Write).

Mini project: the milk planner

Assume 1 litre of milk ≈ 1 cup (a simplification). Simulate 10,000 days and check three stock levels:

import numpy as np
rng = np.random.default_rng(42)
need = rng.normal(360, 60, size=10_000)
for stock in [400, 450, 500]:
    short = (need > stock).mean().round(3)
    print(stock, "litres -> runs out", short)
400 litres -> runs out 0.245
450 litres -> runs out 0.067
500 litres -> runs out 0.011

400 litres runs out on about a quarter of days, 450 on under 7%, 500 on about 1%. A little extra stock buys a lot of safety, and because the seed is fixed, anyone can re-run and check the numbers.

Common mistakes

Code Real error Why Fix
cups + np.array([0, 20, 40, 60]) ValueError: operands could not be broadcast together with shapes (4,7) (4,) Shapes line up from the right: 7 vs 4 bonus.reshape(4, 1)
np.array([320, None, 300]).mean() TypeError: unsupported operand type(s) for +: 'int' and 'NoneType' None makes an object array Use np.nan
week == np.nan (no error: all False) NaN is never equal, even to itself np.isnan(week)
w[2] = np.nan on an int array ValueError: cannot convert float NaN to integer NaN only exists as a float w = w.astype(float)
week.mean() with a gap (no error: nan) NaN propagates np.nanmean(week)
first = cups[0]; first[0] = 99 (no error: cups changed) A slice is a view cups[0].copy()
default_rng() in a report (no error: numbers change every run) No seed default_rng(42)

Practice

1. Simulate 1,000 rolls of two dice with seed 1. How often is the total 7?

Solution
import numpy as np
rng = np.random.default_rng(1)
d1 = rng.integers(1, 7, size=1000)
d2 = rng.integers(1, 7, size=1000)
total = d1 + d2
print((total == 7).mean())
0.164

The exact answer is 6/36 ≈ 0.167; the simulation is close.

2. A 4 x 3 array of marks (students x subjects). Subtract each subject's mean with broadcasting.

Solution
import numpy as np
marks = np.array([[78, 65, 90],
                  [82, 70, 85],
                  [60, 75, 95],
                  [88, 80, 70]])
print(marks.mean(axis=0))
print(marks - marks.mean(axis=0))
[77.  72.5 85. ]
[[  1.   -7.5   5. ]
 [  5.   -2.5   0. ]
 [-17.    2.5  10. ]
 [ 11.    7.5 -15. ]]

3. A week of steps with one missing day: print the sum, the nansum and the number missing.

Solution
import numpy as np
steps = np.array([6200, 4800, np.nan, 7500, 3900, 11000, 8600])
print(steps.sum(), np.nansum(steps), np.isnan(steps).sum())
nan 42000.0 1

4. Prices for chai, samosa, bun are [10, 12, 15]. Two customers bought [[2, 0, 1], [1, 3, 0]]. What does each pay?

Solution
import numpy as np
price = np.array([10, 12, 15])
qty = np.array([[2, 0, 1],
                [1, 3, 0]])
print(qty * price)
print((qty * price).sum(axis=1))
[[20  0 15]
 [10 36  0]]
[35 46]

5. Add [1, 2] to each row of np.arange(6).reshape(2, 3) as a per-row value (row 0 + 1, row 1 + 2).

Solution
import numpy as np
a = np.arange(6).reshape(2, 3)
print(a + np.array([1, 2]))
ValueError: operands could not be broadcast together with shapes (2,3) (2,) 
print(a + np.array([1, 2]).reshape(2, 1))
[[1 2 3]
 [5 6 7]]

Quick quiz

a = np.ones((3, 1))
b = np.ones(5)
print((a + b).shape)

A. (3, 1) B. (5,) C. (3, 5) D. ValueError

Answer

C. (3, 5). Line up from the right: 1 vs 5 → the 1 stretches to 5; 3 vs nothing → stays 3. (np.ones(shape) makes an array full of 1.0.)

Interview questions

  1. What is broadcasting? NumPy's way of doing element-wise operations on arrays of different shapes by virtually stretching size-1 (or missing) dimensions, without copying data.
  2. State the broadcasting rules. Compare shapes from the right; each pair must be equal or one of them 1; missing dimensions count as 1. Otherwise ValueError: operands could not be broadcast together.
  3. How do you add one value per row of a (4, 7) array? Make the values a column: v.reshape(4, 1) or v[:, None].
  4. Why set a seed? For reproducibility: the same seed gives the same random sequence, so results can be checked and re-run.
  5. default_rng vs np.random.seed? default_rng(seed) creates an independent Generator object (the recommended API); np.random.seed sets a hidden global state (legacy).
  6. Why is np.nan == np.nan False? By the IEEE 754 floating-point standard, NaN compares unequal to everything, including itself. Use np.isnan.
  7. Why does an integer column with a missing value become float? NaN is a float value; integer arrays cannot represent it.
  8. mean vs nanmean? mean returns NaN if any value is NaN; nanmean ignores NaNs.
  9. Is a slice a copy? No, basic slicing returns a view sharing memory; masks and fancy indexing return copies. Use .copy() before modifying.

Glossary

Term Meaning Example
Element-wise Applied to each element, one by one cups * 10
Scalar A single number (not an array) 10
Broadcasting Stretching a smaller array to fit a bigger one in maths cups * price
Operands The values an operator works on in a + b: a and b
Broadcasting rules Line up shapes from the right; sizes equal or 1 (4, 7) + (4, 1) works
Row vector / column vector Shape (n,) or (1, n) / shape (n, 1) bonus.reshape(4, 1)
Centring Subtracting the mean so values show above/below average cups - cups.mean(axis=0)
Simulation Imagining many possible outcomes with random numbers 10,000 simulated days
Monte Carlo simulation Estimating a chance by repeating a random experiment many times (days > 450).mean()
Random number generator (rng) An object that produces random numbers np.random.default_rng()
np.random.default_rng Creates a generator (optionally with a seed) default_rng(42)
Pseudo-random Looks random but comes from a fixed recipe NumPy's generator
Seed The starting number of the recipe; same seed, same numbers 42
Reproducibility Getting exactly the same result when re-run seeded simulation
rng.integers Random whole numbers from low up to (not incl.) high rng.integers(1, 7, size=5)
Distribution Which values happen and how often most days near 360
Normal distribution A symmetric bell-shaped distribution heights, daily sales
rng.normal Random values from a normal distribution (mean, std) rng.normal(360, 60, size=30)
Mean Average: total ÷ count sim.mean()
Standard deviation (std) Typical distance of values from the mean 60
Sample The values you actually have or drew 30 simulated days
Probability A chance between 0 and 1 0.6 = 60%
rng.choice Random picks from items, optionally with probabilities p rng.choice(snacks, p=[0.6, 0.3, 0.1])
replace=False Pick without repeats lucky draw
Fraction from a mask Mean of True/False = share of True (days > 450).mean() → 0.0667
Missing value A value that wasn't recorded the unwritten Wednesday
None Python's "no value" object makes an object array
object dtype Array of general Python objects (slow, no number maths) np.array([1, None])
NaN (np.nan) "Not a Number": a float meaning missing/undefined np.array([1, np.nan])
float64 64-bit decimal number dtype week.dtype
NaN propagation Any maths with NaN gives NaN week.sum() → nan
np.isnan True where a value is NaN np.isnan(week).sum() → 1
nan-functions Aggregations that ignore NaN np.nanmean, np.nansum
astype Convert an array to another dtype w.astype(float)
Imputation Filling missing values with a reasonable guess fill with the mean
View An array that shares memory with another cups[0]
Copy An array with its own independent data cups[0].copy()
Shares memory Two arrays point at the same stored numbers np.shares_memory(a, b)
np.ones / np.arange Array of 1.0s / evenly spaced integers (episode 8) np.ones((3, 1)), np.arange(1, 13)

Summary

  • Broadcasting lets arrays of different shapes work together: a scalar or a smaller array is stretched to fit.
  • Rules: line shapes up from the right; each pair must be equal or 1. (4, 7) + (4,) fails; make it a column with reshape(4, 1).
  • np.random.default_rng(seed) gives reproducible random numbers: integers (whole numbers), normal (bell curve), choice (picks with probabilities).
  • A simulation + a mask's mean estimates how often something happens (milk running out).
  • np.nan marks missing numbers; it propagates through maths, is found with np.isnan (never ==), and is skipped by nan-functions. Integer arrays can't hold NaN.
  • Slices are views (changing them changes the original); masks and fancy indexing are copies; use .copy() to be safe.
  • Next episode: pandas 1: the Series.