SyntaxPilot
Courses/Data Analytics with Python/Lesson 9

Data Analytics with Python · Lesson 9 of 90 · 17 min

NumPy 2: indexing, masks, reshaping and axes

indexingslicing 2-D arraysfancy indexingboolean maskcomparison arraysreshapeflattentransposeaxisaggregation (sum, mean, std, min, max)argmaxargminnp.wherenp.sort

These are the notes for Data Analytics with Python, episode 9. In episode 8 we built NumPy arrays and did maths on whole arrays at once. Here we learn to pick exactly the data we want out of an array (indexing, slicing, boolean masks, fancy indexing), to change its shape (reshape, flatten, transpose), and to summarise it along rows or columns (axis, sum, mean, min, max, std, argmax, argmin), plus np.where, np.sort and np.unique. Every term is defined the first time it appears and collected in the Glossary. Every program below was run for real (Python 3.13, NumPy 2.x); outputs are unedited.

The problem: a month of sales in a grid

Ramesh's chai stall now has a whole month of sales: 4 weeks, 7 days each. In his notebook he writes one line per week. That is a grid (a table of numbers):

  • A row is one line going across: here, one week.
  • A column is one line going down: here, one weekday (every Monday, every Tuesday, ...).
  • An array with rows and columns is two-dimensional (2-D). A plain list of numbers is one-dimensional (1-D).
  • The shape (from episode 8) lists the size in each direction: (4, 7) means 4 rows and 7 columns.

Every example below starts from this array (rows = weeks 1 to 4, columns = Mon to Sun):

import numpy as np
cups = np.array([[320, 280, 410, 300, 350, 390, 450],
                 [310, 295, 380, 330, 360, 420, 470],
                 [290, 260, 400, 310, 340, 405, 440],
                 [330, 300, 430, 320, 370, 415, 480]])

Ramesh asks: how many cups on Wednesday of week 2? We need a way to point at exactly one cell.

Indexing: cups[row, column]

Real-life example: a cinema ticket. "Row C, seat 5": two pieces of information find exactly one seat.

Indexing means getting a value by its position. A position (or index) is an item's place number, and NumPy, like Python lists (episode 5), counts from 0. So week 2 is row 1, and Wednesday is column 2. Negative indexes count from the end: -1 is the last.

print(cups.shape)
print(cups[1][2])
print(cups[1, 2])
print(cups[-1, -1])
(4, 7)
380
380
480

cups[1][2] works (it first takes row 1, then item 2 of that row), but the NumPy way is one bracket with a comma: cups[1, 2]. It is faster and it is the form used everywhere (pandas' .iloc uses it too).

Asking for a row that doesn't exist:

print(cups[4, 0])
Traceback (most recent call last):
  File "main.py", line 6, in <module>
    print(cups[4, 0])
          ~~~~^^^^^^
IndexError: index 4 is out of bounds for axis 0 with size 4

Out of bounds means outside the grid: only rows 0 to 3 exist. Note the word axis in the message: it is explained below.

Slicing rows and columns

Slicing takes a range of positions: start:stop, where stop is not included (exactly like lists). In 2-D you slice rows and columns separately, with a comma: cups[rows, columns]. A colon on its own, :, means all rows or all columns. An optional third number is the step.

print(cups[1])
print(cups[:, 0])
print(cups[0:2, 5:7])
print(cups[0, ::2])
[310 295 380 330 360 420 470]
[320 310 290 330]
[[390 450]
 [420 470]]
[320 410 350 450]
Code Means Result
cups[1] row 1 (week 2), all columns 7 numbers
cups[:, 0] all rows, column 0 every Monday
cups[0:2, 5:7] rows 0-1, columns 5-6 weekends of weeks 1-2 (2 x 2)
cups[0, ::2] row 0, every 2nd column Mon, Wed, Fri, Sun of week 1

Comparison arrays and boolean masks

New problem: which days sold more than 400 cups? This is about values, not positions: we don't know where they are.

The loop way from episode 6 works, but it is long, and the obvious shortcut fails on a plain list:

week = [320, 280, 410, 300, 350, 390, 450]
busy = []
for c in week:
    if c > 400:
        busy.append(c)
print(busy)
print(week > 400)
[410, 450]
Traceback (most recent call last):
  File "main.py", line 7, in <module>
    print(week > 400)
          ^^^^^^^^^^
TypeError: '>' not supported between instances of 'list' and 'int'

On a NumPy array, comparisons are element-wise (applied to each element, one by one, as in episode 8):

import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week > 400)
print(week[week > 400])
big = week > 400
print(big.sum(), "busy days")
[False False  True False False False  True]
[410 450]
2 busy days
  • week > 400 is a comparison array: a boolean array (boolean = True or False) with one answer per element.
  • Put a boolean array inside the brackets and NumPy keeps only the positions marked True. That is a boolean mask.
  • True counts as 1 and False as 0, so mask.sum() counts the matches.

Real-life example: a stencil. A card with holes cut in it. The True/False array is the card (True = a hole); laid on the data, you only see the values under the holes.

Masks work on 2-D arrays too; the result is a flat 1-D array of the matching values:

print(cups > 440)
print(cups[cups > 440])
[[False False False False False False  True]
 [False False False False False False  True]
 [False False False False False False False]
 [False False False False False False  True]]
[450 470 480]

All three are Sundays.

Combining conditions: &, |, ~ and the parentheses rule

Normal days: more than 300 and less than 400. Two wrong ways first.

Without brackets:

import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week[week > 300 & week < 400])
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    print(week[week > 300 & week < 400])
               ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

With the word and:

import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week[(week > 300) and (week < 400)])
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    print(week[(week > 300) and (week < 400)])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Why: and, or and not need one True or False. An array holds many, so Python can't decide whether the whole array is "true" (ambiguous = it could mean two things). And Python evaluates & before > and <, so without brackets week > 300 & week < 400 is read as week > (300 & week) < 400, which again needs a single True/False.

The fix: NumPy's element-wise operators, with brackets around every condition:

Operator Means Example
& (ampersand) and, cell by cell (week > 300) & (week < 400)
| (pipe) or, cell by cell (week < 300) | (week > 400)
~ (tilde) not: flips each True/False ~(week > 400)
import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week[(week > 300) & (week < 400)])
print(week[(week < 300) | (week > 400)])
print(week[~(week > 400)])
[320 350 390]
[280 410 450]
[320 280 300 350 390]

.any() (is at least one True?) and .all() (are all True?), the methods the error suggests, are for when you really want a single answer about the whole array.

Fancy indexing: a list of positions

Just Monday and Sunday, positions 0 and 6: not a range, not a condition. Pass a list of positions inside the brackets. That is fancy indexing. Positions can repeat and be in any order.

week = cups[0]
print(week[[0, 6]])
print(week[[6, 0, 0]])
print(cups[[0, 3]])
[320 450]
[450 320 320]
[[320 280 410 300 350 390 450]
 [330 300 430 320 370 415 480]]

On a 2-D array, a list of row numbers picks whole rows (weeks 1 and 4).

Reshaping: reshape, flatten, transpose

New problem: the billing app exports the month as one long line of 28 numbers (shape (28,)). With no rows or columns, "every Monday" means position arithmetic by hand (0, 7, 14, 21).

Reshaping changes an array's shape without changing its data. The values fill the new shape row by row.

import numpy as np
log = np.array([320, 280, 410, 300, 350, 390, 450,
                310, 295, 380, 330, 360, 420, 470,
                290, 260, 400, 310, 340, 405, 440,
                330, 300, 430, 320, 370, 415, 480])
print(log.shape)
cups = log.reshape(4, 7)
print(cups)
print(cups.shape)
(28,)
[[320 280 410 300 350 390 450]
 [310 295 380 330 360 420 470]
 [290 260 400 310 340 405 440]
 [330 300 430 320 370 415 480]]
(4, 7)

The rule: the new shape must hold exactly the same number of items (the size, from episode 8). np.arange(28) is 0 to 27.

import numpy as np
log = np.arange(28)
print(log.reshape(5, 7))
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    print(log.reshape(5, 7))
          ~~~~~~~~~~~^^^^^^
ValueError: cannot reshape array of size 28 into shape (5,7)

5 x 7 = 35, not 28.

Shortcuts:

  • -1 in a shape means "work this one out for me": reshape(-1, 7) gives (4, 7).
  • flatten() turns any array back into one line (a 1-D copy).
  • Transpose (.T) swaps rows and columns: row 1 becomes column 1. Shape (4, 7) becomes (7, 4).
import numpy as np
log = np.arange(1, 29)
print(log.reshape(-1, 7).shape)
print(log.reshape(2, -1).shape)
small = np.array([[1, 2, 3],
                  [4, 5, 6]])
print(small.flatten())
print(small.T)
print(small.shape, small.T.shape)
(4, 7)
(2, 14)
[1 2 3 4 5 6]
[[1 4]
 [2 5]
 [3 6]]
(2, 3) (3, 2)

Real-life example: an egg tray. Reshape = putting 28 eggs into a 4 x 7 tray, one row at a time (a 5 x 7 tray would have 7 empty holes, so NumPy refuses). Flatten = tipping the tray back into one line. Transpose = turning the tray on its side.

Axis: totals per row or per column

New problem: two questions that sound alike: what did I sell each week? and what did I sell on each weekday (all Mondays together...)?

Real-life example: a school marks register. Rows = students, columns = subjects. The class teacher totals each student by adding across a row; the subject teacher totals each subject by adding down a column.

Each direction of an array is an axis. For a 2-D array, axis 0 runs down the rows and axis 1 runs across the columns; the shape (4, 7) lists them in that order.

  • sum(axis=1): squashes the columns → one number per row (per week).
  • sum(axis=0): squashes the rows → one number per column (per weekday).
  • No axis: one number for the whole array.

Memory trick: the axis you name is the one that disappears.

             axis 1 (columns) →
             Mon  Tue  Wed  Thu  Fri  Sat  Sun    sum(axis=1)
axis 0  wk1  320  280  410  300  350  390  450 →  2500
(rows)  wk2  310  295  380  330  360  420  470 →  2565
  ↓     wk3  290  260  400  310  340  405  440 →  2445
        wk4  330  300  430  320  370  415  480 →  2645
              ↓    ↓    ↓    ↓    ↓    ↓    ↓
sum(axis=0) 1250 1135 1620 1260 1420 1630 1840
print(cups.sum())
print(cups.sum(axis=1))
print(cups.sum(axis=0))
print(cups.mean(axis=0))
10155
[2500 2565 2445 2645]
[1250 1135 1620 1260 1420 1630 1840]
[312.5  283.75 405.   315.   355.   407.5  460.  ]

10,155 cups this month; week 4 was best (2,645); Sunday averages 460 cups and Tuesday under 284.

Aggregation: sum, mean, min, max, std

Aggregation turns many numbers into one summary number:

Function Gives Example on the month
sum() total 10155
mean() average (total ÷ count) about 362.7
min() smallest value 260
max() largest value 480
std() standard deviation (spread) about 60.7

All of them accept axis=.

Standard deviation measures spread: roughly, how far the values typically sit from the mean. Two stalls both average 350 cups; one sells exactly 350 every day (std 0), the other swings between 200 and 500 (std about 127.5). Same mean, very different business.

print(cups.min(), cups.max())
print(cups.max(axis=1))
print(round(cups.mean(), 1), round(cups.std(), 1))
steady = np.array([350, 350, 350, 350])
wild = np.array([200, 500, 250, 450])
print(steady.mean(), steady.std())
print(wild.mean(), wild.std().round(1))
260 480
[450 470 440 480]
362.7 60.7
350.0 0.0
350.0 127.5

How it is computed: subtract the mean from each value, square the differences, average them (that average is the variance), then take the square root. NumPy's std() divides by the number of values n (the population formula, ddof=0). The sample version divides by n - 1 (std(ddof=1), and pandas' default); it comes back in the statistics episodes.

argmax and argmin: which position?

max() says the best number was 450, but Ramesh wants to know which day. argmax() returns the position of the largest value; argmin() the position of the smallest. Then use the position to index a labels array (a label is a human-readable name for a position, like "Sun").

import numpy as np
days = np.array(["Mon", "Tue", "Wed", "Thu",
                 "Fri", "Sat", "Sun"])
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week.max(), week.argmax())
print("Best day:", days[week.argmax()])
print("Worst day:", days[week.argmin()])
450 6
Best day: Sun
Worst day: Tue

On a 2-D array:

print(cups.argmax())
print(cups.argmax(axis=1))
print(cups.argmin(axis=0))
27
[6 6 6 6]
[2 2 1 0 2 0 2]
  • No axis: the position as if the array were flattened (27 = the last cell).
  • axis=1: each week's best day (column 6, Sunday, every week).
  • axis=0: for each weekday, the weakest week (row number).

np.where: labels, positions and replacing values

np.where(condition, value_if_true, value_if_false) is the array version of episode 6's conditional expression. With only a condition, it returns the positions where it is True.

import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
labels = np.where(week >= 400, "busy", "ok")
print(labels)
print(np.where(week >= 400))
print(np.where(week < 300, 300, week))
['ok' 'ok' 'busy' 'ok' 'ok' 'ok' 'busy']
(array([2, 6]),)
[320 300 410 300 350 390 450]

The last line replaces every value under 300 with 300 and keeps the rest: a common cleaning step (capping).

np.sort and np.unique

  • np.sort(a) returns a sorted copy (smallest first); the original is unchanged. [::-1] reverses it (largest first).
  • np.unique(a) returns each distinct value once, sorted. With return_counts=True it also returns how many times each appears: a frequency table (pandas calls this value_counts, episode 11).
import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(np.sort(week))
print(np.sort(week)[::-1])
print(week)
snacks = np.array(["samosa", "chai", "chai", "bun",
                   "chai", "samosa", "chai"])
print(np.unique(snacks))
names, counts = np.unique(snacks, return_counts=True)
print(names, counts)
[280 300 320 350 390 410 450]
[450 410 390 350 320 300 280]
[320 280 410 300 350 390 450]
['bun' 'chai' 'samosa']
['bun' 'chai' 'samosa'] [1 4 2]

Mini project: the monthly report

From the flat export to a full report, without a single loop.

import numpy as np
log = np.array([320, 280, 410, 300, 350, 390, 450,
                310, 295, 380, 330, 360, 420, 470,
                290, 260, 400, 310, 340, 405, 440,
                330, 300, 430, 320, 370, 415, 480])
days = np.array(["Mon", "Tue", "Wed", "Thu",
                 "Fri", "Sat", "Sun"])
cups = log.reshape(4, 7)
weekly = cups.sum(axis=1)
print("Weekly totals:", weekly)
print("Best week:", weekly.argmax() + 1)
avg = cups.mean(axis=0)
print("Best weekday:", days[avg.argmax()])
print("Slowest weekday:", days[avg.argmin()])
print("Busy days (400+):", (cups >= 400).sum())
print("Std per weekday:", cups.std(axis=0).round(1))
Weekly totals: [2500 2565 2445 2645]
Best week: 4
Best weekday: Sun
Slowest weekday: Tue
Busy days (400+): 10
Std per weekday: [14.8 15.6 18.  11.2 11.2 11.5 15.8]

+ 1 because people count weeks from 1. Thursday and Friday have the smallest spread: the most predictable days.

Common mistakes

Code Real error Why Fix
cups[4, 0] (only 4 rows) IndexError: index 4 is out of bounds for axis 0 with size 4 Positions start at 0: rows are 0-3 cups[3, 0] or cups[-1, 0]
week > 400 on a plain list TypeError: '>' not supported between instances of 'list' and 'int' Lists don't compare element-wise np.array(week) > 400
week[week > 300 & week < 400] ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() & runs before > and < week[(week > 300) & (week < 400)]
week[(week > 300) and (week < 400)] ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() and needs one True/False Use &, |, ~
np.arange(28).reshape(5, 7) ValueError: cannot reshape array of size 28 into shape (5,7) New shape must hold the same count reshape(4, 7) or reshape(-1, 7)
cups.sum(axis=0) for weekly totals (no error: 7 numbers instead of 4) The named axis disappears cups.sum(axis=1)
week.max() to find the best day (no error: a value, not a day) max gives the value days[week.argmax()]

Practice

Use this step-count array (3 weeks x 7 days):

import numpy as np
steps = np.array([[6200, 4800, 9100, 7500, 3900, 11000, 8600],
                  [5400, 7200, 8800, 4100, 6600, 12500, 9900],
                  [7000, 4500, 9300, 6800, 5200, 10400, 4700]])
days = np.array(["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"])

1. Total steps for each week, and the best weekday on average.

Solution
print(steps.sum(axis=1))
print(days[steps.mean(axis=0).argmax()])
[51100 54500 47900]
Sat

2. List the days under 5,000 steps and count them, two ways.

Solution
low = steps[steps < 5000]
print(low, len(low), (steps < 5000).sum())
[4800 3900 4100 4500 4700] 5 5

3. Label every day "active" (8,000+) or "rest".

Solution
print(np.where(steps >= 8000, "active", "rest"))
[['rest' 'rest' 'active' 'rest' 'rest' 'active' 'active']
 ['rest' 'rest' 'active' 'rest' 'rest' 'active' 'active']
 ['rest' 'rest' 'active' 'rest' 'rest' 'active' 'rest']]

4. With week = np.array([320, 280, 410, 300, 350, 390, 450]): the days from 300 to 350 inclusive, the 3 largest values, and their positions (largest first).

Solution
print(week[(week >= 300) & (week <= 350)])
print(np.sort(week)[-3:])
print(week.argsort()[::-1][:3])
[320 300 350]
[390 410 450]
[6 2 5]

argsort() returns the positions that would sort the array.

5. On the month cups: total weekend cups (Sat + Sun), the shape of the transpose, and the weeks with a total over 2,500.

Solution
print(cups[:, 5:].sum())
print(cups.T.shape)
print(cups[cups.sum(axis=1) > 2500])
3470
(7, 4)
[[310 295 380 330 360 420 470]
 [330 300 430 320 370 415 480]]

The last one is a boolean mask over rows: one True/False per week.

Quick quiz

a = np.array([[1, 2, 3],
              [4, 5, 6]])
print(a.sum(axis=0))

A. [ 6 15] B. [5 7 9] C. 21 D. [1 4]

Answer

B. axis=0 squashes the rows: one total per column (1+4, 2+5, 3+6). [ 6 15] would be axis=1, and 21 is no axis.

Interview questions

  1. What is a boolean mask? A True/False array of the same shape as the data; indexing with it keeps only the True positions (a[a > 400]).
  2. Why does a[a > 1 and a < 5] fail? and needs a single True/False, but an array comparison gives many: ValueError: The truth value of an array ... is ambiguous. Use (a > 1) & (a < 5).
  3. Why are parentheses needed with &? & has higher precedence than >/<, so a > 1 & a < 5 is evaluated as a > (1 & a) < 5.
  4. What does axis=0 mean in sum? Aggregate down the rows (axis 0 disappears): one result per column.
  5. argmax vs max? max returns the largest value; argmax returns its position (flattened position if no axis).
  6. What does reshape(-1, 7) do? Makes 7 columns and works out the number of rows; the total size must divide evenly.
  7. np.where(cond) vs np.where(cond, x, y)? One argument: the positions where cond is True. Three: a new array with x where True, y where False.
  8. Does slicing copy the data? Basic slicing returns a view (it shares memory with the original); boolean masks and fancy indexing return copies. Episode 10 covers copy vs view in detail.

Glossary

Term Meaning Example
Grid / table Numbers arranged in rows and columns 4 weeks x 7 days
Row One line going across week 2: cups[1]
Column One line going down every Monday: cups[:, 0]
1-D array An array with one direction, like a list np.array([320, 280])
2-D array An array with rows and columns cups, shape (4, 7)
Shape Size in each direction (4, 7)
Size Total number of items 28
Index / position An item's place number, counted from 0 Wednesday = 2
Indexing Getting a value by its position cups[1, 2] → 380
Negative index Counts from the end cups[-1, -1] → 480
Out of bounds A position outside the array cups[4, 0] → IndexError
Slicing Taking a range of positions, start:stop:step (stop excluded) cups[0:2, 5:7]
: (alone) All rows or all columns cups[:, 0]
Element-wise Applied to each element, one by one week > 400
Comparison array Result of comparing an array: one True/False per element [False False True ...]
Boolean A True-or-False value True
Boolean mask A True/False array used inside [] to keep the True positions week[week > 400]
& / | / ~ Element-wise and / or / not (a > 1) & (a < 5)
Parentheses rule Put brackets around every condition when using &, | (week > 300) & (week < 400)
Ambiguous Could mean two things "truth value of an array is ambiguous"
.any() / .all() Is at least one / every element True? (week > 400).any() → True
Fancy indexing Indexing with a list of positions week[[0, 6]]
reshape Same data, new shape (same size) log.reshape(4, 7)
-1 in reshape "Work this dimension out" reshape(-1, 7)
flatten Any array → a 1-D copy small.flatten()
Transpose (.T) Swap rows and columns (4, 7) → (7, 4)
Axis One direction of an array: 0 = down the rows, 1 = across the columns sum(axis=1) → per week
Aggregation Turning many values into one summary value sum, mean
sum Total cups.sum() → 10155
mean Average: total ÷ count cups.mean(axis=0)
min / max Smallest / largest value 260 / 480
Standard deviation (std) How far values typically sit from the mean (spread) steady stall 0, wild stall 127.5
Variance The average squared distance from the mean (std squared) wild.var()
argmax / argmin Position of the largest / smallest value week.argmax() → 6
Label A human-readable name for a position days[6] → "Sun"
np.where Pick values by a condition, or find True positions np.where(week >= 400, "busy", "ok")
np.sort A sorted copy of an array np.sort(week)
np.unique Distinct values, sorted (optionally with counts) np.unique(snacks, return_counts=True)
Frequency table Each distinct value with how often it appears bun 1, chai 4, samosa 2
argsort Positions that would sort the array week.argsort()
View An array that shares memory with another (from slicing) cups[1]

Summary

  • Indexing a[row, col] picks one cell (positions from 0, negatives from the end); slicing a[rows, cols] picks blocks, : means all.
  • A comparison gives a boolean array; used inside [] it is a mask that keeps the True cells. mask.sum() counts matches.
  • Combine masks with &, |, ~, with brackets around each condition; and/or raise the "ambiguous" ValueError.
  • Fancy indexing picks a list of positions.
  • reshape changes shape (same size; -1 = work it out), flatten makes it 1-D, .T swaps rows and columns.
  • axis: the axis you name disappears. axis=1 → per row, axis=0 → per column.
  • Aggregation: sum, mean, min, max, std (spread). argmax/argmin give where, not what.
  • np.where labels, finds positions and replaces values; np.sort sorts a copy; np.unique gives distinct values and counts.
  • Next episode: NumPy 3: broadcasting, random numbers and NaN.