These are the notes for Data Analytics with Python, episode 17. In episode 16 we sorted, ranked and created columns with ready-made tools (np.where, np.select). Real businesses have their own rules, though, and no library has a method for them. Here we invent: running your own function on every value (Series.apply) or every row (DataFrame.apply with axis), translating codes with map and replace, changing every cell with DataFrame.map (the old applymap), why vectorised code beats apply, putting numbers into groups (binning with pd.cut and pd.qcut, bins and labels), and writing long analyses as method chains with pipe() in a readable pandas style. Every term is defined the first time it appears and collected in the Glossary. Every program below was run for real (Python 3.13, pandas 3.0.5, NumPy 2.5.3, pyarrow 26.0.0); outputs are unedited.
The examples use Ramesh's week of chai sales. The first line only hides pandas' long internal traceback so errors are short:
import sys; sys.tracebacklimit = 0
import pandas as pd
week = pd.DataFrame({
"day": ["Mon", "Tue", "Wed", "Thu",
"Fri", "Sat", "Sun"],
"cups": [320, 280, 410, 300, 350, 390, 450],
"price": [10, 10, 12, 10, 10, 12, 12],
"weather": ["sunny", "rain", "cold", "sunny",
"sunny", "cold", "rain"],
})
The problem: Ramesh's own rule
On a day with 400 or more cups, Ramesh calls in a helper: a rush day. Everything else is calm. A rule like this goes in a function (episode 7). It works for one number, so the obvious move is to hand it the whole column:
def tag(cups):
if cups >= 400:
return "rush"
return "calm"
print(tag(450))
week["tag"] = tag(week["cups"])
rush
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
Inside the function, cups is now a whole Series of seven numbers, and if needs exactly one True/False.
Series.apply: your function on every value
Series.apply(func) calls func once for every value in the Series and collects the answers into a new Series with the same index:
def tag(cups):
if cups >= 400:
return "rush"
return "calm"
week["tag"] = week["cups"].apply(tag)
print(week[["day", "cups", "tag"]])
day cups tag
0 Mon 320 calm
1 Tue 280 calm
2 Wed 410 rush
3 Thu 300 calm
4 Fri 350 calm
5 Sat 390 calm
6 Sun 450 rush
Pass the function without brackets (apply(tag), not apply(tag())): you hand over the function itself, and apply does the calling. A lambda works too: week["cups"].apply(lambda c: "rush" if c >= 400 else "calm"). (That one-line a if cond else b is the conditional expression from episode 6.)
DataFrame.apply and axis
Axis is the direction you move through a table. Think of a marks register: you can total each subject (going down a column) or each student (going across a row).
| axis | Direction | df.apply(f, axis=...) gives f … |
Result |
|---|---|---|---|
0 (default, "index") |
down | one column at a time | one value per column |
1 ("columns") |
across | one row at a time | one value per row |
nums = week[["cups", "price"]]
print(nums.apply(max))
print(nums.apply(max, axis=1).tolist())
cups 450
price 12
dtype: int64
[320, 280, 410, 300, 350, 390, 450]
With axis=1, each row arrives as a Series whose index is the column names, so inside the function you read values with row["col"]. That is how to apply a rule that needs several columns. Here the bill is cups × price, plus ₹200 for umbrella delivery on rainy days:
def bill(row):
total = row["cups"] * row["price"]
if row["weather"] == "rain":
total = total + 200
return total
week["bill"] = week.apply(bill, axis=1)
print(week[["day", "weather", "bill"]])
day weather bill
0 Mon sunny 3200
1 Tue rain 3000
2 Wed cold 4920
3 Thu sunny 3000
4 Fri sunny 3500
5 Sat cold 4680
6 Sun rain 5600
(The vectorised version, much faster on big data, is week["cups"] * week["price"] + np.where(week["weather"] == "rain", 200, 0).)
Translating codes: map and replace
The billing app stores payment types as one letter. A codebook (a dictionary from code to name) translates them. Series.map(dict) looks up every value in the dictionary; Series.replace(dict) only swaps the values it finds and leaves the rest:
week["pay"] = ["U", "C", "U", "K", "U", "C", "X"]
names = {"U": "UPI", "C": "Cash", "K": "Card"}
week["pay_name"] = week["pay"].map(names)
week["pay_fix"] = week["pay"].replace(names)
print(week[["day", "pay", "pay_name", "pay_fix"]])
day pay pay_name pay_fix
0 Mon U UPI UPI
1 Tue C Cash Cash
2 Wed U UPI UPI
3 Thu K Card Card
4 Fri U UPI UPI
5 Sat C Cash Cash
6 Sun X NaN X
| Value not in the dict | Use it for | |
|---|---|---|
map(dict) |
becomes NaN | translating a whole column; then isna() finds every unknown code |
replace(dict) |
kept as it was | fixing a few known wrong values ({"upi": "UPI"}) |
map also accepts a function (s.map(str.upper)) or another Series used as a lookup table. map(names, na_action="ignore") leaves existing NaNs alone instead of passing them to a function.
Every cell: DataFrame.map (formerly applymap)
To change every cell of a table, older tutorials use applymap. pandas 2.1 renamed it to DataFrame.map and deprecated the old name (pandas 2.1 release notes), and in pandas 3 it is gone:
print(week[["cups"]].applymap(str))
AttributeError: 'DataFrame' object has no attribute 'applymap'
money = pd.DataFrame({"revenue": week["cups"] * week["price"],
"cost": week["cups"] * 4 + 500})
print(money.map(lambda v: f"Rs {v:,}"))
revenue cost
0 Rs 3,200 Rs 1,780
1 Rs 2,800 Rs 1,620
2 Rs 4,920 Rs 2,140
3 Rs 3,000 Rs 1,700
4 Rs 3,500 Rs 1,900
5 Rs 4,680 Rs 2,060
6 Rs 5,400 Rs 2,300
f"{v:,}" is an f-string (episode 4) with the format code ,, which adds thousands separators. The result is text: format only at the very end, for display, never before calculations.
| Method | Works on | Function receives |
|---|---|---|
Series.map |
one column | each value (or a dict / Series lookup) |
Series.apply |
one column | each value |
DataFrame.apply(axis=0) |
a table | each column (a Series) |
DataFrame.apply(axis=1) |
a table | each row (a Series) |
DataFrame.map |
a table | each single cell |
Speed: apply vs vectorised
apply is a loop in disguise: it calls a Python function once per value. Vectorised operations (episode 8) run over the whole column in fast compiled code. 2 lakh (200,000) orders, add 18% GST both ways:
import pandas as pd, numpy as np, time
rng = np.random.default_rng(1)
orders = pd.DataFrame({"amt": rng.integers(10, 1000, 200_000)})
t = time.time()
a = orders["amt"].apply(lambda v: v * 1.18)
print("apply: ", round(time.time() - t, 4), "s")
t = time.time()
b = orders["amt"] * 1.18
print("vectorised:", round(time.time() - t, 4), "s")
print(a.equals(b))
apply: 0.038 s
vectorised: 0.0012 s
True
Identical results (True), and the vectorised line is many times faster. Exact times change from run to run and machine to machine; the gap does not. You can measure it yourself in Jupyter with %timeit (episode 29). Rule: column maths, np.where, np.select, .str methods, cut first; apply only for rules with no vectorised version.
Binning with pd.cut
Binning means putting numbers into groups (bins), like clothes sizes: chest sizes up to one mark are S, then M, then L. pd.cut(values, bins=edges, labels=names) uses the edges you choose; n+1 edges make n bins, and you need one label per bin:
week["level"] = pd.cut(week["cups"], bins=[0, 300, 400, 500],
labels=["low", "medium", "high"])
print(week[["day", "cups", "level"]])
print(week["level"].value_counts(sort=False))
day cups level
0 Mon 320 medium
1 Tue 280 low
2 Wed 410 high
3 Thu 300 low
4 Fri 350 medium
5 Sat 390 medium
6 Sun 450 high
level
low 2
medium 3
high 2
Name: count, dtype: int64
The result has the category dtype (a fixed set of labels, in order; episode 20). value_counts(sort=False) keeps the bins in their natural order instead of sorting by count.
Edges: which side is included?
print(pd.cut(pd.Series([0, 300, 301, 520]), bins=[0, 300, 400, 500]))
0 NaN
1 (0.0, 300.0]
2 (300.0, 400.0]
3 NaN
dtype: category
Categories (3, interval[int64, right]): [(0, 300] < (300, 400] < (400, 500]]
An interval is written like (0, 300]: a round bracket means that end is excluded, a square bracket means included. By default each bin includes its right edge (right=True), so 300 is in (0, 300] but 0 is not in any bin, and 520 is past the last edge: both become NaN, with no error. Fixes: include_lowest=True to include the very first edge, right=False for [a, b) bins, and edges like [0, 300, 400, float("inf")] for "and above".
pd.cut(values, bins=4) with a number instead of edges makes 4 bins of equal width between the min and max.
Equal-sized groups with pd.qcut
Sometimes you want groups with the same number of rows, like splitting a class into three equal groups. A quantile is a value that splits sorted data into equal parts (the median is the 50% quantile; episode 35). pd.qcut(values, q=3) picks the edges at the quantiles for you:
week["band"] = pd.qcut(week["cups"], q=3, labels=["bottom", "middle", "top"])
print(week[["day", "cups", "band"]])
print(week["band"].value_counts(sort=False))
day cups band
0 Mon 320 bottom
1 Tue 280 bottom
2 Wed 410 top
3 Thu 300 bottom
4 Fri 350 middle
5 Sat 390 middle
6 Sun 450 top
band
bottom 3
middle 2
top 2
Name: count, dtype: int64
Seven rows can't split into three equal groups, so qcut gets as close as it can (3, 2, 2). q=4 gives quartiles, q=10 deciles, and q=[0, 0.5, 0.9, 1] custom quantiles. With many repeated values, qcut can't find distinct edges:
print(pd.qcut(week["price"], q=3))
ValueError: Bin edges must be unique: Index([10.0, 10.0, 12.0, 12.0], dtype='float64', name='price').
You can drop duplicate edges by setting the 'duplicates' kwarg
Fixes: fewer groups, pd.cut with your own edges, or duplicates="drop" (which merges the duplicate edges into fewer bins).
pd.cut |
pd.qcut |
|
|---|---|---|
| Edges | you choose (or equal width) | from the data's quantiles |
| Group sizes | can be very different | (roughly) equal counts |
| Use for | business rules: low < 300 < medium | "top 10% of customers" |
Method chaining and pipe()
Real analysis is many steps. Written as nested calls it must be read inside out:
print(pd.cut(week.assign(rev=week["cups"] *
week["price"]).query("weather != 'rain'")["rev"],
bins=[0, 3500, 6000], labels=["ok", "great"]).value_counts())
rev
ok 3
great 2
Name: count, dtype: int64
Method chaining: every pandas method returns a table, so call the next method on it directly, one step per line, like a recipe card. A line break normally ends a Python statement:
summary = week
.query("weather != 'rain'")
File "main.py", line 12
.query("weather != 'rain'")
IndentationError: unexpected indent
Wrap the whole chain in round brackets: inside brackets Python keeps reading across lines. For your own steps, df.pipe(func, *args, **kwargs) calls func(df, *args, **kwargs), so your functions fit into the chain:
def add_revenue(df, rate=1):
return df.assign(rev=df["cups"] * df["price"] * rate)
summary = (
week
.pipe(add_revenue)
.query("weather != 'rain'")
.assign(level=lambda d: pd.cut(d["rev"], bins=[0, 3500, 6000],
labels=["ok", "great"]))
.sort_values("rev", ascending=False)
[["day", "rev", "level"]]
)
print(summary)
day rev level
2 Wed 4920 great
5 Sat 4680 great
4 Fri 3500 ok
0 Mon 3200 ok
3 Thu 3000 ok
def add_revenue(df, rate=1):
return df.assign(rev=df["cups"] * df["price"] * rate)
print(week.pipe(add_revenue, rate=0.95)[["day", "rev"]].head(3))
day rev
0 Mon 3040.0
1 Tue 2660.0
2 Wed 4674.0
Inside a chain, use lambda d: ... in assign (and in loc[...]) to refer to the table at that step, not the original week.
Readable pandas style: a checklist
- Vectorised first;
applyonly for custom rules that have no vectorised version. mapfor codebooks, then checkisna()for unknown codes.cutfor your own edges,qcutfor equal-sized groups; check for NaN after binning.- Chains in
( ), one step per line, starting each line with the dot. - Your own steps as small named functions, joined with
pipe. - Don't overwrite the original in every step; build a new name (
summary = (...)). - Format numbers as text only at the very end.
Common mistakes
| Code | Real error / symptom | Why | Fix |
|---|---|---|---|
tag(week["cups"]) |
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). |
if inside the function got a whole column |
week["cups"].apply(tag) |
week["cups"].apply(tag()) |
TypeError: tag() missing 1 required positional argument: 'cups' |
Brackets call the function immediately | apply(tag) |
df.applymap(str) |
AttributeError: 'DataFrame' object has no attribute 'applymap' |
Removed in pandas 3 | df.map(str) |
pay.map(codes) with an unknown code |
(no error: NaN) | map has no fallback |
Check isna(), or replace |
pd.cut(..., bins=[0, 300, 400, 500]) on 0 or 520 |
(no error: NaN) | Outside the bins / left edge excluded | include_lowest=True, wider edges |
pd.cut with 3 bins and 2 labels |
ValueError: Bin labels must be one fewer than the number of bin edges |
n+1 edges → n labels | One label per bin |
pd.qcut(week["price"], q=3) |
ValueError: Bin edges must be unique: Index([10.0, 10.0, 12.0, 12.0], dtype='float64', name='price'). |
Too many ties | Fewer q, cut, or duplicates="drop" |
x = df then .query(...) on the next line |
IndentationError: unexpected indent |
The line break ended the statement | Wrap the chain in ( ) |
apply on 1,000,000 rows |
(no error: slow) | A hidden Python loop | Vectorised maths |
Practice
All practice questions use this table:
import pandas as pd, numpy as np
marks = pd.DataFrame({
"name": ["Asha", "Ravi", "Meena", "Kabir", "Zoya", "Dev"],
"maths": [81, 67, 92, 74, 88, 67],
"science": [75, 80, 95, 60, 70, 85],
"city": ["Pune", "Agra", "Pune", "Delhi", "Agra", "Pune"],
})
1. Write a function initials(name) returning the first two letters in capitals, and apply it to the name column.
Solution
def initials(name):
return name[:2].upper()
print(marks["name"].apply(initials).tolist())
['AS', 'RA', 'ME', 'KA', 'ZO', 'DE']
The vectorised version is marks["name"].str[:2].str.upper().
2. With apply(axis=1), add a column best holding the name of each student's better subject ("maths" or "science").
Solution
marks["best"] = marks.apply(
lambda r: "maths" if r["maths"] >= r["science"] else "science", axis=1)
print(marks[["name", "maths", "science", "best"]])
name maths science best
0 Asha 81 75 maths
1 Ravi 67 80 science
2 Meena 92 95 science
3 Kabir 74 60 maths
4 Zoya 88 70 maths
5 Dev 67 85 science
Vectorised: np.where(marks["maths"] >= marks["science"], "maths", "science").
3. Map the cities to states with {"Pune": "MH", "Agra": "UP"}. Which row becomes NaN, and how do you count such rows?
Solution
marks["state"] = marks["city"].map({"Pune": "MH", "Agra": "UP"})
print(marks[["name", "city", "state"]])
print(marks["state"].isna().sum(), "city not in the codebook")
name city state
0 Asha Pune MH
1 Ravi Agra UP
2 Meena Pune MH
3 Kabir Delhi NaN
4 Zoya Agra UP
5 Dev Pune MH
1 city not in the codebook
4. Bin maths into grades: below 70 → "C", 70 to 84 → "B", 85 and above → "A", and count each grade. Then split the students into two equal halves by maths with qcut.
Solution
marks["grade"] = pd.cut(marks["maths"], bins=[0, 69, 84, 100], labels=["C", "B", "A"])
print(marks[["name", "maths", "grade"]])
print(marks["grade"].value_counts(sort=False))
print(pd.qcut(marks["maths"], q=2, labels=["lower", "upper"]).tolist())
name maths grade
0 Asha 81 B
1 Ravi 67 C
2 Meena 92 A
3 Kabir 74 B
4 Zoya 88 A
5 Dev 67 C
grade
C 2
B 2
A 2
Name: count, dtype: int64
['upper', 'lower', 'upper', 'lower', 'upper', 'lower']
Whole-number marks, so edges 69 and 84 with the right edge included give "below 70", "70-84" and "85+".
5. Write one chain: add total with a function add_total(df) through pipe, keep students with a total of 150 or more, add rank (biggest first, method="min", as int) and sort by rank.
Solution
def add_total(df):
return df.assign(total=df["maths"] + df["science"])
top = (
marks
.pipe(add_total)
.query("total >= 150")
.assign(rank=lambda d: d["total"].rank(ascending=False, method="min").astype(int))
.sort_values("rank")
[["name", "total", "rank"]]
)
print(top)
name total rank
2 Meena 187 1
4 Zoya 158 2
0 Asha 156 3
5 Dev 152 4
Quick quiz
bins = [0, 300, 400, 500]
lab = ["low", "medium", "high"]
x = pd.cut([400], bins, labels=lab)
print(x[0])
A. medium B. high C. NaN D. low
Answer
A. Bins include their right edge by default, so (300, 400] contains 400. The real output:
import pandas as pd
bins = [0, 300, 400, 500]
lab = ["low", "medium", "high"]
x = pd.cut([400], bins, labels=lab)
print(x[0])
medium
With right=False, 400 would fall in [400, 500): high.
Interview questions
mapvsapplyvsDataFrame.map?Series.mapmaps each value through a dict, Series or function;Series.applycalls a function per value;DataFrame.applycalls a function per column (axis=0) or per row (axis=1);DataFrame.mapcalls a function on every cell.- What does
axis=1mean inapply? The function receives each row (across the columns), and returns one value per row. - Why is
applyslow, and what do you use instead? It runs Python code once per element; vectorised operations (+,np.where,np.select,.str,cut) run in compiled code over whole arrays. map(dict)vsreplace(dict)? Unmatched values become NaN withmapand stay unchanged withreplace.- What happened to
applymap? Renamed toDataFrame.mapin pandas 2.1 (deprecated), removed in pandas 3. cutvsqcut?cutuses fixed edges (or equal-width bins);qcutuses quantiles, giving roughly equal counts per bin.- Why do some values become NaN after
cut? They are outside the edges, or exactly on an excluded edge (the left edge by default). Useinclude_lowest=True,right=Falseor wider edges. - What does "Bin edges must be unique" mean in
qcut? Many repeated values make several quantiles equal; use fewer bins,cut, orduplicates="drop". - What is method chaining, and why use it? Calling methods one after another on returned objects, one step per line in parentheses; it reads top to bottom, avoids temporary variables and accidental changes to the original.
- What does
pipedo?df.pipe(f, *args, **kwargs)isf(df, *args, **kwargs); it lets your own functions sit inside a chain.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Function | A named, reusable block of code (episode 7) | def tag(cups): ... |
| Business rule | A rule from the business, not from pandas | 400+ cups = rush day |
Series.apply |
Call a function on every value of a column | week["cups"].apply(tag) |
DataFrame.apply |
Call a function on every column or every row | week.apply(bill, axis=1) |
| Axis | Direction: 0 = down (per column), 1 = across (per row) | apply(max, axis=1) |
| Row function | A function that receives a whole row | def bill(row): row["cups"] ... |
| Lambda | A small one-line function without a name | lambda v: v * 1.18 |
| Conditional expression | a if condition else b in one line |
"rush" if c >= 400 else "calm" |
| Codebook | A dictionary from codes to names | {"U": "UPI"} |
map (Series) |
Look every value up in a dict/Series/function | pay.map(names) |
replace |
Swap only the values found in a dict; keep the rest | pay.replace(names) |
DataFrame.map |
Call a function on every cell of a table | money.map(str) |
applymap |
Old name of DataFrame.map, removed in pandas 3 |
AttributeError |
| Element-wise | Applied to each single value | DataFrame.map |
| Vectorised | Whole-column operations in fast compiled code | orders["amt"] * 1.18 |
| Hidden loop | Code that looks like one call but loops in Python | apply |
| Binning | Putting numbers into groups (bins) | low / medium / high |
| Bin | One range of values | (300, 400] |
| Bin edges | The boundaries between bins | [0, 300, 400, 500] |
| Labels | Names for the bins (one per bin) | ["low", "medium", "high"] |
pd.cut |
Bin with fixed edges or equal widths | pd.cut(cups, bins=[...]) |
| Interval | A range with included/excluded ends | (0, 300] |
right=True |
Each bin includes its right edge (default) | 300 is in (0, 300] |
include_lowest |
Also include the very first edge | 0 in the first bin |
| Category dtype | A column holding a fixed set of labels | the result of cut |
| Quantile | A value that splits sorted data into equal parts | median = 0.5 quantile |
| Quartile / decile | 4 / 10 equal parts | q=4, q=10 |
pd.qcut |
Bin into groups with equal counts | pd.qcut(cups, q=3) |
duplicates="drop" |
Let qcut merge repeated edges | many ties |
| Method chaining | One method after another, one step per line | (df.query(...).sort_values(...)) |
pipe |
Put your own function into a chain | df.pipe(add_revenue, rate=0.95) |
| Readable pandas style | Habits that make analysis easy to follow | chains, named steps |
Summary
Series.apply(f)runs your function on every value;DataFrame.apply(f, axis=1)on every row (axis=0, the default, on every column).map(dict)translates codes (unknown → NaN);replace(dict)swaps only what matches.DataFrame.map(f)changes every cell;applymapno longer exists in pandas 3.applyis a hidden loop: vectorised first,applylast.pd.cut: bins with your edges (right edge included; outside → NaN).pd.qcut: equal-sized groups from quantiles.- Method chaining in round brackets, one step per line, with
pipefor your own functions, keeps long analysis readable. - Next episode: data quality and profiling before cleaning.