SyntaxPilot
Courses/Data Analytics with Python/Lesson 17

Data Analytics with Python · Lesson 17 of 90 · 14 min

pandas 7: apply, map, binning and method chaining

apply (axis)mapDataFrame.map (applymap)replacevectorised vs apply performancecutqcutbinningbins and labelsmethod chainingpipe()readable pandas style

These are the notes for Data Analytics with Python, episode 17. In episode 16 we sorted, ranked and created columns with ready-made tools (np.where, np.select). Real businesses have their own rules, though, and no library has a method for them. Here we invent: running your own function on every value (Series.apply) or every row (DataFrame.apply with axis), translating codes with map and replace, changing every cell with DataFrame.map (the old applymap), why vectorised code beats apply, putting numbers into groups (binning with pd.cut and pd.qcut, bins and labels), and writing long analyses as method chains with pipe() in a readable pandas style. Every term is defined the first time it appears and collected in the Glossary. Every program below was run for real (Python 3.13, pandas 3.0.5, NumPy 2.5.3, pyarrow 26.0.0); outputs are unedited.

The examples use Ramesh's week of chai sales. The first line only hides pandas' long internal traceback so errors are short:

import sys; sys.tracebacklimit = 0
import pandas as pd
week = pd.DataFrame({
    "day": ["Mon", "Tue", "Wed", "Thu",
            "Fri", "Sat", "Sun"],
    "cups": [320, 280, 410, 300, 350, 390, 450],
    "price": [10, 10, 12, 10, 10, 12, 12],
    "weather": ["sunny", "rain", "cold", "sunny",
                "sunny", "cold", "rain"],
})

The problem: Ramesh's own rule

On a day with 400 or more cups, Ramesh calls in a helper: a rush day. Everything else is calm. A rule like this goes in a function (episode 7). It works for one number, so the obvious move is to hand it the whole column:

def tag(cups):
    if cups >= 400:
        return "rush"
    return "calm"
print(tag(450))
week["tag"] = tag(week["cups"])
rush
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Inside the function, cups is now a whole Series of seven numbers, and if needs exactly one True/False.

Series.apply: your function on every value

Series.apply(func) calls func once for every value in the Series and collects the answers into a new Series with the same index:

def tag(cups):
    if cups >= 400:
        return "rush"
    return "calm"
week["tag"] = week["cups"].apply(tag)
print(week[["day", "cups", "tag"]])
   day  cups   tag
0  Mon   320  calm
1  Tue   280  calm
2  Wed   410  rush
3  Thu   300  calm
4  Fri   350  calm
5  Sat   390  calm
6  Sun   450  rush

Pass the function without brackets (apply(tag), not apply(tag())): you hand over the function itself, and apply does the calling. A lambda works too: week["cups"].apply(lambda c: "rush" if c >= 400 else "calm"). (That one-line a if cond else b is the conditional expression from episode 6.)

DataFrame.apply and axis

Axis is the direction you move through a table. Think of a marks register: you can total each subject (going down a column) or each student (going across a row).

axis Direction df.apply(f, axis=...) gives f … Result
0 (default, "index") down one column at a time one value per column
1 ("columns") across one row at a time one value per row
nums = week[["cups", "price"]]
print(nums.apply(max))
print(nums.apply(max, axis=1).tolist())
cups     450
price     12
dtype: int64
[320, 280, 410, 300, 350, 390, 450]

With axis=1, each row arrives as a Series whose index is the column names, so inside the function you read values with row["col"]. That is how to apply a rule that needs several columns. Here the bill is cups × price, plus ₹200 for umbrella delivery on rainy days:

def bill(row):
    total = row["cups"] * row["price"]
    if row["weather"] == "rain":
        total = total + 200
    return total
week["bill"] = week.apply(bill, axis=1)
print(week[["day", "weather", "bill"]])
   day weather  bill
0  Mon   sunny  3200
1  Tue    rain  3000
2  Wed    cold  4920
3  Thu   sunny  3000
4  Fri   sunny  3500
5  Sat    cold  4680
6  Sun    rain  5600

(The vectorised version, much faster on big data, is week["cups"] * week["price"] + np.where(week["weather"] == "rain", 200, 0).)

Translating codes: map and replace

The billing app stores payment types as one letter. A codebook (a dictionary from code to name) translates them. Series.map(dict) looks up every value in the dictionary; Series.replace(dict) only swaps the values it finds and leaves the rest:

week["pay"] = ["U", "C", "U", "K", "U", "C", "X"]
names = {"U": "UPI", "C": "Cash", "K": "Card"}
week["pay_name"] = week["pay"].map(names)
week["pay_fix"] = week["pay"].replace(names)
print(week[["day", "pay", "pay_name", "pay_fix"]])
   day pay pay_name pay_fix
0  Mon   U      UPI     UPI
1  Tue   C     Cash    Cash
2  Wed   U      UPI     UPI
3  Thu   K     Card    Card
4  Fri   U      UPI     UPI
5  Sat   C     Cash    Cash
6  Sun   X      NaN       X
Value not in the dict Use it for
map(dict) becomes NaN translating a whole column; then isna() finds every unknown code
replace(dict) kept as it was fixing a few known wrong values ({"upi": "UPI"})

map also accepts a function (s.map(str.upper)) or another Series used as a lookup table. map(names, na_action="ignore") leaves existing NaNs alone instead of passing them to a function.

Every cell: DataFrame.map (formerly applymap)

To change every cell of a table, older tutorials use applymap. pandas 2.1 renamed it to DataFrame.map and deprecated the old name (pandas 2.1 release notes), and in pandas 3 it is gone:

print(week[["cups"]].applymap(str))
AttributeError: 'DataFrame' object has no attribute 'applymap'
money = pd.DataFrame({"revenue": week["cups"] * week["price"],
                      "cost": week["cups"] * 4 + 500})
print(money.map(lambda v: f"Rs {v:,}"))
    revenue      cost
0  Rs 3,200  Rs 1,780
1  Rs 2,800  Rs 1,620
2  Rs 4,920  Rs 2,140
3  Rs 3,000  Rs 1,700
4  Rs 3,500  Rs 1,900
5  Rs 4,680  Rs 2,060
6  Rs 5,400  Rs 2,300

f"{v:,}" is an f-string (episode 4) with the format code ,, which adds thousands separators. The result is text: format only at the very end, for display, never before calculations.

Method Works on Function receives
Series.map one column each value (or a dict / Series lookup)
Series.apply one column each value
DataFrame.apply(axis=0) a table each column (a Series)
DataFrame.apply(axis=1) a table each row (a Series)
DataFrame.map a table each single cell

Speed: apply vs vectorised

apply is a loop in disguise: it calls a Python function once per value. Vectorised operations (episode 8) run over the whole column in fast compiled code. 2 lakh (200,000) orders, add 18% GST both ways:

import pandas as pd, numpy as np, time
rng = np.random.default_rng(1)
orders = pd.DataFrame({"amt": rng.integers(10, 1000, 200_000)})
t = time.time()
a = orders["amt"].apply(lambda v: v * 1.18)
print("apply:     ", round(time.time() - t, 4), "s")
t = time.time()
b = orders["amt"] * 1.18
print("vectorised:", round(time.time() - t, 4), "s")
print(a.equals(b))
apply:      0.038 s
vectorised: 0.0012 s
True

Identical results (True), and the vectorised line is many times faster. Exact times change from run to run and machine to machine; the gap does not. You can measure it yourself in Jupyter with %timeit (episode 29). Rule: column maths, np.where, np.select, .str methods, cut first; apply only for rules with no vectorised version.

Binning with pd.cut

Binning means putting numbers into groups (bins), like clothes sizes: chest sizes up to one mark are S, then M, then L. pd.cut(values, bins=edges, labels=names) uses the edges you choose; n+1 edges make n bins, and you need one label per bin:

week["level"] = pd.cut(week["cups"], bins=[0, 300, 400, 500],
                       labels=["low", "medium", "high"])
print(week[["day", "cups", "level"]])
print(week["level"].value_counts(sort=False))
   day  cups   level
0  Mon   320  medium
1  Tue   280     low
2  Wed   410    high
3  Thu   300     low
4  Fri   350  medium
5  Sat   390  medium
6  Sun   450    high
level
low       2
medium    3
high      2
Name: count, dtype: int64

The result has the category dtype (a fixed set of labels, in order; episode 20). value_counts(sort=False) keeps the bins in their natural order instead of sorting by count.

Edges: which side is included?

print(pd.cut(pd.Series([0, 300, 301, 520]), bins=[0, 300, 400, 500]))
0               NaN
1      (0.0, 300.0]
2    (300.0, 400.0]
3               NaN
dtype: category
Categories (3, interval[int64, right]): [(0, 300] < (300, 400] < (400, 500]]

An interval is written like (0, 300]: a round bracket means that end is excluded, a square bracket means included. By default each bin includes its right edge (right=True), so 300 is in (0, 300] but 0 is not in any bin, and 520 is past the last edge: both become NaN, with no error. Fixes: include_lowest=True to include the very first edge, right=False for [a, b) bins, and edges like [0, 300, 400, float("inf")] for "and above".

pd.cut(values, bins=4) with a number instead of edges makes 4 bins of equal width between the min and max.

Equal-sized groups with pd.qcut

Sometimes you want groups with the same number of rows, like splitting a class into three equal groups. A quantile is a value that splits sorted data into equal parts (the median is the 50% quantile; episode 35). pd.qcut(values, q=3) picks the edges at the quantiles for you:

week["band"] = pd.qcut(week["cups"], q=3, labels=["bottom", "middle", "top"])
print(week[["day", "cups", "band"]])
print(week["band"].value_counts(sort=False))
   day  cups    band
0  Mon   320  bottom
1  Tue   280  bottom
2  Wed   410     top
3  Thu   300  bottom
4  Fri   350  middle
5  Sat   390  middle
6  Sun   450     top
band
bottom    3
middle    2
top       2
Name: count, dtype: int64

Seven rows can't split into three equal groups, so qcut gets as close as it can (3, 2, 2). q=4 gives quartiles, q=10 deciles, and q=[0, 0.5, 0.9, 1] custom quantiles. With many repeated values, qcut can't find distinct edges:

print(pd.qcut(week["price"], q=3))
ValueError: Bin edges must be unique: Index([10.0, 10.0, 12.0, 12.0], dtype='float64', name='price').
You can drop duplicate edges by setting the 'duplicates' kwarg

Fixes: fewer groups, pd.cut with your own edges, or duplicates="drop" (which merges the duplicate edges into fewer bins).

pd.cut pd.qcut
Edges you choose (or equal width) from the data's quantiles
Group sizes can be very different (roughly) equal counts
Use for business rules: low < 300 < medium "top 10% of customers"

Method chaining and pipe()

Real analysis is many steps. Written as nested calls it must be read inside out:

print(pd.cut(week.assign(rev=week["cups"] *
    week["price"]).query("weather != 'rain'")["rev"],
    bins=[0, 3500, 6000], labels=["ok", "great"]).value_counts())
rev
ok       3
great    2
Name: count, dtype: int64

Method chaining: every pandas method returns a table, so call the next method on it directly, one step per line, like a recipe card. A line break normally ends a Python statement:

summary = week
    .query("weather != 'rain'")
  File "main.py", line 12
    .query("weather != 'rain'")
IndentationError: unexpected indent

Wrap the whole chain in round brackets: inside brackets Python keeps reading across lines. For your own steps, df.pipe(func, *args, **kwargs) calls func(df, *args, **kwargs), so your functions fit into the chain:

def add_revenue(df, rate=1):
    return df.assign(rev=df["cups"] * df["price"] * rate)

summary = (
    week
    .pipe(add_revenue)
    .query("weather != 'rain'")
    .assign(level=lambda d: pd.cut(d["rev"], bins=[0, 3500, 6000],
                                   labels=["ok", "great"]))
    .sort_values("rev", ascending=False)
    [["day", "rev", "level"]]
)
print(summary)
   day   rev  level
2  Wed  4920  great
5  Sat  4680  great
4  Fri  3500     ok
0  Mon  3200     ok
3  Thu  3000     ok
def add_revenue(df, rate=1):
    return df.assign(rev=df["cups"] * df["price"] * rate)
print(week.pipe(add_revenue, rate=0.95)[["day", "rev"]].head(3))
   day     rev
0  Mon  3040.0
1  Tue  2660.0
2  Wed  4674.0

Inside a chain, use lambda d: ... in assign (and in loc[...]) to refer to the table at that step, not the original week.

Readable pandas style: a checklist

  1. Vectorised first; apply only for custom rules that have no vectorised version.
  2. map for codebooks, then check isna() for unknown codes.
  3. cut for your own edges, qcut for equal-sized groups; check for NaN after binning.
  4. Chains in ( ), one step per line, starting each line with the dot.
  5. Your own steps as small named functions, joined with pipe.
  6. Don't overwrite the original in every step; build a new name (summary = (...)).
  7. Format numbers as text only at the very end.

Common mistakes

Code Real error / symptom Why Fix
tag(week["cups"]) ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). if inside the function got a whole column week["cups"].apply(tag)
week["cups"].apply(tag()) TypeError: tag() missing 1 required positional argument: 'cups' Brackets call the function immediately apply(tag)
df.applymap(str) AttributeError: 'DataFrame' object has no attribute 'applymap' Removed in pandas 3 df.map(str)
pay.map(codes) with an unknown code (no error: NaN) map has no fallback Check isna(), or replace
pd.cut(..., bins=[0, 300, 400, 500]) on 0 or 520 (no error: NaN) Outside the bins / left edge excluded include_lowest=True, wider edges
pd.cut with 3 bins and 2 labels ValueError: Bin labels must be one fewer than the number of bin edges n+1 edges → n labels One label per bin
pd.qcut(week["price"], q=3) ValueError: Bin edges must be unique: Index([10.0, 10.0, 12.0, 12.0], dtype='float64', name='price'). Too many ties Fewer q, cut, or duplicates="drop"
x = df then .query(...) on the next line IndentationError: unexpected indent The line break ended the statement Wrap the chain in ( )
apply on 1,000,000 rows (no error: slow) A hidden Python loop Vectorised maths

Practice

All practice questions use this table:

import pandas as pd, numpy as np
marks = pd.DataFrame({
    "name": ["Asha", "Ravi", "Meena", "Kabir", "Zoya", "Dev"],
    "maths": [81, 67, 92, 74, 88, 67],
    "science": [75, 80, 95, 60, 70, 85],
    "city": ["Pune", "Agra", "Pune", "Delhi", "Agra", "Pune"],
})

1. Write a function initials(name) returning the first two letters in capitals, and apply it to the name column.

Solution
def initials(name):
    return name[:2].upper()
print(marks["name"].apply(initials).tolist())
['AS', 'RA', 'ME', 'KA', 'ZO', 'DE']

The vectorised version is marks["name"].str[:2].str.upper().

2. With apply(axis=1), add a column best holding the name of each student's better subject ("maths" or "science").

Solution
marks["best"] = marks.apply(
    lambda r: "maths" if r["maths"] >= r["science"] else "science", axis=1)
print(marks[["name", "maths", "science", "best"]])
    name  maths  science     best
0   Asha     81       75    maths
1   Ravi     67       80  science
2  Meena     92       95  science
3  Kabir     74       60    maths
4   Zoya     88       70    maths
5    Dev     67       85  science

Vectorised: np.where(marks["maths"] >= marks["science"], "maths", "science").

3. Map the cities to states with {"Pune": "MH", "Agra": "UP"}. Which row becomes NaN, and how do you count such rows?

Solution
marks["state"] = marks["city"].map({"Pune": "MH", "Agra": "UP"})
print(marks[["name", "city", "state"]])
print(marks["state"].isna().sum(), "city not in the codebook")
    name   city state
0   Asha   Pune    MH
1   Ravi   Agra    UP
2  Meena   Pune    MH
3  Kabir  Delhi   NaN
4   Zoya   Agra    UP
5    Dev   Pune    MH
1 city not in the codebook

4. Bin maths into grades: below 70 → "C", 70 to 84 → "B", 85 and above → "A", and count each grade. Then split the students into two equal halves by maths with qcut.

Solution
marks["grade"] = pd.cut(marks["maths"], bins=[0, 69, 84, 100], labels=["C", "B", "A"])
print(marks[["name", "maths", "grade"]])
print(marks["grade"].value_counts(sort=False))
print(pd.qcut(marks["maths"], q=2, labels=["lower", "upper"]).tolist())
    name  maths grade
0   Asha     81     B
1   Ravi     67     C
2  Meena     92     A
3  Kabir     74     B
4   Zoya     88     A
5    Dev     67     C
grade
C    2
B    2
A    2
Name: count, dtype: int64
['upper', 'lower', 'upper', 'lower', 'upper', 'lower']

Whole-number marks, so edges 69 and 84 with the right edge included give "below 70", "70-84" and "85+".

5. Write one chain: add total with a function add_total(df) through pipe, keep students with a total of 150 or more, add rank (biggest first, method="min", as int) and sort by rank.

Solution
def add_total(df):
    return df.assign(total=df["maths"] + df["science"])

top = (
    marks
    .pipe(add_total)
    .query("total >= 150")
    .assign(rank=lambda d: d["total"].rank(ascending=False, method="min").astype(int))
    .sort_values("rank")
    [["name", "total", "rank"]]
)
print(top)
    name  total  rank
2  Meena    187     1
4   Zoya    158     2
0   Asha    156     3
5    Dev    152     4

Quick quiz

bins = [0, 300, 400, 500]
lab = ["low", "medium", "high"]
x = pd.cut([400], bins, labels=lab)
print(x[0])

A. medium B. high C. NaN D. low

Answer

A. Bins include their right edge by default, so (300, 400] contains 400. The real output:

import pandas as pd
bins = [0, 300, 400, 500]
lab = ["low", "medium", "high"]
x = pd.cut([400], bins, labels=lab)
print(x[0])
medium

With right=False, 400 would fall in [400, 500): high.

Interview questions

  1. map vs apply vs DataFrame.map? Series.map maps each value through a dict, Series or function; Series.apply calls a function per value; DataFrame.apply calls a function per column (axis=0) or per row (axis=1); DataFrame.map calls a function on every cell.
  2. What does axis=1 mean in apply? The function receives each row (across the columns), and returns one value per row.
  3. Why is apply slow, and what do you use instead? It runs Python code once per element; vectorised operations (+, np.where, np.select, .str, cut) run in compiled code over whole arrays.
  4. map(dict) vs replace(dict)? Unmatched values become NaN with map and stay unchanged with replace.
  5. What happened to applymap? Renamed to DataFrame.map in pandas 2.1 (deprecated), removed in pandas 3.
  6. cut vs qcut? cut uses fixed edges (or equal-width bins); qcut uses quantiles, giving roughly equal counts per bin.
  7. Why do some values become NaN after cut? They are outside the edges, or exactly on an excluded edge (the left edge by default). Use include_lowest=True, right=False or wider edges.
  8. What does "Bin edges must be unique" mean in qcut? Many repeated values make several quantiles equal; use fewer bins, cut, or duplicates="drop".
  9. What is method chaining, and why use it? Calling methods one after another on returned objects, one step per line in parentheses; it reads top to bottom, avoids temporary variables and accidental changes to the original.
  10. What does pipe do? df.pipe(f, *args, **kwargs) is f(df, *args, **kwargs); it lets your own functions sit inside a chain.

Glossary

Term Meaning Example
Function A named, reusable block of code (episode 7) def tag(cups): ...
Business rule A rule from the business, not from pandas 400+ cups = rush day
Series.apply Call a function on every value of a column week["cups"].apply(tag)
DataFrame.apply Call a function on every column or every row week.apply(bill, axis=1)
Axis Direction: 0 = down (per column), 1 = across (per row) apply(max, axis=1)
Row function A function that receives a whole row def bill(row): row["cups"] ...
Lambda A small one-line function without a name lambda v: v * 1.18
Conditional expression a if condition else b in one line "rush" if c >= 400 else "calm"
Codebook A dictionary from codes to names {"U": "UPI"}
map (Series) Look every value up in a dict/Series/function pay.map(names)
replace Swap only the values found in a dict; keep the rest pay.replace(names)
DataFrame.map Call a function on every cell of a table money.map(str)
applymap Old name of DataFrame.map, removed in pandas 3 AttributeError
Element-wise Applied to each single value DataFrame.map
Vectorised Whole-column operations in fast compiled code orders["amt"] * 1.18
Hidden loop Code that looks like one call but loops in Python apply
Binning Putting numbers into groups (bins) low / medium / high
Bin One range of values (300, 400]
Bin edges The boundaries between bins [0, 300, 400, 500]
Labels Names for the bins (one per bin) ["low", "medium", "high"]
pd.cut Bin with fixed edges or equal widths pd.cut(cups, bins=[...])
Interval A range with included/excluded ends (0, 300]
right=True Each bin includes its right edge (default) 300 is in (0, 300]
include_lowest Also include the very first edge 0 in the first bin
Category dtype A column holding a fixed set of labels the result of cut
Quantile A value that splits sorted data into equal parts median = 0.5 quantile
Quartile / decile 4 / 10 equal parts q=4, q=10
pd.qcut Bin into groups with equal counts pd.qcut(cups, q=3)
duplicates="drop" Let qcut merge repeated edges many ties
Method chaining One method after another, one step per line (df.query(...).sort_values(...))
pipe Put your own function into a chain df.pipe(add_revenue, rate=0.95)
Readable pandas style Habits that make analysis easy to follow chains, named steps

Summary

  • Series.apply(f) runs your function on every value; DataFrame.apply(f, axis=1) on every row (axis=0, the default, on every column).
  • map(dict) translates codes (unknown → NaN); replace(dict) swaps only what matches.
  • DataFrame.map(f) changes every cell; applymap no longer exists in pandas 3.
  • apply is a hidden loop: vectorised first, apply last.
  • pd.cut: bins with your edges (right edge included; outside → NaN). pd.qcut: equal-sized groups from quantiles.
  • Method chaining in round brackets, one step per line, with pipe for your own functions, keeps long analysis readable.
  • Next episode: data quality and profiling before cleaning.