Fairness explorer

About the data

The cases are hypothetical and the numbers come from a seeded random number generator. Each case is loosely based on a real system, linked in its description, and the numbers do not estimate how that system performs.

A risk score rates people from 0 to 100, and everyone at or above the cutoff is flagged. Groups A and B are scored by the same tool but have different base rates, as in the Week 3 reading. Nothing is saved or sent.

Case

Error costs

Confusion matrix

The counts cover all 1,200 people at the current cutoff.

Flagged
Not flagged
Actually high-risk
True positive 0caught
False negative 0a miss (Type II error)
Actually not high-risk
False positive 0a false alarm (Type I error)
True negative 0correctly cleared
Each area is in proportion to its count.
Recall / sensitivity
The share of everyone actually high-risk who are flagged: TP / (TP + FN).
Precision / PPV
The share of everyone flagged who are actually high-risk: TP / (TP + FP).
False-positive rate (Type I)
The share of everyone actually not high-risk who are wrongly flagged: FP / (FP + TN).
False-negative rate (Type II)
The share of everyone actually high-risk who are missed: FN / (FN + TP).

Consequences at this cutoff

Which cutoff would you choose for this case?

Group A

Real base rate: 26% actually high-risk

Group B

Real base rate: 50% actually high-risk

Fairness tests

Each test compares one score between the groups and passes when they are within 5 percentage points.

    Conflict between the tests

    When groups have different base rates, no single cutoff or pair of cutoffs can equalize the flag rate, the false-positive rate and the predictive value at once, except at extremes where almost everyone or almost no one is flagged (Chouldechova, 2017; Kleinberg, Mullainathan, and Raghavan, 2016). Predictive value rises with the base rate, so equal error rates give the group with the higher base rate a higher predictive value and a higher flag rate. Choosing which test to meet and which error to accept is a decision about values.

    More scores

    MeasureGroup AGroup BAll

    Calculations

    TP stands for true positives, FP for false positives, FN for false negatives and TN for true negatives.

    Score distributions

    Bars count people in five-point score bands, with people who have the outcome above the line and people without it below. Dark bars left of the cutoff are misses and light bars right of it are false alarms. Dragging on either chart moves the cutoff.

    Group A
    Group B

    has the outcome does not have the outcome

    In practice

    The bars overlap because some people with the outcome score low and some without it score high. Inside the overlap every cutoff trades misses against false alarms, and a tool whose scores overlap less forces a smaller trade.

    ROC curve

    Each point is one cutoff, plotting recall (vertical) against the false-positive rate (horizontal); the circle and square mark the current cutoffs for groups A and B. The dashed diagonal is chance, and curves nearer the top-left corner separate people with and without the outcome better.

    The area under the curve (AUC) is the probability that a randomly chosen person with the outcome scores higher than a randomly chosen person without it. An AUC of 0.5 is chance, 1.0 is perfect ranking, and the cutoff does not change it.

    group A (circle at its cutoff) group B (square at its cutoff)

    In practice

    A program acts at one cutoff, and two groups with nearly the same AUC can have very different error rates there. AUC also says nothing about whether a score of 70 means a 70 in 100 chance; calibration checks that.

    Calibration

    A score is calibrated when about 70 percent of people scored around 70 have the outcome. Each point is a ten-point score band, plotting average score (horizontal) against the share with the outcome (vertical), and points on the dashed diagonal are calibrated. Calibration does not change with the cutoff.

    group A group B. Bands with fewer than 15 people are left out.

    In practice

    A caseworker who sees a score of 70 reads it as roughly 70 in 100. A tool can be well calibrated in one group and off in another, so the same number carries different risk depending on whom it describes.

    Calibration calculation

    Predicted share = average score in the band / 100. Observed share = people in the band with the outcome / people in the band. A score can be calibrated and still have poor recall.

    Score bandGroup A predicted and observedGroup B predicted and observed

    Model training: how a score like this is learned from past records, and how overfitting happens.