Evaluating Classifiers
for Changing Environments
ROC curves and the ROC Convex Hull

Classifier Induction

Comparing classifiers

Why not just use classification accuracy?

Why not just use classification accuracy?

Confusion matrix

Comparing classifiers

We could calculate expected cost/profit

Unfortunately, target conditions are difficult to know exactly . . .

ROC Space

ROC Curves

ROC space and iso-performance lines

Different conditions identify different “best” classifiers

ROC Convex Hull

ROC Convex Hull

Adding new classifiers

Sensitivity analysis

Using the ROCCH as an optimal hybrid classifier

Summary
ROC Convex Hull (ROCCH)

Extensions and future work

[End]
supplementary slides follow

Area under ROC curve (AUC)

Ranking Classifiers

Ranking Classifiers (cont’d)

Advanced Topics
Variable costs/benefits (case by case)

Advanced Topics
Variable costs/benefits (continued)

Other evaluation metrics

Maximizing precision at cutoff

Precision/recall/coverage curves

Changing priors changes curves

Example: Profit curve

Lift & cumulative response curves