Binary_Classification_Metrics_Exercises
ipynb5 kBPython · 5 cellsSolution of TP 3: Binary classification metrics
Binary Classification Metrics Exercises
Exercise 1 — Basic Metrics
A classifier produces the following confusion matrix for a test set of 100 samples:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | 40 | 10 |
| Actual Negative | 20 | 30 |
Tasks
- Compute Accuracy, Precision, Recall (Sensitivity), Specificity, and F1-score.
- Interpret what each metric tells you about the model’s performance.
- Discuss how increasing the classification threshold might affect precision and recall.
Exercise 2 — Comparing Two Models
Two classifiers (A and B) yield the following confusion matrices on the same dataset of 200 samples
| Model | TP | FN | FP | TN |
|---|---|---|---|---|
| Model A | 70 | 30 | 20 | 80 |
| Model B | 90 | 50 | 10 | 50 |
Tasks
- Compute the precision, recall, F1-score, and accuracy for both models.
- Which model would you prefer if false negatives are more costly (e.g. disease detection)?
- Which would you prefer if false positives are more costly (e.g., fraud detection)?
- Which has better overall balance (based on F1)?
Exercise 3 — ROC and AUC (Manual Points)
A binary classifier outputs the following predicted probabilities and true labels for 8 samples:
| Sample | True Label | Predicted Probability |
|---|---|---|
| 1 | 1 | 0.95 |
| 2 | 0 | 0.90 |
| 3 | 1 | 0.85 |
| 4 | 0 | 0.70 |
| 5 | 1 | 0.60 |
| 6 | 0 | 0.40 |
| 7 | 1 | 0.20 |
| 8 | 0 | 0.10 |
Tasks:
- Compute the TPR (Recall) and FPR at thresholds = {0.9, 0.8, 0.6, 0.4, 0.2, 0.1}.
- Plot the ROC curve (TPR vs FPR).
- Estimate the AUC (Area Under Curve) using the trapezoidal rule.
- Interpret what the AUC value means in terms of model performance
Exercise 4 — Threshold Sensitivity
A medical test gives the following results for 10 patients:
| Patient | True | Score |
|---|---|---|
| P1 | 1 | 0.95 |
| P2 | 1 | 0.92 |
| P3 | 0 | 0.89 |
| P4 | 1 | 0.75 |
| P5 | 0 | 0.72 |
| P6 | 1 | 0.65 |
| P7 | 0 | 0.55 |
| P8 | 0 | 0.40 |
| P9 | 1 | 0.35 |
| P10 | 0 | 0.10 |
Tasks:
- For thresholds 0.5 and 0.8, compute confusion matrices.
- For each threshold, compute precision, recall, and F1-score.
- Plot how precision and recall change as the threshold increases (Precision–Recall trade-off).
- Explain why AUC-PR may be more informative than AUC-ROC for imbalanced datasets
Exercise 5 — Imbalanced Dataset Scenario
Out of 10,000 emails:
- 200 are spam (positive class)
- 9,800 are not spam (negative class)
A model flags 300 emails as spam, correctly identifying 150 of the 200 spams.
Tasks:
- Compute precision, recall, and F1-score.
- Compute accuracy.
- Explain why accuracy can be misleading in this scenario.
- Discuss which metric would be more informative (and why)