Evaluation

Step 5 of the machine learning workflow

NoteEVL· at a glance

Workflow step	Step 5 of 5 · Evaluate
What you’ll learn	How to interpret and evaluate models
Standards assessed	EVAL · TRAIN
Key terms	overfitting · test set · cross-validation · confusion matrix · accuracy · RMSE · ethical matrix
Materials	readings · slides · homework · lab
Self-quiz	standards & practice →

Overview

Model evaluation is the final checkpoint in our machine learning workflow, though it often sends us back to earlier steps. We run the trained model on a held-out subset of the data called the test set to measure how accurate and reliable it is on data it hasn’t seen. Choosing metrics that fit the problem (such as precision, recall, or RMSE) matters, since accuracy alone can be misleading. Evaluation also answers a practical question: is the model good enough compared to a baseline or our requirements? If not, we iterate by tuning hyperparameters, changing the algorithm, revisiting data preparation or features, or gathering more data. Finally, evaluation includes ethical considerations. We should check performance across different groups: for whom is the model most accurate, and for whom does it perform poorly?

Readings

Before class, read the assigned material and take notes.

ISLP selected readings

Dive deeper in the content we’ll be covering. These articles can be quite math-heavy. If anything is unclear, don’t hesitate to reach out to your instructor. Aim to grasp the big-picture ideas so you can explain each topic to a non-technical audience.

• Intro to ML section 2.1
  
• Model Accuracy section 2.2
  
• Cross-validation section 5.1
  
• Bootstrap section 5.2
  

Related articles

• Accountable algorithms describes principles for investigating algorithms and holding them accountable.
  

• Algorithm guidance expands off of the accountable algorithms article to show some real-world examples. This is a bit long, so skim as needed. Pay attention to the real-world examples.
  

Slides

Cross Validation

EVL1: Overview of bias/variance tradeoff, over/underfitting, and cross validation
Open in new tab

Evaluation Metrics

EVL2: Evaluation metrics for supervised classification and regression.
Open in new tab

Auditing Algorithms

EVL3: How to audit algorithms and create an ethical matrix for our models
Open in new tab

Homework

This homework goes over selecting the correct modeling approach, and calculating and interpreting evaluation metrics. Click to view pdf.

Lab

In this assignment, you will explore the Wisconsin Breast Cancer dataset and focus on key aspects of model evaluation and resampling techniques. You will comment on the complexity of chosen models in a developed workflow, discuss resampling techniques, as well as the implications on various evaluation metrics. You will also be asked to audit the algorithms using the ethical matrix framework discussed in class. Click to view Colab Notebook.

Self-quiz

Use the standards below as a checklist, then try the practice questions to test yourself.

Standards

TRAIN

You should be able to:

• Know ways to separate train, validation, and test sets
• Define overfitting, underfitting, cross-fold validation
• Understand and give examples of the bias(underfitting) - variance (overfitting) tradeoff

EVAL

You should be able to:

• Identify which evaluation metrics to use for various ML tasks
• Compute metrics (formulas provided)
• Interpret metrics in an accessible way

Formulas given

\(\text{Accuracy} = \frac{TP + TN}{TP + FP + FN + TN}\)

\(\text{Precision} = \frac{TP}{TP + FP}\)

\(\text{Recall} = \frac{TP}{TP + FN}\)

\(F1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}\)

\(MAE = \frac{1}{n} \sum |y_{obs} - y_{pred}|\)

\(MSE = \frac{1}{n} \sum (y_{obs} - y_{pred})^2\)

\(RMSE = \sqrt{MSE}\)