Exploring Data

Step 1 of the machine learning workflow

NoteExploring Data · at a glance

Workflow step	Step 1 of 5 · Explore
What you’ll learn	How to summarize a dataset, read its distributions, and reason about feature types.
Standards assessed	FLOW · EDA-STAT
Key terms	target · features · distribution · feature type · summary statistic · skew
Materials	readings · slides · homework · lab
Self-quiz	standards & practice →

Exploratory data analysis (EDA) (one-sentence definition goes here — bold the term, define it plainly.) (Then 1–2 more sentences previewing what this chapter covers. This lead should stand alone as a summary.)

Overview

(1–3 short paragraphs: what this is, why it matters, and where it sits in the five-step workflow. Flesh out later.)

Key ideas

NoteKey idea

(State the single most important takeaway of this chapter.)

(List or briefly explain the 2–4 core concepts. Split into ### subsections as you flesh out.)

Readings

Before class, read the assigned material and take notes.

Related articles

• Stats distributions describes how by only knowing statistical properties we may be making inaccurate assumptions of what our data actually looks like visually.
  

• Implications behind visualizations (excusing the explicit language) has loads of examples of how biases can creep into our data insights. This article shows how we can initially interpret visualizations, and perhaps how to start questioning if the visualization is telling the story we think it is.
  

• Using LLMs to storyboard goes through the process of how a data-driven story is created at The Pudding, but using an LLM. This article is a great way to understand how data-driven stories are put together, and how good (or not so good) LLMs are at replicating tasks associated with the design and implementation process.
  

• ggplot vs plotnine compares the graphing power of R’s ggplot to Python’s plotnine (R wrapper for ggplot). Useful for those who have experience with creating visualizations in R, or for those who are unimpressed by matplotlib’s capabilities. Only read if you have a background using R.

• Statistics for ML walks through basic definitions of statistical measures, charts, and distributions.

Cheat sheets

The following cheatsheets may be helpful for your reference: data storytelling, data visualizations, matplotlib, plotly, seaborn.

Slides

Model workflow

EDA1: Overview of the 5step workflow, learning paradigms, and features types.
Open in new tab

statistics

EDA2: Descriptive statistics (mean, median, standard deviation), statistical tests and hypothesis testing.
Open in new tab

Visualizations

EDA3: How to tell a story, and creating visualizations with mathplotlib, plotnine, seaborn.
Open in new tab

Homework

This homework goes over looking at summary tables for a dataset, explaining statistical terms to non-tech folks, exploring statistics, and interpretting statistical tests. Click to view pdf.

Lab

In this lab, we will explore how to critic and refine code outputted by a language model, and practice a bit of prompt engineering. We will be using the Titanic dataset to perform exploratory data analysis tasks, including calculating statistical measures, conducting statistical tests, and creating visualizations. Click to view Colab Notebook.

Self-quiz

Use the standards below as a checklist, then try the practice questions to test yourself.

Standards

FLOW

You should be able to:

• Define learning approaches (supervised vs. unsupervised learning) and learning tasks (regression, classification, and clustering).
• Interpret a dataset summary table.
• Describe a model’s goal and explain who might benefit from the model.
• Describe the five steps ML workflow and give a few examples of tasks for each step

EDA-STAT

You should be able to:

• Understand and describe statistical measures, distributions, and tests
• Read and interpret charts, including identifying key features (skew, balance, shape) and chart types (histogram, boxplot, scatterplot)
• Identify different feature types (numeric, discrete, continuous, ordinal, categorical, binary, or none).
• Explain what factors to consider when deciding whether a feature type is appropriate to include in a machine learning model.

Practice questions

Click a question to reveal the answer.

NoteQ1. (Add a practice question.)

(Answer — students click to reveal.)

NoteQ2. (Add a practice question.)

(Answer.)