Data Wrangling

Raw data, as it arrives from a database export, a scraped website, or a survey tool, is rarely usable as-is — dates come in inconsistent formats, some fields are missing, duplicate records slip in, and a value here or there is obviously wrong. Data wrangling (also called data munging) is the process of converting that raw data into a clean, structured, consistent format before any real analysis begins. It's worth being upfront about scope: data wrangling is the broader umbrella that includes data cleaning (covered in its own dedicated lesson) as one step among several — cleaning fixes errors and gaps in the data that already exists, while wrangling also covers restructuring, enriching, and validating it, end to end. In practice, this stage consumes a genuinely large share of an analyst's time — reportedly more than the actual analysis that follows it — precisely because getting the data into a trustworthy shape is what makes every later step possible.

The Six-Step Wrangling Process, Applied to a Real Example

Rather than treat the standard six-step wrangling process as an abstract checklist, it's worth walking one small, messy dataset through every step to see exactly what each one changes. Start with six raw order records, deliberately messy in the ways real exported data usually is:

ID  Name    Order Date   Amount
1   Alice   2024-01-05   120
2   Bob     01/06/2024   85
3   Alice   2024-01-05   120     <- identical to row 1
4   Carla   (missing)    95
5   Dave    2024-01-09   9999    <- suspiciously large
6   Eve     2024-01-10   60

1. Discovery. Before touching anything, establish what the data actually contains and what the analysis goal is. Here: the goal is to compute total order revenue by month, and a first look already surfaces three problems — inconsistent date formats (row 2 uses 01/06/2024 while the rest use 2024-01-05-style ISO dates), a missing date (row 4), and a suspiciously large amount (row 5).

2. Organization (structuring). Standardize the format so every record is directly comparable — converting row 2's date to 2024-01-06 to match the ISO format used everywhere else. Structuring problems like this have to be fixed before cleaning can even meaningfully evaluate the data, since a script checking "is this a valid date" needs one consistent format to check against.

3. Cleaning. This is the step covered in depth in the dedicated lesson on data cleaning: removing duplicates (row 3 is an exact duplicate of row 1 and gets dropped), and deciding how to handle the missing date in row 4 — here, since the analysis goal is a monthly total, a record with no date can't be assigned to any month, so it gets dropped rather than guessed at. Row 5's amount of 9999 isn't automatically removed at this stage — it gets flagged for review rather than deleted, since a value being unusual doesn't necessarily mean it's wrong (the lesson on outliers in data mining covers exactly this judgment call in depth).

4. Data enrichment. Add information the analysis needs that isn't in the original data — deriving a Month column (2024-01) from each Order Date so the records can actually be grouped and summed by month, which the raw date field alone doesn't directly support.

5. Validation. Check that the cleaned, structured, enriched data actually satisfies the rules the analysis depends on — every amount is a positive number, every date parses correctly, every record has a valid month. This is also where row 5's flagged value gets resolved one way or the other, and the resolution matters more than it might look: with the $9999 order included, the four remaining records total $10,264; with it excluded pending verification, they total $265 — a nearly 39-fold difference in reported monthly revenue depending entirely on a single validation decision. This is exactly why flagging rather than silently keeping or silently dropping an unusual value is the right instinct: the decision needs to be made deliberately, not by accident.

6. Publishing. Save and document the final, clean dataset in a form ready for the next stage — whether that's a data mining model, a report, or a dashboard. Documenting what was changed and why (the duplicate removed, the missing-date record dropped, the outlier's resolution) matters here too, since anyone consuming the final numbers should be able to trace how they were derived.

Note that this process is iterative in practice, not strictly linear — validation often reveals a problem that sends the data back through cleaning again, and that back-and-forth is normal rather than a sign something went wrong the first time.

Where Wrangled Data Ends Up

After wrangling, data is typically organized into one of a few recognizable shapes depending on what it's for: transactional data (individual business events like the order records above), an analytical base table (ABT) — a single flat table with one row per entity and all the features a model needs, purpose-built for feeding into a data mining or machine learning step — time-series data (values indexed by time, as covered in the lesson on sequence mining), or document data (wrangled text, such as cleaned emails or reports, ready for text mining).

Tools Commonly Used for Data Wrangling

• Excel / Power Query — spreadsheet-based wrangling, widely used for smaller datasets.
• OpenRefine — a dedicated tool for cleaning and reconciling messy, inconsistent data at scale.
• Google Cloud Dataprep — a cloud-based visual data preparation service.
• Tabula — extracts tabular data specifically out of PDF files.
• pandas (Python) — the standard library for programmatic data wrangling in Python, providing the merging, reshaping, and cleaning operations the six-step process above relies on.
• csvkit — a set of command-line tools purpose-built for working with CSV files directly from a terminal.

Why This Step Can't Be Skipped

The order-total example above makes the underlying point concrete rather than abstract: a single unresolved data-quality issue (one flagged value) changed a computed result by a factor of nearly 40. No data mining technique — however sophisticated the classification, clustering, or regression model built on top of it — can produce a trustworthy answer from data that hasn't gone through this process first. Wrangling doesn't make the analysis smarter; it makes sure the analysis is actually about the data it's supposed to be about.