Need for Data Mining

This lesson's question — why is data mining needed at all — is close enough to the question answered in the lesson on the importance of data mining that most of the domain-by-domain case (marketing, fraud detection, healthcare, education) would simply repeat that lesson a second time under a different title, and the limitations covered in the source material (data quality, scalability, privacy, computational complexity) are the same ones covered in full in the lesson on the challenges of data mining. What's worth pulling out separately is the specific argument for necessity rather than benefit: not "mining makes things better," but "manual analysis of this much data is not actually an option."

Why Manual Analysis Doesn't Scale

The clearest way to see this is to put real numbers on what "too much data to review manually" costs. Suppose a mid-sized retailer records 10,000 transactions per day, and a human analyst could review one transaction — checking it against known fraud patterns, say — in a generous 2 minutes:

Total review time needed per day = 10,000 transactions x 2 minutes = 20,000 minutes = 333.3 hours

At 8 working hours per analyst per day:
Analysts required just to review one day's transactions = 333.3 / 8 ≈ 42 analysts

Forty-two analysts, working full-time, doing nothing else, just to keep pace with a single mid-sized retailer's transaction volume for a single day — and this is before accounting for the fact that a human reviewer checking transactions one at a time, without ever comparing a given transaction against patterns across the full dataset simultaneously, isn't actually capable of finding the kind of statistical anomaly the z-score-based fraud detection covered in the lesson on data mining applications catches automatically. This is the real argument for data mining's necessity: it isn't a faster version of what a large team of analysts was already doing, it's the only way to do a task that direct manual review was never actually capable of doing at this scale in the first place, regardless of headcount.

Where the Rest of This Topic Is Covered

• Business value by domain (marketing, healthcare, fraud detection, scientific research, education) — covered in the lesson on the importance of data mining and the lesson on data mining applications.
• Limitations (data quality dependency, scalability, privacy, computational cost) — covered in full in the lesson on the challenges of data mining.