Data Processing in Data Mining

Raw data, on its own, answers no questions. A spreadsheet of timestamps and transaction amounts does not tell anyone whether sales are rising, and a folder of sensor readings does not tell a self-driving car where the pedestrian is. Data processing is the sequence of steps that turns collected raw data into a form that is actually usable — filtered, organized, computed on, and presented so that a person or a system downstream can act on it. Data mining depends entirely on this step happening correctly first: no algorithm, however sophisticated, produces trustworthy patterns from data that was never properly processed.

The Six Stages of Data Processing

Every data processing pipeline, regardless of the tools used, moves data through the same six stages.

1. Data Collection. Raw data is gathered from its sources — financial records, website activity logs, sensor readings, user behavior data. The quality of everything downstream is bounded by the quality of this step: no later stage can fully recover information that was never captured accurately here.

2. Data Preparation. The raw data is sorted and filtered — duplicate entries, missing values, and incorrect records are handled so that only reliable data moves forward. (This stage is data cleaning in miniature; the mechanics of finding and fixing these problems are covered in full in the lesson on data cleaning.)

3. Data Input. The cleaned data is converted into a machine-readable format and loaded into whatever system will process it — a database, a data warehouse, or an analysis tool.

4. Data Processing. Algorithms, statistical methods, or machine learning techniques are applied to the loaded data to produce results — aggregations, classifications, predictions, or clusters, depending on what question is being asked.

5. Data Output / Interpretation. Results are presented in a form people can act on: tables, charts, dashboards, reports. A result that cannot be interpreted by the person who needs to act on it has not actually finished being processed.

6. Data Storage. The processed results (and often the cleaned input data as well) are stored for future reference, so that later analyses do not have to repeat the earlier stages from scratch.

These six stages are sequential in principle, but stage 4 is where the how varies enormously — and that "how" is the real substance of this lesson.

Batch vs. Real-Time Processing: A Concrete Comparison

The single biggest design decision in stage 4 is when processing happens relative to when data arrives. The two extremes are batch processing and real-time processing, and the difference between them is not just terminology — it produces a measurable difference in how quickly results become available.

Batch processing collects data over a period of time and processes all of it together, in one run. Payroll is the classic example: hours worked are recorded continuously through a pay period, but wages are calculated and paid out only once, at the end of it.

Real-time (or online) processing processes each piece of data immediately, as it arrives, with no waiting for a batch window. An ATM withdrawal is authorized and recorded the moment it happens, not at the end of the day.

Worked Example: Measuring the Difference in Latency

Suppose eight transactions arrive at random times across one 24-hour day. Compare what happens under each processing strategy:

import numpy as np

rng = np.random.default_rng(7)
arrival_hours = np.sort(rng.uniform(0, 24, 8))
print("Arrival times (hour of day):", np.round(arrival_hours, 2))

# Batch: all transactions are processed together at the end of the day (hour 24)
batch_latency = 24.0 - arrival_hours
print("Batch latency (hours):", np.round(batch_latency, 2))
print("Average batch latency:", round(batch_latency.mean(), 2), "hours")

# Real-time: each transaction is processed ~36 seconds (0.01 hours) after it arrives
realtime_latency = np.full(8, 0.01)
print("Average real-time latency:", round(realtime_latency.mean(), 4), "hours")

Output:

Arrival times (hour of day): [ 0.13  5.4   7.2  15.   18.62 19.71 20.97 21.53]
Batch latency (hours): [23.87 18.6  16.8   9.    5.38  4.29  3.03  2.47]
Average batch latency: 10.43 hours
Average real-time latency: 0.01 hours

Latency here means the time between when a piece of data arrives and when it has actually been processed. Under batch processing, a transaction that arrives just after midnight waits nearly 24 hours before it is processed (23.87 hours in this run); one arriving late in the day waits only a couple of hours. Averaged across all eight, the wait is over 10 hours. Under real-time processing, every single transaction is handled within seconds, regardless of when it arrived.

This single number — average latency — is the entire reason both approaches still exist rather than one simply replacing the other. Batch processing is not a worse version of real-time processing; it is the right choice whenever the 10-hour delay in this example is acceptable, because batching brings a real efficiency advantage in return: doing the payroll calculation once for 1,000 employees together is cheaper, per employee, than running the same calculation 1,000 separate times throughout the day, each with its own processing overhead. Real-time processing pays a per-transaction processing cost continuously in exchange for near-zero latency — the right trade when the delay itself is the problem, as it would be for an ATM withdrawal or a fraud check that has to block a transaction before it completes.

The Full Spectrum of Processing Types

Batch and real-time sit at two ends of a spectrum; several other processing types occupy specific points along it, each suited to a specific constraint:

Real production systems frequently combine these rather than picking exactly one — a distributed system, for instance, can run either batch or real-time workloads across its network of machines, and modern data platforms like Hadoop (batch-oriented) and Storm (stream/real-time-oriented) reflect this split directly in their design.

Manual, Mechanical, and Electronic Processing

A separate axis describes what actually does the processing, independent of timing:

• Manual processing — people perform every step (collecting, sorting, calculating) without electronic devices. It requires no infrastructure investment but is slow and, because it depends entirely on human accuracy, prone to error at any volume beyond the trivial.
• Mechanical processing — physical machines such as calculators or early tabulating devices assist the work. It reduces the error rate of pure manual work but does not scale to the volumes modern datasets require.
• Electronic processing — computers and software perform the work. This is effectively universal in data mining today, because the volumes involved (the numeric example above used only 8 records; real systems process millions) make manual or mechanical approaches impossible at any relevant scale.

This is a historical axis as much as a technical one — manual and mechanical processing are described here mainly because the terms still appear in data processing literature, not because they are realistic choices for a modern data mining workload.

Why This Matters for Data Mining Specifically

Data mining sits downstream of every stage described above, and it inherits whatever that pipeline delivers. A classification model trained on batch-processed data that is 10 hours stale, as in the example above, is describing the world as it was 10 hours ago — usually a reasonable trade-off for a monthly sales report, but a serious problem for a fraud-detection model meant to catch a transaction before it completes.

Choosing how data is processed — batch or real-time, centralized or distributed — is therefore not a separate infrastructure decision made independently of the data mining task. It sets a hard ceiling on how current the mined patterns can ever be, and that ceiling has to match what the eventual use of those patterns actually requires.