Noise in data mining

Noisy data refers to information that contains a lot of irrelevant or meaningless content, often called "noise." This can include corrupted data or data that a system cannot understand or interpret, like unstructured text. Noisy data can negatively impact data analysis and lead to misleading conclusions if not handled correctly.

Noisy data is often distorted, corrupted, or has a poor Signal-to-Noise Ratio. If the methods used to remove noise are incorrect or poorly documented, they can give a false impression of accuracy and lead to wrong conclusions.

In simple terms:
Data = true information + noise

Noisy data takes up more storage space and can disrupt data analysis or mining. Statistical tools can help identify and remove noise by using patterns from past data, making it easier to analyze and extract useful insights.

Noisy data can come from various sources, such as hardware problems, software bugs, or messy input from tools like speech or optical character recognition (OCR) programs. Spelling mistakes, industry jargon, and slang can also make it hard for machines to read and understand data.

Noise is a common challenge in collecting and preparing data for analysis in Data Mining. It typically comes from two main sources:

Implicit errors: These happen because of issues with measurement tools, like sensors.
Random errors: These occur during processes like batch data handling or when experts collect data, such as when documents are digitized.

Sources of Noise in Data

Noise in data happens when the measured values differ from the true values due to various factors.

Random noise is one of the main causes. It includes a wide range of frequencies and is often called "white noise," similar to how different colors of light combine to create white. Random noise is unavoidable and commonly affects the process of collecting and preparing data, leading to errors.

There are two main sources of noise:

Errors from measurement tools, like sensors that aren't accurate.
Random errors during data processing or collection, often caused by human experts or batch processes.

Improper filtering can also add noise. For example:

Moving average filters may cause delays or distortions in the data, like cutting off peaks.
Differentiating filters can make random noise worse by amplifying it.

Managing noise carefully is essential to ensure accurate data analysis.

Types of Noise in Data

Class Noise (Label Noise)
This happens when a data point is labeled incorrectly. It can occur due to mistakes or inconsistencies during the labeling process, such as:

Subjectivity: Labels depend on personal judgment.
Data entry errors: Mistakes made while recording data.
Lack of information: Not enough details to assign the correct label.

Class noise has two types:

Contradictory examples: These are duplicate data points with the same features but different labels. For example, if two items with identical properties (e.g., color = red, value = 0.25) are labeled as "positive" and "negative," they are contradictory.
Misclassified examples: These are data points with the wrong label. For instance, if a point (e.g., value = 0.99, color = green) is labeled as "negative" but should be "positive," it is a misclassification.

Attribute Noise
This occurs when one or more feature values are incorrect or unclear. Types of attribute noise include:

Wrong attribute values: For example, if a point (e.g., value = 1.02, color = green) has an incorrect value for its first feature, it has attribute noise.
Missing or unknown values: If a point (e.g., value = 2.05, color = ?, class = negative) has a missing feature value, it contains noise.
Incomplete or irrelevant values: If a point (e.g., value = blank, color = green, class = positive) has a feature value that does not matter or affect the rest of the data, it is noisy.

Handling these types of noise is important to improve the accuracy and reliability of data analysis.

1. Binning

Binning is a technique used to reduce noise in data. Here's how it works:

First, you sort the data.
Then, you divide the sorted data into groups called "bins."
After dividing the data, you can replace the values in each bin with one of the following to smooth out the data:
- Bin mean: Replace the values in the bin with the average value of that bin.
- Bin median: Replace the values with the middle value of the bin.
- Bin boundary: Replace the values with the minimum or maximum value of the bin.

This method helps to make the data less noisy by replacing extreme values with more typical ones.

2. Regression

Binning is used to make data smoother and handle unnecessary or messy data. When analyzing data, regression helps figure out which variables are important.

Linear regression is about finding the best straight line that connects two variables so you can use one to predict the other.
Multiple linear regression works with more than two variables to make predictions.

Using regression helps create a mathematical equation that fits the data and reduces noise, making it easier to understand and analyze.

3. Clustering

Clustering is a method used to find outliers (unusual data points) and group similar data together. It's mostly used in unsupervised learning, where the data isn't labeled, and the goal is to find patterns or groupings on its own.

4. Outlier Analysis

Outlier Analysis is used to detect unusual or extreme values in data. These values are called outliers, and they stand out from the rest of the data. Outliers can happen because of measurement errors, experimental mistakes, or they might show something new or interesting.

Here are the different types of outliers:

Univariate outliers are found when looking at the values of just one feature.
Multivariate outliers are found when looking at multiple features at once, but it's hard for humans to see them, so we need a model to help.
Point outliers are single data points that are far from the rest.
Contextual outliers are things that might seem unusual in certain contexts, like extra symbols in text or background noise in speech.
Collective outliers are groups of data points that together are unusual and might show something new, like a signal that could suggest a new discovery.

Data cleaning is important because if your data has a lot of noise or errors, your results will be inaccurate.

Data Cleaning removes noise and fills in missing values. It’s the first step in preparing your data for analysis. After cleaning, other steps in data pre-processing include:

Aggregation: Combining data from different sources.
Feature Construction: Creating new features (data columns) to help with analysis.
Normalization: Adjusting data so it fits within a certain range.
Discretization: Turning continuous data into categories.
Concept hierarchy generation: Organizing data into higher-level concepts.

These steps focus on making the data consistent and ready for analysis. In fact, data pre-processing can take up to 90% of the entire process.

Noise in data mining