Data Integration

Data integration combines data from different sources into a single unified view. In data mining, data typically arrives from many places — databases, data warehouses, data cubes, flat files — each with its own format and structure.

The goal is a merged dataset that is accurate and consistent enough to analyze. Getting there means resolving redundancy, inconsistency, and duplication, and these problems are rarely trivial: two systems that both record "customers" may disagree about what counts as one.

Data integration is formally described using a (G, S, M) triple:

• G (Global schema) — the structure of the integrated view users query against
• S (Source schemas) — the structures of the individual data sources
• M (Mapping) — the correspondence between source data and the global schema

The mapping is where the real work sits. Defining a global schema is straightforward; specifying exactly how each source's fields translate into it, including the cases where they don't translate cleanly, is the hard part.

What Data Integration Achieves

Integration combines sources — databases, data cubes, flat files, and separate information systems — into one consistent dataset, while removing:

• Inconsistent data
• Duplicate records
• Conflicting information
• Redundant attributes

This matters for data mining specifically because mining algorithms assume their input is coherent. An algorithm given the same customer twice will treat them as two people and weight their behaviour double, and no amount of algorithmic sophistication corrects for that afterward.

Why It Matters

Unified view — scattered data is brought into a single system that can actually be analyzed as a whole.

Better decisions — integrated data supports business intelligence, analytics, and reporting across the organization rather than within silos.

Improved accuracy — duplicates and inconsistencies are removed rather than silently distorting results.

Faster analysis — questions spanning several systems can be answered without manual reconciliation each time.

Example: Healthcare

Integrating patient records from different hospitals and clinics lets doctors access a complete patient history rather than a fragment, identify conditions faster with fuller information, and improve treatment decisions. It also improves insurance processing and record accuracy.

It also illustrates the difficulty well: two hospitals may identify the same patient by different identifiers, record the same condition under different codes, and store dates in different formats. All three must be resolved before the merged record is trustworthy.

Two Approaches

1. Tight Coupling

Data from different sources is collected, transformed, and stored in a central repository using ETL:

• Extract — collect data from the sources
• Transform — convert it into a common format
• Load — store it in the central repository

All integrated data lives in one physical location. This gives fast, consistent queries, at the cost of storage and of data that is only as current as the last load.

A modern variant reverses the last two steps — ELT loads raw data into a cloud warehouse first and transforms it there, taking advantage of the warehouse's processing power and keeping the untransformed original available.

2. Loose Coupling

Data stays in its original source systems. When a user submits a query:

1. The system translates it into the formats each source understands
2. The query is sent to each source
3. Results are collected, combined, and returned

Nothing is copied — only the results are integrated. This keeps data current by definition and avoids duplicate storage, but every query depends on all sources being available and fast enough to answer in real time. This approach is now commonly called data virtualization.

Issues in Data Integration

1. Entity identification problem

The same real-world entity may be represented differently across sources. One system stores Customer ID, another stores Customer Number — both may refer to the same customer and must be matched correctly.

This gets harder when there is no shared identifier at all, and matching must rely on names, addresses, or dates of birth that may contain typos or have legitimately changed.

2. Structural conflicts

Different systems structure the same information differently. In one system a discount applies to the whole order; in another it applies per item. These differences must be reconciled before merging, and doing so may require decisions about which representation is authoritative.

3. Redundancy and correlation

Redundant data is unnecessarily repeated information. If one dataset stores date of birth and another stores age, age is redundant — it can be derived from the date of birth.

Correlation analysis identifies relationships between attributes and helps detect redundancy, which is covered in detail in this series' lesson on redundancy and correlation.

4. Tuple duplication

Duplicate records may appear when data is collected from multiple sources — the same customer record appearing several times in the integrated dataset. These must be detected and removed, which is harder than exact matching suggests when the duplicates differ slightly in spelling or formatting.

5. Data conflict detection and resolution

Different sources may represent the same fact with different values. A hotel price stored in Indian Rupees and the same price stored in US Dollars are both correct but not comparable. Conflicts like these — differing units, currencies, scales, or reference dates — must be identified and converted to a common basis.

Data Integration Techniques

1. Manual integration

An analyst collects, cleans, and merges data by hand. Simple for small datasets, but very time-consuming and unworkable at organizational scale.

2. Middleware integration

Middleware software connects different systems and translates between data formats, acting as a bridge between legacy and modern systems — for example, connecting an old database to a new application.

3. Application-based integration

Purpose-built software extracts, transforms, and loads data from the various sources. Fast and automated, but requires technical expertise to build and maintain.

4. Uniform access integration

Data stays in its original location while users see a single integrated view, with results combined only when requested. This is the loose coupling approach described above.

5. Data warehousing

Integrated data is stored in a separate centralized warehouse. This supports complex queries and is well-suited to analysis and reporting, at the cost of additional storage and higher maintenance.

Data Integration Tools

On-premise tools integrate data from local systems and databases within an organization, where data residency or regulatory requirements prevent using external services.

Open-source tools are free and customizable — Apache NiFi and Talend Open Studio are widely used examples — but the organization takes on security, support, and maintenance itself.

Cloud-based tools provide Integration Platform as a Service (iPaaS), handling integration through managed cloud platforms. These are scalable, easy to access, and well-suited to large organizations, though they require sending data to a third-party service.

Where Integration Sits in the Mining Process

Data integration is not a standalone activity — it is one step in the larger pipeline:

  Sources → Integration → Cleaning → Transformation → Mining → Evaluation

It appears explicitly as a step in the KDD process and within CRISP-DM's Data Preparation phase, both covered elsewhere in this series. The ordering matters: integrating before cleaning means cleaning has to handle the inconsistencies integration introduced, while cleaning each source first means doing similar work repeatedly. In practice the two interleave.

The quality of everything downstream depends on this step. Patterns discovered from badly integrated data can be artefacts of the integration rather than facts about the world — a spike in sales that is really the same transactions counted twice from two systems.

Related Concepts

ETL and the data warehouse are covered in this series' lesson on data mining vs data warehousing. Redundancy and correlation analysis, listed above as issue 3, has its own lesson. The wider preprocessing sequence this step belongs to appears in the KDD process and data mining implementation process lessons.