Difference Between Data Mining and Web Mining
Data mining and web mining are both used to extract useful information from large amounts of data, but they work on different kinds of data sources — and the relationship between them is not the one the phrasing "difference between" suggests.
Data mining is the process of discovering useful patterns, relationships, and knowledge in large datasets stored in databases or data warehouses.
Web mining applies data mining techniques to data available on the internet — web pages, hyperlinks, and server logs — in order to find useful information and patterns there.
The key point, stated up front: web mining is not a competitor to data mining, it is a specialization of it. Web mining uses the same algorithms — classification, clustering, association rules — applied to a data source with unusual properties. The differences that follow all come from those properties.
What Is Data Mining?
Data mining is the process of extracting meaningful information, hidden patterns, and trends from large datasets, so that organizations can analyze their data and make better decisions.
It works with many types of data: text, images, videos, files, and structured database records. In its classic form it operates on structured data — tables with defined columns, consistent types, and a known schema, typically stored in a database or data warehouse.
The main goal is to convert raw data into useful knowledge.
The Data Mining Process
The standard process, known as CRISP-DM, runs through six phases:
1. Business Understanding — defining the problem and the objectives of the analysis.
2. Data Understanding — collecting the data and studying its characteristics and quality.
3. Data Preparation — cleaning, transforming, and organizing the data so it can be analyzed.
4. Modeling — applying algorithms and techniques to identify patterns.
5. Evaluation — checking whether the model meets the business objectives.
6. Deployment — implementing the results so they can be used in real applications.
What Is Web Mining?
Web mining is the process of using data mining techniques to extract useful information and patterns from data available on the internet. This includes analyzing web pages, hyperlinks, and web server logs to discover insights about user behaviour, website structure, and content.
Its objective is to find patterns in web data that can improve services, marketing strategies, and user experience.
What Makes Web Data Different
Before looking at the types of web mining, it is worth being precise about why web data needs its own treatment. Ordinary data mining assumes a table. Web data is not one.
- It is unstructured or semi-structured. An HTML page has markup but no consistent schema; two pages describing the same thing may organize it completely differently.
- It has a link structure. Pages point at other pages, forming a graph. That graph carries information no individual page contains.
- It is enormous and constantly changing. Pages are created, edited, and removed continuously, so any collected dataset is a snapshot.
- Much of it is noise. Navigation menus, advertisements, boilerplate footers, spam pages, and automated bot traffic all have to be identified and filtered out.
- It is generated by people acting freely. Unlike a transaction database, nobody designed it to be analyzed.
Each of the three types of web mining below is a response to a different one of these properties.
Types of Web Mining
Web mining is divided into three types, according to which part of the web data is being analyzed.
1. Web Content Mining
Web content mining extracts useful information from the content of web pages — text, images, videos, and other multimedia data published on websites.
Techniques such as web scraping and text mining are used to collect and analyze this content.
Example: an organization planning a conference may collect information from many websites to choose a venue — city, state, distance from likely participants, and the popularity of the location. Extracting that information from web pages is web content mining.
Because page content is mostly natural language, this branch depends heavily on text mining methods: tokenization, stop word removal, stemming, and conversion of text to numeric vectors before any pattern discovery can happen.
2. Web Structure Mining
Web structure mining analyzes the structure of the web itself, by studying the hyperlinks between pages.
Pages are connected through hyperlinks, and analyzing those connections reveals relationships that the pages' contents do not. Viewing a person's profile page, for instance, may reveal links to their social media pages, and following those links gathers information spread across several connected pages.
The deeper idea is that the link graph carries information about importance and authority. A link from page A to page B is, loosely, a vote for B. This is the basis of the two classic web structure mining algorithms:
- PageRank, developed at Stanford by Larry Page and Sergey Brin and published in 1998, scores a page by how many pages link to it, weighted by how important those linking pages themselves are. A single link from a heavily-linked page counts for more than many links from obscure ones.
- HITS (Hyperlink-Induced Topic Search), developed by Jon Kleinberg around the same time, scores each page twice: as an authority (a page many good pages link to) and as a hub (a page that links to many good authorities).
Both were foundational to web search, and both are examples of graph analysis — a family of methods that ordinary tabular data mining does not need at all.
3. Web Usage Mining
Web usage mining analyzes user behaviour, by studying web server logs and interaction data.
When users visit a website, the server records information such as which pages were visited, how long was spent on each, and the navigation path taken through the site. Analyzing this data shows organizations how visitors actually use their sites, as opposed to how they were expected to.
A server log entry typically records the requesting address, the timestamp, the requested page, the referring page, and the browser used. Turning that raw stream into analyzable data requires two preparatory steps that are specific to this branch:
- Cleaning — removing requests for images, stylesheets, and scripts, plus traffic from search engine crawlers and bots, none of which represent a human visit.
- Sessionization — grouping the remaining requests into individual visits, so that a sequence of page views can be attributed to one person's journey through the site.
Only after that does the data resemble the transaction table that association rule mining or sequence discovery expects.
Side-by-Side Comparison
| Data Mining | Web Mining | |
|---|---|---|
| Data source | Databases, data warehouses, internal systems | Web pages, hyperlinks, server logs |
| Data structure | Mostly structured, with a known schema | Unstructured and semi-structured |
| Data ownership | Usually owned by the organization | Usually published by third parties |
| How data is obtained | Queried from internal storage | Crawled, scraped, or logged |
| Extra structure available | Rows and columns only | A hyperlink graph and a usage sequence |
| Typical noise | Missing values, entry errors | Boilerplate, advertisements, spam, bot traffic |
| Stability | Relatively stable | Changes continuously |
| Distinctive methods | Classification, clustering, association rules | The same, plus PageRank, HITS, sessionization |
| Relationship | The general discipline | A specialized application of it |
Applications of Web Mining
- Web advertising — displaying targeted advertisements based on browsing behaviour.
- E-commerce personalization — recommending products from a customer's preferences and past activity.
- Web spam filtering — detecting and removing spam websites and malicious links.
- Web page categorization — organizing pages into categories for easier searching and indexing.
- Search ranking — ordering results by relevance and authority, which is web structure mining applied at scale.
- Website improvement — identifying where visitors abandon a site, which is web usage mining applied to a business question.
Legal and Ethical Limits
One constraint applies to web mining that has no equivalent in ordinary data mining: the data usually belongs to someone else.
Collecting it is governed by each site's terms of service, by its robots.txt file, by copyright in the content, and — where the data describes identifiable people — by data protection law such as the GDPR. Access to public platform data has also narrowed sharply in recent years, with several widely-used research APIs restricted or withdrawn.
The practical rule: with web mining, establish what you are permitted to collect before asking what you are technically able to collect.
Common Mistakes
- Treating web mining as a rival technique. It is data mining applied to a particular source, not an alternative to it.
- Confusing web scraping with web mining. Scraping is collection — retrieving page content. Web mining is the analysis performed afterwards. Scraping without analysis produces a pile of HTML, not knowledge.
- Analyzing raw server logs directly. Without removing bot traffic and asset requests, and without grouping requests into sessions, log analysis measures crawlers rather than customers.
- Ignoring the link graph. Treating web pages as independent documents discards the structural information that makes web data distinctive.
Related Concepts
The three types of web mining are covered in more depth in this series' lesson on data mining and the World Wide Web, and the text mining methods that web content mining depends on are covered in the text data mining lesson. The CRISP-DM phases summarized above are covered in full in the data mining implementation process lesson.