Social Media Mining Methods

Applying data mining techniques to social media is a relatively new research area compared to traditional social network analysis. The study of social networks itself goes back to the 1930s — Jacob Moreno's work on sociometry, published in 1934, is usually taken as its starting point — but those early studies mapped small groups by hand. Applying automated data mining to networks with millions of members is a much more recent development, driven by the volume of data social platforms began producing.

Today, companies and research organizations use social media analytics to track how people discuss products, services, and events. Analysts apply text mining and information propagation models to blogs and other platforms to understand not just what is being said, but how it spreads.

Common application areas include:

• Community or group detection
• Information diffusion
• Audience propagation analysis
• Topic detection and tracking
• Individual behaviour analysis
• Group behaviour analysis
• Market research

Representing Social Media Data as a Graph

Social media data is most commonly represented as a graph, made up of:

• Nodes (vertices) — representing users
• Edges (links) — representing relationships between users

In a social networking site, each user becomes a node, and connections such as friendships, follows, or interactions become the links between them.

    Ana ─────── Ben
     │  ╲        │
     │    ╲      │
    Cara ── Dev ─┘

In this small example, Ana and Ben are directly connected, while Ana reaches Dev through two different paths. That structural information — who connects to whom, how many steps apart two people are, which nodes hold the network together — is what graph representation makes analyzable, and it's information no table of individual user profiles contains.

Graph representation works well beyond friendship networks. Blogs, wikis, and discussion forums can all be modeled the same way.

Representing Blog Data

Blog data can be represented in two different ways:

1. Blog network — each blog is represented as a node.

2. Post network — each blog post is a node, and a link is created when one post references another.

Another approach used in blog analysis is Internet Online Analytical Processing (iOLAP), which considers several aspects at once rather than one at a time: individuals, relationships, content, and time.

In wiki platforms, authors can be represented as nodes, with links created when multiple authors contribute to the same content.

Why Graphs Are Hard to Work With at Scale

Representing data as a graph lets researchers apply graph theory and network analysis, but social media graphs are difficult to handle in practice:

• Networks can be extremely large — hundreds of millions or billions of nodes.
• Large graphs demand significant memory and processing power, and many graph algorithms scale poorly as node counts grow.
• Data may contain spam, bots, or fake accounts that distort the structure.
• Different platforms use different data formats.
• Both the content and the network structure change constantly.

Core Network Analysis Methods

Two families of method do most of the work in social network analysis, and both are worth understanding concretely rather than as names on a list.

Centrality measures answer the question "who matters in this network, and why?" Different measures define importance differently:

• Degree centrality counts how many direct connections a node has — a simple measure of popularity or reach.
• Betweenness centrality measures how often a node sits on the shortest path between two other nodes. A node with high betweenness may have few connections but still act as a bridge between otherwise separate groups, which makes it critical to how information travels.
• Closeness centrality measures how few steps it takes to reach everyone else, indicating how quickly a node can spread information across the network.

An account with modest follower numbers but high betweenness can matter more to how a message spreads than an account with far more followers sitting inside a single tight community.

Community detection finds groups of nodes that are more densely connected to each other than to the rest of the network. This is clustering applied to graph structure rather than to attribute values, and it can reveal groupings — a professional circle, a shared-interest community, a coordinated network of accounts — that no one on the platform ever explicitly labeled.

Text mining techniques can then analyze the content attached to nodes and links, combining what the network structure shows with what people are actually saying.

Data Mining as a Process

Applying data mining to social media follows the same process discipline covered earlier in this series, and the choice of technique depends mainly on the problem being solved:

• Classification techniques are useful when the categories you want to sort data into are already known.
• Clustering techniques are used when the patterns or groupings in the data are not yet known.

Because of this, understanding the data comes before choosing an algorithm — and a domain expert or subject analyst is often needed to interpret a dataset correctly. Social media data is particularly easy to misread without context: sarcasm, in-group slang, and coordinated inauthentic activity all look like ordinary signal to an algorithm that hasn't been designed to account for them.

Data Preprocessing

Before any mining technique is applied, the data has to be prepared and cleaned. Preprocessing here typically includes:

• Cleaning incorrect or incomplete data
• Removing spam, bot activity, or irrelevant content
• Formatting data into a suitable structure
• Reducing the dataset size for faster processing

Privacy protection belongs in this stage too. Social media contains a large amount of publicly visible data, but visibility is not permission — protecting individual privacy and respecting copyright are obligations regardless of how accessible the data is.

Why Time Matters

Time plays an unusually important role in social media analysis, because the same analysis run on different days can produce genuinely different results:

• Topics trending today may disappear tomorrow.
• Networks grow and shrink as accounts are created and abandoned.
• Group interests and individual behaviour shift constantly.

Time is especially critical in topic detection, information diffusion, network evolution, and influence analysis — all of which are about change, and none of which are meaningful from a single snapshot. This is why social media datasets are normally timestamped and analyzed as a sequence rather than treated as a static table.

Collecting Data: Network Crawling

When social media data is represented as a graph, collection usually begins from a set of seed nodes that act as starting points. The process of exploring outward from those nodes and collecting data through their connections is called network crawling.

During crawling:

• The crawler starts from the seed nodes.
• It follows links to discover new nodes and connections.
• Newly found data is stored in a repository for analysis.
• The network structure is continuously updated as new nodes appear.

Crawlers have to handle several practical problems: restricted or access-controlled sites, changes in page formats that break parsing, and broken or invalid links.

Collecting Data: APIs

Many platforms provide Application Programming Interfaces (APIs) that let developers and researchers request data directly, rather than scraping pages. APIs return structured data, respect the platform's own access rules, and are generally the sanctioned route.

They are also far more restricted than they used to be, and this is the part of the subject that has changed most:

• Twitter/X offered a free API widely used in academic research for over a decade. That free access ended in 2023, replaced by paid tiers; many researchers studying misinformation and public discourse lost access entirely.
• Facebook and Instagram sharply restricted public data access after the 2018 Cambridge Analytica episode. CrowdTangle, the tool most used for studying public Meta content, was shut down in August 2024 and replaced by the more restricted Meta Content Library.
• Technorati, a blog search engine frequently cited in earlier social media mining literature, discontinued its blog search and ranking service years ago and no longer serves this purpose at all.

Platforms that do offer APIs typically cap the number of requests allowed per day depending on the caller's permission level, so large studies must be planned around rate limits from the start. Data can sometimes be collected without an API, but scraping generally conflicts with platform terms of service — and because social media datasets are enormous, researchers usually have to limit collection scope regardless of method.

The lesson for anyone starting a project today: check current API terms before designing the study, because access conditions in this field change faster than textbooks describing them.

After collection, post-processing validates the data, removes errors, and cleans unwanted information before analysis begins.

Example: Profile-Based Platforms

Platforms such as Facebook or LinkedIn consist of users connected through profiles. Each profile holds information such as name, relationship status, birthday, email address, and hometown, and users interact by sharing posts, photos, videos, news, and links. Privacy settings let users control who sees what.

Those same settings are a real constraint on data mining: a mining application only ever sees the subset of the network that is visible to it, which means most social media studies work with an incomplete graph and should say so.

The volume of personal data on these platforms has also raised serious privacy and security concerns. Anonymization is a weaker protection than it sounds — advanced analysis can sometimes re-identify individuals from supposedly anonymous data by matching patterns of connections or behaviour against other available information. Removing names is not the same as making data anonymous.

Related Concepts

Community detection is clustering applied to graph structure — the general technique is covered in this series' lesson on clustering. The ethical and legal framework governing this kind of collection is covered in the lesson on Facebook data mining.