Data Mining vs Big Data

 DATA MINING VS BIG DATA

Data mining leverages tools like statistical models, machine learning, and data visualization to extract valuable insights and patterns from large datasets. In contrast, Big Data refers to the vast amounts of data that are generated at high velocity and volume, which are difficult to process using traditional databases and analysis programs.

Big Data:

Big Data refers to the vast amounts of data that can be structured, semi-structured, or unstructured, often ranging in terabytes or more. Processing such large datasets on a single system is challenging because of the immense volume and complexity. To manage this, intermediate calculations are stored in a computer's RAM during processing and analysis. However, when attempting to process such large amounts of data, the time required for computation can be substantial, and the system can become overloaded, causing performance issues.

To illustrate this concept, consider a real-world example: Big Bazaar, a popular retail chain. Each month, customers visit Big Bazaar stores across India, making purchases that are tracked by the company. With over 250 stores in India alone, each store monitors the sale of every item, including the product details and store location. This real-time information is fed into a central server system. When you consider that each piece of data represents a customer’s purchase, the total amount of data generated each month can easily reach around 1 terabyte for just one chain of stores. This enormous volume of data highlights the challenges of managing and processing Big Data.

Big Data is characterized by the 5Vs: Volume, Variety, Velocity, Veracity, and Value.

• Volume: In Big Data, volume refers to the massive amount of data that is generated, which can be enormous when dealing with large datasets.

• Variety: Variety in Big Data refers to the different types of data, including structured data like company records, and unstructured data such as web server logs and social media posts.

• Velocity: Velocity refers to the speed at which data is generated and processed. In the context of Big Data, data is growing exponentially at a rapid pace.

• Veracity: Veracity in Big Data refers to the uncertainty or reliability of the data, acknowledging that not all data is accurate or trustworthy.

• Value: Value in Big Data refers to the usefulness of the data being stored and processed. It focuses on determining whether the data provides meaningful insights and how it can be leveraged to create value.

• Apache Hadoop is one of the most popular frameworks for processing and storing Big Data. It provides a distributed storage and processing system that allows you to handle massive amounts of data efficiently. The Hadoop ecosystem is built around several key components that work together to process Big Data:

  Key Components of Apache Hadoop

1. Hadoop Distributed File System (HDFS):

   • Purpose: HDFS is the storage layer of Hadoop. It allows you to store large volumes of data across multiple machines in a distributed manner.
   • How It Works: Data is divided into large blocks (typically 128 MB or 256 MB in size) and distributed across a cluster of nodes. Each block is replicated (usually 3 times) for fault tolerance.

2. MapReduce:

   • Purpose: MapReduce is the processing model that handles large-scale data processing. It splits the job into smaller tasks and processes them in parallel across the cluster.
   • How It Works:
     • Map Phase: The data is first split into smaller chunks (mappers) that perform initial processing, such as filtering or transformation.
     • Reduce Phase: The intermediate results from the Map phase are then aggregated, sorted, or merged to produce the final output.
   • Example: In a word count program, the Map phase might count the occurrences of words in chunks of data, and the Reduce phase would aggregate the results to get the total count for each word.

3. YARN (Yet Another Resource Negotiator):

   • Purpose: YARN is the resource management layer of Hadoop. It manages and schedules resources across the cluster, ensuring that jobs are assigned to available nodes and resources are distributed efficiently.
   • How It Works: YARN manages jobs submitted to the cluster, allocates resources, and tracks job status. It is responsible for job coordination and resource scheduling in the cluster.

4. Hadoop Common:

   • Purpose: Hadoop Common is a set of shared utilities and libraries that support the core Hadoop modules. It provides the necessary tools to work with HDFS, MapReduce, and other components.

5. HBase (Optional):

   • Purpose: HBase is a NoSQL database built on top of HDFS for handling real-time read/write operations on large datasets. It is particularly useful for scenarios where low-latency access is required for large data volumes.

6. Hive (Optional):

   • Purpose: Hive is a data warehousing and SQL-like query interface built on top of Hadoop. It allows you to query large datasets using a SQL-like language (HiveQL).

7. Pig (Optional):

• Purpose: Pig is a high-level platform for processing data, where you can write scripts (Pig Latin) to perform data transformations. It abstracts away the complexity of writing low-level MapReduce code.

  Steps to Process Big Data Using Apache Hadoop

1. Set Up the Hadoop Cluster:

   • Installation: First, install Hadoop on your cluster of machines or use cloud-based Hadoop services like AWS EMR, Google Dataproc, or Azure HDInsight.
   • Configuration: Configure HDFS, YARN, and MapReduce for your cluster. Ensure proper setup of network, storage, and resource management.

2. Ingest Data:

   • Use tools like Apache Flume or Apache Sqoop for data ingestion.
   • Flume is used to collect streaming data, like logs, and send them to HDFS.
   • Sqoop is used for importing data from relational databases into HDFS.

3. Store Data in HDFS:

   • Load your raw data into HDFS using the Hadoop commands (hadoop fs -put).
   • HDFS handles storing the data in a distributed manner and provides fault tolerance by replicating blocks across nodes.

4. Process Data with MapReduce:

   • Write MapReduce Jobs: Implement your data processing logic in the form of MapReduce jobs.
   • Use the MapReduce API in Java, Python, or other supported languages to write mappers and reducers.
   • Map Function: It processes the input data, splits it into key-value pairs, and prepares it for the Reduce phase.
   • Reduce Function: It aggregates the intermediate results and produces the final output.

5. Run the Job on YARN:

   • Submit your MapReduce job to the YARN Resource Manager.
   • YARN schedules and allocates resources across the cluster to run your MapReduce jobs efficiently.

6. Access Processed Data:

   • Once the MapReduce job is finished, the output is stored back into HDFS.
   • You can access the results using Hadoop commands or connect tools like Hive, Pig, or custom applications to retrieve and analyze the processed data.

7. Query Data (Optional):

   • If you're using Hive, you can run SQL-like queries on the processed data stored in HDFS.
   • If you're using HBase, you can access the data in real-time.

8. Monitor and Scale:

   • Use Hadoop's monitoring tools (e.g., ResourceManager UI, JobHistory Server) to track the progress of your jobs.
   • Scale the Cluster: You can scale your Hadoop cluster by adding more nodes to handle larger datasets or improve performance.