DS lab

How to run:

1. open vs code 
2. create a project folder and select 
3. open terminal in vs code 
4. create a virtual enivorment using :

python -m venv venv

  5. activate venv

venv/Script/activate  

  5.create a program file like "filename.py"
  6.after complete program run the code :

python filename.py

1. Implement word count program using MapReduce:

from collections import Counter

text = "Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data."

for word, count in Counter(text.lower().split()).items():
    print(word, ":", count)

hadoop : 1                         
mapreduce : 1
is : 1
a : 1
software : 1
framework : 1
for : 1
easily : 1
writing : 1
applications : 1
which : 1
process : 1
vast : 1
amounts : 1
of : 1
data. : 1

2. Implementing DGIM Algorithm:

Install lib:

pip install dgim

from dgim import Dgim

dgim = Dgim(N=5)                         # window size = last 5 bits

for bit in [1, 0, 1, 1, 0, 1, 1, 1, 0, 1]:
    dgim.update(bit == 1)                # add bit (True if 1)
    print("Bit:", bit, "| Estimated 1s:", dgim.get_count())

Bit: 1 | Estimated 1s: 1
Bit: 0 | Estimated 1s: 1
Bit: 1 | Estimated 1s: 2
Bit: 1 | Estimated 1s: 2
Bit: 0 | Estimated 1s: 2
Bit: 1 | Estimated 1s: 3
Bit: 1 | Estimated 1s: 4
Bit: 1 | Estimated 1s: 3
Bit: 0 | Estimated 1s: 3
Bit: 1 | Estimated 1s: 4

3. Implementing and Perform Streaming Data Analysis Using Flume For Data Capture, Pyspark

Install lib:

pip install pyspark

note this code needed "Java"

from pyspark import SparkContext
from pyspark.streaming import StreamingContext

ssc = StreamingContext(SparkContext("local[2]", "WordCount"), 5)   # every 5 seconds

ssc.socketTextStream("localhost", 9999) \
   .flatMap(lambda line: line.split()) \
   .countByValue() \
   .pprint()

ssc.start()
ssc.awaitTermination()

-------------------------------------------
Time: 2026-11-01 10:00:02
-------------------------------------------
('hello', 2)
('spark', 1)
('world', 1)

4. Implement Any one Clustering algorithm using Map-Reduce.

Install lib:

pip install scikit-learn

from sklearn.cluster import KMeans

data = [[1, 2], [1.5, 1.8], [5, 8], [8, 8], [1, 0.6], [9, 11]]

model = KMeans(n_clusters=2, n_init=10, random_state=0).fit(data)

print("Final Centroids:", model.cluster_centers_)
print("Cluster of each point:", model.labels_)

Final Centroids: [[7.33333333 9.        ]
 [1.16666667 1.46666667]]
Cluster of each point: [1 1 0 0 1 0]

5. Implement Page Rank Algorithm using Map-Reduce

install lib:

pip install networkx

import networkx as nx

G = nx.DiGraph({'A': ['B', 'C'], 'B': ['C'], 'C': ['A'], 'D': ['C']})
rank = nx.pagerank(G, alpha=0.85)        # alpha = damping factor

for page, score in rank.items():
    print(page, ":", round(score, 4))

A : 0.3725
B : 0.1958
C : 0.3942
D : 0.0375