Introduction to K-Means Clustering using MLIB

Inference: If one wants to know what how the Schema of the dataset looks like then they can use the print schema() function. Here it has returned two columns one is the label and the other is the features column in the vector format.

Implementation

As we know that we don’t predict any labels in K-means instead we used to make a cluster to segment the data in the best possible way, but here comes the complex part as there are no 100% optimal ways to find the best number of clusters either one should have the amazing domain knowledge or elbow method, sometimes for the easier problem, we can give a shot to hit and trial method, that’s what we are gonna do here:

1. When clusters = 2 i.e. when the dataset will be separated into two groups.
2. When clusters = 3 i.e. when the dataset will be separated into three groups.

When clusters = 2

kmeans_2_cluster = KMeans().setK(2).setSeed(1)

Inference: Above code is the way how we set the number of clusters from the set() method. Note that we are using the settled() function as well as KMeans to choose the random number of data for each group so if we set an appropriate number then after each time it will execute it will take the same random distribution.

first_model = kmeans_2_cluster.fit(dataset_kmeans)

Inference: Now we are fitting the KMeans model i.e. training the model based on the available dataset. Note that we are feeding the complete data for training as we don’t have the labels hence there is no point in splitting the dataset.

predictions_first_model = first_model.transform(dataset_kmeans)
predictions_first_model.show()

Output:

Inference: Now it’s time to make some predictions (here let’s make a relevant cluster based on the input data) and for that, we are using the transform method on the whole dataset. When we saw the output we can see that there are two groups one is tagged as 1 and another one as 2.

from pyspark.ml.evaluation import ClusteringEvaluator
evaluator = ClusteringEvaluator()

Inference: Building the model is one thing and testing and evaluating it is more important as it will allow us to consider whether this model is good to go or need any changes. Hence, we are using the clustering evaluator object to get some Kmeans evaluation methods.

silhouette_2_clusters = evaluator.evaluate(predictions_first_model)
print("Silhouette evaluation results = " + str(silhouette_2_clusters))

Output:

Silhouette evaluation results = 0.9997530305375207

Inference: The silhouette is a statistical method that checks upon the consistency within the clusters of data. Its coefficient value ranges between [-1,1] the more positive the coefficient the better the more the data point is within that particular cluster.

two_centroid = first_model.clusterCenters()
print("Center of clusters: ")
for c in two_centroid:
    print(c)

Output:

Center of clusters: 
[9.1 9.1 9.1]
[0.1 0.1 0.1]

kmeans_3_clusters = KMeans().setK(3).setSeed(1)
second_model = kmeans_3_clusters.fit(dataset_kmeans)
predictions_second_model = second_model.transform(dataset_kmeans)

from pyspark.ml.evaluation import ClusteringEvaluator
evaluator = ClusteringEvaluator()

silhouette_3_clusters = evaluator.evaluate(predictions_second_model)
print("Silhouette evaluation results = " + str(silhouette_3_clusters))

Silhouette evaluation results = 0.6248737134600261

three_centroid = second_model.clusterCenters()
print("Center of clusters: ")
for c in three_centroid:
    print(c)

Center of clusters: 
[9.1 9.1 9.1]
[0.05 0.05 0.05]
[0.2 0.2 0.2]