An Introduction to MapReduce with a Word Count Example

This article was published as a part of the Data Science Blogathon.

Introduction

Hadoop facilitates the processing of large datasets in a distributed manner and provides the foundation on which other services and applications can be built. MapReduce and HDFS are the two main components of Hadoop. For a detailed look at HDFS, you can refer to this article: Working of Hadoop Distributed File System(HDFS). Hadoop MapReduce processes the massive amount of structured and unstructured data stored in HDFS. In this article, we will look at MapReduce’s architecture and workflow. We will also discuss some of the benefits and limitations of using MapReduce.

What is MapReduce?

MapReduce is a Hadoop framework and programming model for processing big data using automatic parallelization and distribution in the Hadoop ecosystem. MapReduce consists of two essential tasks, i.e., Map and Reduce. Reducing tasks always follow map tasks. The reduce task always follows the map task. In the Map task, data are divided into chunks and processed in parallel. The output of a map task is used as input in reduce tasks, and the data is shuffled and reduced.

Advantages of using MapReduce

The advantages of using MapReduce are as follows:

• MapReduce can define mapper and reducer in several different languages using Hadoop streaming.
• MapReduce facilitates automatic parallelization and distribution, reducing the time required to run programs.
• MapReduce provides fault tolerance by re-executing, writing map output to a distributed file system, and restarting failed map or reducer tasks.
• Processing of data using MapReduce is a cost-effective solution.
• MapReduce processes large volumes of unstructured data very quickly.
• Using HDFS and HBase security, MapReduce ensures data security by allowing only approved users to access data stored in the system.
• MapReduce programming utilizes a simple programming model to handle tasks more efficiently and quickly and is easy to learn.
• MapReduce is flexible and works with several Hadoop languages to handle and store data.

MapReduce Architecture

MapReduce process has the following phases:

1. Input Splits
2. Mapping
3. Shuffling
4. Sorting
5. Reducing

Input Splits

MapReduce splits the input into smaller chunks called input splits, representing a block of work with a single mapper task.

Mapping

The input data is processed and divided into smaller segments in the mapper phase, where the number of mappers is equal to the number of input splits. RecordReader produces a key-value pair of the input splits using TextFormat, which Reducer later uses as input. The mapper then processes these key-value pairs using coding logic to produce an output of the same form.

Shuffling

In the shuffling phase, the output of the mapper phase is passed to the reducer phase by removing duplicate values and grouping the values. The output remains in the form of keys and values in the mapper phase. Since shuffling can begin even before the mapper phase is complete, it saves time.

Sorting

Sorting is performed simultaneously with shuffling. The Sorting phase involves merging and sorting the output generated by the mapper. The intermediate key-value pairs are sorted by key before starting the reducer phase, and the values can take any order. Sorting by value is done by secondary sorting.

Reducing

In the reducer phase, the intermediate values from the shuffling phase are reduced to produce a single output value that summarizes the entire dataset. HDFS is then used to store the final output.

Here’s an example of using MapReduce to count the frequency of each word in an input text. The text is, “This is an apple. Apple is red in color.”

• The input data is divided into multiple segments, then processed in parallel to reduce processing time. In this case, the input data will be divided into two input splits so that work can be distributed over all the map nodes.
• The Mapper counts the number of times each word occurs from input splits in the form of key-value pairs where the key is the word, and the value is the frequency.
• For the first input split, it generates 4 key-value pairs: This, 1; is, 1; an, 1; apple, 1; and for the second, it generates 5 key-value pairs: Apple, 1; is, 1; red, 1; in, 1; color.
• It is followed by the shuffle phase, in which the values are grouped by keys in the form of key-value pairs. Here we get a total of 6 groups of key-value pairs.
• The same reducer is used for all key-value pairs with the same key.
• All the words present in the data are combined into a single output in the reducer phase. The output shows the frequency of each word.
• Here in the example, we get the final output of key-value pairs as This, 1;  is, 2; an, 1; apple, 2; red, 1; in, 1;  color, 1.
• The record writer writes the output key-value pairs from the reducer into the output files, and the final output data is by default stored on HDFS.

Limitations of MapReduce

MapReduce also faces some limitations, and they are as follows:

• MapReduce is a low-level programming model which involves a lot of writing code.
• The batch-based processing nature of MapReduce makes it unsuitable for real-time processing.
• It does not support data pipelining or overlapping of Map and Reduce phases.
• Task initialization, coordination, monitoring, and scheduling take up a large chunk of MapReduce’s execution time and reduce its performance.
• MapReduce cannot cache the intermediate data in memory, thereby diminishing Hadoop’s performance.

Conclusion

In this article, we discussed the importance of MapReduce for the Hadoop framework. The MapReduce framework has helped us deal with huge amounts of data and find solutions to previously considered impossible problems. In addition to analyzing large data sets in data warehouses, the MapReduce framework can also be applied by companies offering online services to optimize their marketing strategies. The following are our key takeaways from this article:

• MapReduce is fast, scalable, cost-effective, and can handle large volumes and complex data.
• In MapReduce architecture, each phase takes input in the form of key-value pairs.
• It provides parallel computing, fault tolerance, distribution, and security.
• The number of mappers in the mapper phase equals the number of input splits.
• It is recommended that you take 1/4th of the number of mappers as the number of reducers.
• The final output is by default stored in the HDFS storage.

Hopefully, you have understood the architecture of MapReduce and how it works. In case you have any questions, you can leave a comment below or connect with me on LinkedIn.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.