An Introduction to MapReduce with Map Reduce Example

Introduction

Hadoop facilitates the processing of large datasets in a distributed manner and provides the foundation on which other services and applications can be built. Map Reduce example and HDFS are the two main components of Hadoop. For a detailed look at HDFS, you can refer to this article: Working of Hadoop Distributed File System(HDFS). Hadoop MapReduce processes the massive amount of structured and unstructured data stored in HDFS. In this article, we will look at MapReduce’s architecture and workflow. We will also discuss some of the benefits and limitations of using Map Reduce.

This article was published as a part of the Data Science Blogathon.

What is MapReduce?

MapReduce is a programming model and framework within the Hadoop ecosystem that enables efficient processing of big data by automatically distributing and parallelizing the computation. It consists of two fundamental tasks: Map and Reduce.

In the Map phase, the input data is divided into smaller chunks and processed independently in parallel across multiple nodes in a distributed computing environment. Each chunk is transformed or “mapped” into key-value pairs by applying a user-defined function. The output of the Map phase is a set of intermediate key-value pairs.

The Reduce phase follows the Map phase. It gathers the intermediate key-value pairs generated by the Map tasks, performs data shuffling to group together pairs with the same key, and then applies a user-defined reduction function to aggregate and process the data. The output of the Reduce phase is the final result of the computation.

Map Reduce example allows for efficient processing of large-scale datasets by leveraging parallelism and distributing the workload across a cluster of machines. It simplifies the development of distributed data processing applications by abstracting away the complexities of parallelization, data distribution, and fault tolerance, making it an essential tool for big data processing in the Hadoop ecosystem.

Advantages of using MapReduce

The advantages of using MapReduce are as follows:

• MapReduce can define mapper and reducer in several different languages using Hadoop streaming.
• MapReduce facilitates automatic parallelization and distribution, reducing the time required to run programs.
• MapReduce provides fault tolerance by re-executing, writing map output to a distributed file system, and restarting failed map or reducer tasks.
• Processing of data using MapReduce is a cost-effective solution.
• MapReduce processes large volumes of unstructured data very quickly.
• Using HDFS and HBase security, Map Reduce ensures data security by allowing only approved users to access data stored in the system.
• MapReduce programming utilizes a simple programming model to handle tasks more efficiently and quickly and is easy to learn.
• MapReduce is flexible and works with several Hadoop languages to handle and store data.

MapReduce Architecture

Map Reduce example process has the following phases:

1. Input Splits
2. Mapping
3. Shuffling
4. Sorting
5. Reducing

Input Splits

MapReduce splits the input into smaller chunks called input splits, representing a block of work with a single mapper task.

Mapping

The input data is processed and divided into smaller segments in the mapper phase, where the number of mappers is equal to the number of input splits. RecordReader produces a key-value pair of the input splits using TextFormat, which Reducer later uses as input. The mapper then processes these key-value pairs using coding logic to produce an output of the same form.

Shuffling

In the shuffling phase, the output of the mapper phase is passed to the reducer phase by removing duplicate values and grouping the values. The output remains in the form of keys and values in the mapper phase. Since shuffling can begin even before the mapper phase is complete, it saves time.

Sorting

Sorting is performed simultaneously with shuffling. The Sorting phase involves merging and sorting the output generated by the mapper. The intermediate key-value pairs are sorted by key before starting the reducer phase, and the values can take any order. Sorting by value is done by secondary sorting.

Reducing

In the reducer phase, the intermediate values from the shuffling phase are reduced to produce a single output value that summarizes the entire dataset. HDFS is then used to store the final output.

Here’s an Map Reduce example to count the frequency of each word in an input text. The text is, “This is an apple. Apple is red in color.”

• The input data is divided into multiple segments, then processed in parallel to reduce processing time. In this case, the input data will be divided into two input splits so that work can be distributed over all the map nodes.
• The Mapper counts the number of times each word occurs from input splits in the form of key-value pairs where the key is the word, and the value is the frequency.
• For the first input split, it generates 4 key-value pairs: This, 1; is, 1; an, 1; apple, 1; and for the second, it generates 5 key-value pairs: Apple, 1; is, 1; red, 1; in, 1; color.
• It is followed by the shuffle phase, in which the values are grouped by keys in the form of key-value pairs. Here we get a total of 6 groups of key-value pairs.
• The same reducer is used for all key-value pairs with the same key.
• All the words present in the data are combined into a single output in the reducer phase. The output shows the frequency of each word.
• Here in the example, we get the final output of key-value pairs as This, 1;  is, 2; an, 1; apple, 2; red, 1; in, 1;  color, 1.
• The record writer writes the output key-value pairs from the reducer into the output files, and the final output data is by default stored on HDFS.

Limitations of MapReduce

Map Reduce example also faces some limitations, and they are as follows:

• MapReduce is a low-level programming model which involves a lot of writing code.
• The batch-based processing nature of MapReduce makes it unsuitable for real-time processing.
• It does not support data pipelining or overlapping of Map and Reduce phases.
• Task initialization, coordination, monitoring, and scheduling take up a large chunk of MapReduce’s execution time and reduce its performance.
• MapReduce cannot cache the intermediate data in memory, thereby diminishing Hadoop’s performance.

Conclusion

In this article, we discussed the importance of Map Reduce example for the Hadoop framework. The MapReduce framework has helped us deal with huge amounts of data and find solutions to previously considered impossible problems. In addition to analyzing large data sets in data warehouses, the MapReduce framework can also be applied by companies offering online services to optimize their marketing strategies. The following are our key takeaways from this article:

• MapReduce is fast, scalable, cost-effective, and can handle large volumes and complex data.
• In MapReduce architecture, each phase takes input in the form of key-value pairs.
• It provides parallel computing, fault tolerance, distribution, and security.
• The number of mappers in the mapper phase equals the number of input splits.
• It is recommended that you take 1/4th of the number of mappers as the number of reducers.
• The final output is by default stored in the HDFS storage.

Hopefully, you have understood the architecture of MapReduce and how it works. In case you have any questions, you can leave a comment below or connect with me on LinkedIn.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Frequently Asked Questions