An Introduction to MapReduce with Map Reduce Example

anushkakhatri Last Updated : 22 Nov, 2024
5 min read

Hadoop facilitates processing large datasets in a distributed manner and provides the foundation for building other services and applications. Map Reduce example and HDFS are the two main components of Hadoop. For a detailed look at HDFS, you can refer to this article: Working of Hadoop Distributed File System(HDFS). Hadoop MapReduce processes the massive amount of structured and unstructured data stored in HDFS. In this article, we will look at MapReduce’s architecture and workflow. We will also discuss some of the benefits and limitations of using Map Reduce.

This article was published as a part of the Data Science Blogathon.

What is MapReduce?

MapReduce is a programming model and framework within the Hadoop ecosystem that enables efficient processing of big data by automatically distributing and parallelizing the computation. It consists of two fundamental tasks: Map and Reduce.

In the Map phase, the input data divides into smaller chunks and processes independently in parallel across multiple nodes in a distributed computing environment. Each chunk transforms or “maps” into key-value pairs by applying a user-defined function. The output of the Map phase is a set of intermediate key-value pairs.

The Reduce phase follows the Map phase. It gathers the intermediate key-value pairs generated by the Map tasks, performs data shuffling to group together pairs with the same key, and then applies a user-defined reduction function to aggregate and process the data. The output of the Reduce phase is the final result of the computation.

Map Reduce example allows for efficient processing of large-scale datasets by leveraging parallelism and distributing the workload across a cluster of machines. It simplifies the development of distributed data processing applications by abstracting away the complexities of parallelization, data distribution, and fault tolerance, making it an essential tool for big data processing in the Hadoop ecosystem.

Advantages of using MapReduce

The advantages of using MapReduce are as follows:

  • MapReduce can define mapper and reducer in several different languages using Hadoop streaming.
  • MapReduce facilitates automatic parallelization and distribution, reducing the time required to run programs.
  • MapReduce provides fault tolerance by re-executing, writing map output to a distributed file system, and restarting failed map or reducer tasks.
  • Processing of data using MapReduce is a cost-effective solution.
  • MapReduce processes large volumes of unstructured data very quickly.
  • Using HDFS and HBase security, Map Reduce ensures data security by allowing only approved users to access data stored in the system.
  • MapReduce programming utilizes a simple programming model to handle tasks more efficiently and quickly and is easy to learn.
  • MapReduce is flexible and works with several Hadoop languages to handle and store data.

MapReduce Architecture

Map Reduce example process has the following phases:

  1. Input Splits
  2. Mapping
  3. Shuffling
  4. Sorting
  5. Reducing
MapReduce

Input Splits

MapReduce splits the input into smaller chunks called input splits, representing a block of work with a single mapper task.

Mapping

The input data is processed and divided into smaller segments in the mapper phase, where the number of mappers is equal to the number of input splits. RecordReader produces a key-value pair of the input splits using TextFormat, which Reducer later uses as input. The mapper then processes these key-value pairs using coding logic to produce an output of the same form.

Shuffling

In the shuffling phase, the output of the mapper phase is passed to the reducer phase by removing duplicate values and grouping the values. The output remains in the form of keys and values in the mapper phase. Since shuffling can begin even before the mapper phase is complete, it saves time.

Sorting

Sorting is performed simultaneously with shuffling. The Sorting phase involves merging and sorting the output generated by the mapper. The intermediate key-value pairs are sorted by key before starting the reducer phase, and the values can take any order. Sorting by value is done by secondary sorting.

Reducing

In the reducer phase, the system reduces the intermediate values from the shuffling phase to produce a single output value that summarizes the entire dataset. The process then uses HDFS to store the final output.

Here’s an Map Reduce example to count the frequency of each word in an input text. The text is, “This is an apple. Apple is red in color.”

MapReduce
  • The input data divides into multiple segments and then processes in parallel to reduce processing time. In this case, we will divide the input data into two input splits to distribute work over all the map nodes.
  • The Mapper counts the number of times each word occurs from input splits in the form of key-value pairs where the key is the word, and the value is the frequency.
  • For the first input split, it generates 4 key-value pairs: This, 1; is, 1; an, 1; apple, 1; and for the second, it generates 5 key-value pairs: Apple, 1; is, 1; red, 1; in, 1; color.
  • It is followed by the shuffle phase, in which the values are grouped by keys in the form of key-value pairs. Here we get a total of 6 groups of key-value pairs.
  • The same reducer handles all key-value pairs with the same key.
  • Combine all the words present in the data into a single output during the reducer phase. The output shows the frequency of each word.
  • Here in the example, we get the final output of key-value pairs as This, 1;  is, 2; an, 1; apple, 2; red, 1; in, 1;  color, 1.
  • The record writer writes the output key-value pairs from the reducer into the output files, and by default, HDFS stores the final output data.

Limitations of MapReduce

Map Reduce example also faces some limitations, and they are as follows:

  • MapReduce is a low-level programming model which involves a lot of writing code.
  • The batch-based processing nature of MapReduce makes it unsuitable for real-time processing.
  • It does not support data pipelining or overlapping of Map and Reduce phases.
  • Task initialization, coordination, monitoring, and scheduling take up a large chunk of MapReduce’s execution time and reduce its performance.
  • MapReduce cannot cache the intermediate data in memory, thereby diminishing Hadoop’s performance.

Conclusion

In this article, we discussed the importance of Map Reduce example for the Hadoop framework. The MapReduce framework has helped us deal with huge amounts of data and find solutions to previously considered impossible problems. In addition to analyzing large data sets in data warehouses, companies offering online services can apply the MapReduce framework to optimize their marketing strategies. The following are our key takeaways from this article:

  • MapReduce is fast, scalable, cost-effective, and can handle large volumes and complex data.
  • In MapReduce architecture, each phase takes input in the form of key-value pairs.
  • It provides parallel computing, fault tolerance, distribution, and security.
  • The number of mappers in the mapper phase equals the number of input splits.
  • It is recommended that you take 1/4th of the number of mappers as the number of reducers.
  • The final output is by default stored in the HDFS storage.

Hopefully, you have understood the architecture of MapReduce and how it works. In case you have any questions, you can leave a comment below or connect with me on LinkedIn.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Frequently Asked Questions

Q1. What is MapReduce in simple words?

A. MapReduce is a programming model that simplifies large-scale data processing. It breaks tasks into smaller “map” functions that process data independently, and then combines results through “reduce” functions. This approach allows parallel computation on multiple machines, speeding up tasks like analyzing vast datasets or counting occurrences. It’s widely used for tasks involving big data and distributed computing.

Q2. What is an example of MapReduce in real life?

A. One real-life example of MapReduce is analyzing social media data. Imagine you want to analyze millions of tweets to find the most common hashtags. The “map” step could involve splitting the data into smaller chunks and counting the occurrences of hashtags in each chunk. Then, the “reduce” step combines these counts from different chunks to find the overall most popular hashtags. This parallel processing significantly speeds up the analysis of massive amounts of data.

Passionate about solving business problems with data-driven solutions. Skilled in the field of Data Science and Big Data Engineering.
I have worked on various projects, including developing predictive models, analyzing complex data sets, and designing and implementing data architectures and pipelines.
I enjoy exploring new data science and data engineering techniques and keeping up with the latest industry trends.

Responses From Readers

Clear

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details