Child pages
  • MRUnit Tutorial
Skip to end of metadata
Go to start of metadata

Setup development environment

  1. Download the latest version of MRUnit jar from Apache website: https://repository.apache.org/content/repositories/releases/org/apache/mrunit/mrunit/. For example if you are using the Hadoop version 1.0.3, download mrunit-x.x.x-incubating-hadoop2.jar.
  2. Include the jar in your IDE classpath. Also download the latest verison of mokito (http://code.google.com/p/mockito/) and JUnit jar and add them to your class path of development environment.

Using from Maven add dependency like.

<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>0.9.0-incubating</version>
<classifier>hadoop1</classifier>
</dependency>

Use Classifier as hadoop2 if you are using Hadoop 2 version

Writing MRUnit test cases

MRUnit testing framework is based on JUnit and it can test Map Reduce programs written on 0.20 , 0.23.x , 1.0.x , 2.x version of Hadoop.

Following is an example to use MRUnit to unit test a Map Reduce program that does SMS CDR (call details record) analysis.

The records look like

  1. CDRID;CDRType;Phone1;Phone2;SMS Status Code
    655209;1;796764372490213;804422938115889;6
    353415;0;356857119806206;287572231184798;4
    835699;1;252280313968413;889717902341635;0

The MapReduce program analyzes these records, finds all records with CDRType as 1, and note its corresponding SMS Status Code. For example, the Mapper outputs are

6, 1
0, 1

The Reducer takes these as inputs and output number of times a particular status code has been obtained in the CDR records.

The corresponding Mapper and Reducer are

The corresponding Reducer code is

The MRUnit test class for the Mapper, Reduce and full MapReduce is

Run the test class as JUnit class and it will pass or fail the test depending upon if the mapper is correctly written or not.

Testing Counters

One common use of self-created Counter is to track malformed records in the input.

For example, when the input CDR record is not SMS type, the Mapper can ignore that record and increase the counter.

The revised Mapper with Counter is shown below.

The revised testMapper() method:

When the CDR record is of non SMS type out counter should be incremented by Mapper class , we are checking this by assertion that it is really incremented by one.

Similarly you can test counters for Reducer and its Counter.

Passing arguments for Testing

In next part we would see how to pass arguments to test class using the Configuration class

Configuration parameters are fetched using

Configuration.get() methods in Mapper and Reducer classes

Declare new Configruation object for your test class

In setUp() method add following

Your test class would pass on these parameters to the mappers.

Happy Testing

  • No labels

7 Comments

  1. Anonymous

    Found a typo. Hadoop version 1.0.3 should go with mrunit-x.x.x-incubating-hadoop1.jar instead of mrunit-x.x.x-incubating-hadoop2.jar

  2. I notice that you setup MapReduceDriver but it is not used in the tests.

  3. Anonymous

    I cut and pasted the code. I am getting a value of 0 for the counter. I am using mrunit 0.90 mapreduce1.jar. Any thoughts.

  4. Anonymous

    Does it support MultipleOutputs?

  5. Anonymous

    The map function output more than one records with a input record,can the MRUnit word?

  6. Anonymous

    The map function output more than one records with a input record,can the MRUnit word?

  7. Anonymous

    This article discusses MRUnit in a little bit more detailed and with a lot more example http://m-mansur-ashraf.blogspot.com/2013/02/testing-mapreduce-with-mrunit.html