Skip to end of metadata
Go to start of metadata

This document comprehensively describes the procedure of running a Pig job using Oozie. Its targeted audience is all forms of users who will install, use and operate Oozie.

Overview

Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. Refer to Pig documentation for information on Pig.

Although Pig jobs can be launched independently, there are obvious advantages on submitting them via Oozie such as:

  • Managing complex workflow dependencies
  • Frequency-based execution
  • Operational flexibility

An execution of a Pig job is referred as a Pig action in Oozie. A Pig action can be specified in the workflow definition (xml) file. The workflow job will wait until the Pig job completes before continuing to the next action.

The Pig action has to be configured with the Pig script and the necessary parameters and configuration to run the Pig job. A Pig script contains Pig Latin statements and Pig commands in a single file. For configuration related to job-tracker, name-node, job-xml etc., refer to section Configuring Actions in the Workflow_ANCHOR_

Syntax of Pig action

The "script" element contains the Pig script to execute.

The "argument" element, if present, contains arguments to be passed to the Pig script. This can be used for parameter substitution and other purposes.

As with Hadoop map-reduce jobs, it is possible to add files and archives to be available to the Pig job, refer to section Adding Files and Archives for your Job_ANCHOR_

Use cases

CASE 1: Launch a simple Pig job

Oozie allows the user to run a Pig job by specifying the Pig script and other necessary arguments. A command line way to launch a Pig job is:

To accomplish the same using Oozie, the complete xml file for specifying the workflow is below:

<job-tracker> element is used to specify the url of the hadoop job tracker.

Format: jobtracker_hostname:port_number

Example: localhost:9001, abc.xyz.yahoo.com:50300

<name-node> element is used to specify the url of the hadoop namenode.

Format: hdfs://namenode_hostname:port_number

Example: hdfs://localhost:9000, hdfs://abc.xyz.yahoo.com:8020

jobtracker and namenode need to be same as the ones defined in the hadoop configuration files. If they are different, they would need to be updated.
<prepare> element is used to specify a list of operations needed to be performed before beginning an action such as deleting an existing output directory (<delete>) or creating a new one (<mkdir>).

<configuration> element is used to specify key/value properties. Some common properties include:
mapred.job.queue.name specifies the queuename that the job will be submitted to. If not mentioned, the default queue default is assumed.

<script> element is used to specify the Pig script

CASE 2: Running a Pig job by passing parameters through command line

Many users want to create a template Pig script and run it with different parameters. This can be accomplished using the -param construct in Pig. For more information, refer to parameter substitution. A command line way to run a Pig job by using the -param construct is:

In order to accomplish the same using Oozie, the <argument> element needs to be included inside the <pig> action.

A partial xml file for such a Pig action:

CASE 3: Running a Pig job by passing parameters through a parameter file

A parameter file can also be used to pass parameters. It is primarily used when the number of parameters to be passed are high. A command line way to run a Pig job by using the -param_file construct to pass parameters through file is:

The are multiple ways of running such a Pig job through Oozie.

a) Using the absolute hdfs path

Partial xml file for Pig action:

Pig expects the parameter file to be a local file. As Oozie runs on compute node, the location of the parameter file in hdfs should be specified.

The Pig action requires the Pig jar file in the hdfs. Such libraries which are used in the workflow can be stored in the lib directory. During runtime, the Oozie server picks up contents of this directory and deploys them on the actual compute node using Hadoop distributed cache. This lib directory has to be manually copied over to the HDFS before the workflow can run.

hadoop fs -put command can be used to copy the files to hdfs.

hadoop fs -ls command can be used to list the files in hdfs.

Layout of application directory in hdfs:

b) Storing the parameter file in "lib" directory

The parameter file can be stored in the lib directory as contents of this directory are automatically added to the classpath by Oozie server.

Partial xml file for Pig action:

Layout of application directory in hdfs:

c) Using the <file> element

Partial xml file for Pig action

The files under the <file> element are added to the distributed cache. To force a symlink for a file, '#' is used followed by the symlink name. Hence, the file /user/ninja/param/paramfile.txt can be accessed locally using the symbolic name symlink.txt. For detailed usage of symbolic link in <file> and <archive>, refer to Adding Files and Archives for the Job (put link)

Layout of application directory in hdfs:

CASE 4: Pig Actions with UDF

Pig provides support for user-defined functions (UDFs) as a way to specify custom processing. Refer to Pig UDF for information on Pig User Defined Functions (UDF)

Following is an example script file using UDF

A command line way to run this Pig job is:

While running through Oozie, the UDF binary has to reside on the compute node.

There are multiple ways of specifying Pig UDF:
a) Using the <archive> element

Specify the name of the customized jar under the <archive> element and use 'REGISTER' in pig script.

Partial xml file using the <archive> element:

The archive tutorial.jar will be placed into a directory by the name udfjar in the current working directory of the tasks. Hence the jar file in the distributed cache will be available locally to the Pig job.

Layout of application directory in hdfs:

b) Using the <file> element

Specify the name of the customized jar under the <file> element and use 'REGISTER' in Pig script.

Following is an example script file

Partial xml file using the <file> element

The files under the <file> element are added to the distributed cache. To force a symlink for a file, '#' is used followed by the symlink name. Hence, the file archive/tutorial.jar can be accessed locally using the symbolic name udfjar.jar. For detailed usage of symbolic link in <file> and <archive>, refer to Adding Files and Archives for the Job (put link)

Layout of application directory in hdfs:

c) Storing the customized jar in "lib" directory

Jars in the lib directory are automatically added to the classpath by Oozie server. So, if the customized jar (tutorial.jar) is in lib directory, then the jar file should not be in <archive> and hence, "REGISTER" should be removed from Pig script.

Partial xml file:

Layout of application directory in hdfs.

HOW TO USE THE OOZIE WEB-CONSOLE

Oozie web-console provides a way to view all the submitted workflow and coordinator jobs in a browser. Each job could be examined in detail to reveal its job configuration, workflow definition and all the actions defined for it. It can be accessed by visiting the url used to submit the job, for eg: http://localhost:4080/oozie.

Note: Please note that the web-console is read-only user interface and it cannot be used to submit a job or modify its status.

Below are some screenshots describing how a job could be drilled down for further details using the web-console.

All the jobs are listed in the grid with filters available above to view the desired job

Clicking a job displays the job details and all actions defined under it.

Each action could be further drilled down by clicking on the browse icon beside the Console URL field.

Hadoop job logs can be viewed

The map task of the launcher job can be accessed by clicking around

Pig produces a sequence of Map-Reduce programs. The details of all these Map-Reduce jobs can be obtained through the task log files.

FAQs

Question: How can one increase the memory for the PIG launcher job?

Answer: You can define a property (oozie.launcher.*) in your action:

  • No labels