Eclipse Setup for Impala Development

be-cdt-includes.xml

This doc describes how to setup Eclipse for:

• The Java Frontend (FE)
• The C++ Backend (BE)
• The Python test framework

After this setup you will be able to debug the FE and BE simultaneously in Eclipse almost seamlessly.

Prerequisites

For all steps below, you must have sourced $IMPALA_HOME/bin/impala-config.sh and $IMPALA_HOME/bin/set-classpath.sh.

In particular, JAVA_HOME, IMPALA_HOME, IMPALA_HADOOP_VERSION, PYTHON_PATH, and LD_LIBRARY_PATH must be set properly.

Downloading Eclipse

Get the tarball Eclipse IDE for Java Developers from the Eclipse homepage. The instructions in this doc have been confirmed to work with the version Kepler SR2.

Eclipse Memory Settings

For a good experience, it is recommended to allow Eclipse to use a good bit of memory. In your eclipse installation folder, open eclipse.ini and change the existing settings to the following recommended values:

• -Xms512m, -Xmx4g

Eclipse Launch Script

In addition to the sourced scripts above, you must add a build path to LD_LIBRARY_PATH. Easiest is to create a script that does all this each time you start Eclipse. For example, we might create a script "ecl". (Adjust paths for your environment.)

#! /bin/bash                                                                              

export IMPALA_HOME=$HOME/Impala

source $IMPALA_HOME/bin/impala-config.sh

source $IMPALA_HOME/bin/set-classpath.sh

export LD_LIBRARY_PATH=${IMPALA_HOME}/be/build/debug/service:/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH

exec eclipse/java-photon/eclipse/eclipse

Java Frontend (FE)

The FE is a maven project which integrates very nicely with Eclipse. Follow these steps:

1. Create an eclipse project in the shell. From $IMPALA_HOME/fe run: "mvn eclipse:eclipse -DdownloadSources=true". Be patient.
2. Start eclipse. File->Import->General->Existing Project Into Workspace. Navigate to $IMPALA_HOME/fe. Done!

You can verify the import by trying to execute the planner tests in PlannerTest.java. This step requires a running Impala cluster.

FE Debugging

Assuming you started an Impala cluster via $IMPALA_HOME/bin/start-impala-cluster.py you can attach the Java debugger with these steps:

• In Eclipse Run->Debug Configurations->Remote Java Application. Use port 30000 for the first impalad, port 30001 for the second impalad, etc.
• Use port 30000 + N for the catalogd FE assuming you started N impalads.

C++ Backend (BE)

The BE is a CMake project that can be manually integrated using the Eclipse CDT plugin. Follow these steps:

1. Install the Eclipse CDT plugin. In Eclipse go to Help->Install New Software. Add the CDT download site for your version of Eclipse (tested on Kepler). Follow instructions.

   If you have already installed "Eclipse for Java Developers" or other Eclipse packages, you could install the CDT plug-in as follows:
   Launch Eclipse ⇒ Help ⇒ Install New Software ⇒ In "Work with" field, pull down the drop-down menu and select "Kepler - http://download.eclipse.org/releases/kepler" (or juno for Eclipse 4.2; or helios for Eclipse 3.7).
   In "Name" box, expand "Programming Language" node ⇒ Check "C/C++ Development Tools" ⇒ "Next" ⇒ ... ⇒ "Finish".

2. Make sure you have generated the Impala Makefile via CMake. In $IMPALA_HOME run "cmake ."
3. Import the BE project. In Eclipse go to File->Import->C/C++->Existing Code as Makefile Project. Navigate to $IMPALA_HOME/be. Select the "Linux GCC" toolchain.
4. Set the Include Paths for the project. In Eclipse right-click the BE and go to Properties->C/C++ General->Paths and Symbols. Select GNU-C++ and click on "Import Settings". Import this include settings file. Note: the included xml is very obsolete. I use this script to create fresher ones: gen_eclipse_imports.py
   1. If for some reason the above steps did not work, you can manually add the includes via Properties->C/C++ General->Paths and Symbols. You can look in $IMPALA_HOME/be/CMakeFiles/CMakeDirectoryInformation.cmake for the exact includes you need.
5. Rebuild the C++ index to resolve the includes. Project->C/C++ Index->Rebuild. After this step you shouldn't see any undefined macros and/or types anymore.

BE Debugging

1. Ensure proper ptrace settings. On Ubuntu edit /etc/sysctl.d/10-ptrace.conf and set kernel.yama.ptrace_scope = 0.

2. Ensure appropriate .gdbinit in your home directory. It must contain at least the following two lines:

   handle SIGSEGV nostop noprint pass
   handle SIGPIPE nostop noprint pass

3. In Eclipse go to Run->Debug Configurations->C/C++ Attach To Application. Use "build/debug/service/impalad" as the application and your BE project.
4. At the bottom of this menu, click on "Select other..." to switch the "Attach to Process Launcher".
   1. The default GDB DSF launcher does not workwith Impala.
   2. Enable "Use Configuration Specific Settings" and select the Standard Attach To Process Launcher. Click OK.
      
5. On the "Debug" panel of the debug configuration, select the gdb/mi debugger.
   1. As "GDB Command Set" use Standard (Linux)

   2. As "Protocol" use mi/2

      If GDB of impala toolchain cause errors,  you main try use system GDB. At "GDB debugger" use /usr/bin/gdb

6. When you click "Debug" a menu will pop up listing the processes you can attach to. Search for "impalad" and select the one you want to debug.

You may want to enable pretty printing in gdb as described here Eclipse Setup for Impala Development.

Note: the above is for Eclipse Juno. The described 

Python Tests

To develop and debug Python in Eclipse I recommend the PyDev plugin. Follow these steps to setup Eclipse for developing/debugging most Impala Python components:

1. Download and install the PyDev plugin. In Eclipse go to Help->Install New Software. Enter the PyDev update site. Install only the basics.
2. Setup your interpreter. In Eclipse go to Window->Preferences->PyDev->Interpreters->Python Interpreter.
   
   1. If there are existing entries, remove them.
      
   2. Select "Quick Auto-Config". Check that the libraries list what you have in your PYTHON_PATH. Click OK.
3. Create a PyDev project for the Python tests. File->New->Other->PyDev Project.
   1. Use name "tests"
   2. Uncheck "Use default". Navigate to $IMPALA_HOME/tests. Done!BE Debugging.

Debugging Py.test Unit Tests

As an example, follow these instructions to debug the test query_test/test_explain.py in Eclipse:

• In Eclipse go to Run->Debug Configurations. Select "Python unittest".
• Select "tests" as the project and query_test/test_explain.py as the "Main Module".
• In the arguments tab, enable "Override PyUnit preference for this launch?". Select the the Py.test runner.
• Remove the default "–verbosity 0" argument (if present).
• Add an environment variable “LD_LIBRARY_PATH” (via the tab of ENVIRONMENT, then add a new environment variable by clicking on the New button) and then put the following value: “$IMPALA_HOME/toolchain/cdh_components-1009254/kudu-1.10.0-cdh6.x-SNAPSHOT/debug/lib”.
• Debug!
  

Combined FE/BE Debugging

If you followed the FE/BE debugging steps above, then you should have individual Eclipse Debug Configurations for the FE and BE.

You can create a launch group for starting those configurations simultaneously as follows.

• In Eclipse, go to Run->Debug Configurations->Launch Group.
• Add the BE and FE launch configurations.
• Ensure the FE configuration is launched first to avoid issues.

Vim Wrapper for Eclipse

If you are a vi/vim user there is a plugin which permits you to use vim keystrokes in the eclipse editor. In eclipse goto

Help > Install New Software... 

and enter the site:

http://vrapper.sourceforge.net/update-site/stable

It will load for a bit, then select the Vrapper click box and continue. It will load for a bit and then ask if you want to restart. Do so. Documentation is here:

http://vrapper.sourceforge.net/documentation/?page=1