Debugging Flume

Flume's web applications

As of v0.9.4, the default setup for Flume is to run its servlets from .WAR files that include precompiled jsps.

One can have the node or master start specfic servlets .WARs, by pointing the following properties in the system's flume-site.conf file, like below.

  <property>
    <name>flume.master.webapps.root</name>
    <value>webapps/flumemaster.war</value>
    <description>
    Path where Flume master war lives.  If a file it will load the
    war, if a dir it will load all *.war in that dir.
    </description>
  </property>

  <property>
    <name>flume.node.webapps.root</name>
    <value>webapps/flumemaster.war</value>
    <description>
    Path where Flume node war lives.  If a file it will load the
    war, if a dir it will load all *.war in that dir.
    </description>
  </property>

How do I get the webapps to show up when I am in debugging in eclipse?

You need to build the webapp WAR files and then set configuration values in your flume-site.xml file to point to the appropriate files.

This set of instructions not quite right yet
Alternately you can use symlinks and add pointers to where the compiled WAR files. Eclipse using m2eclipse will execute the FlumeNode program and FlumeMaster program from the ./flume-core dir so you will need to add links to ./conf and ./webapps. Here's some example commands to execute while in flume project root dir.

$ mvn install -DskipTest
... this was successful
$ mkdir flume-core/webapps
$ cd flume-core/webapps
$ ln -s ../../flume-distribution/target/flume-distribution-0.9.5-SNAPSHOT-bin/flume-0.9.5-SNAPSHOT/webapps/flumemaster.war
$ ln -s ../../flume-distribution/target/flume-distribution-0.9.5-SNAPSHOT-bin/flume-0.9.5-SNAPSHOT/webapps/flumeagent.war 

You will also likely need to include a link to the configuration files (0.9.4 release does not have a default webapps pointer so conf files are required).

(from flume project core dir)

$ cd flume-core
$ ln -s ../conf

Tests

Certain tests always fail on my system. (RHEL6)

Some concurrency tests spawn many threads (on the order of 1000), which may be higher than the ulimit for processes on your system. Make sure to up that number or disable that test.

One particular test in v0.9.4 is TestNaiveFileWALManagerConcurrently.testSharedDecoHuge fails with

java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:597)
...

Microbenchmark suite.