Collecting Impala CPU and Heap Profiles

Impala implements the pprof remote server handlers to make easy to collect CPU and heap profiles from a running Impala (statestore or impalad) services using pprof.

There are currently three supported profiles:

  • heap profile (exported under: hostname:25000/pprof/heap)
  • heap-profiling delta (heap growth) (exported at: hostname:25000/pprof/growth)
  • cpu profile (exported at: hostname:25000/pprof/profile)

How to Collect Profile Data

The best way to collect profile data is the run pprof on a machine that has access to the Impala debug web server. Currently, Impala does not fully implement symbol resolution. To get symbolized results, you must pass in a matching impalad binary to pprof. This can be done by either copying remote server binary locally or running pprof on the remote machine. See usage examples below.

In a pinch (e.g. running on a locked down server) you can get the profile with curl or wget, e.g.

wget http://hostname:25000/pprof/profile
wget http://hostname:25000/pprof/heap
wget http://hostname:25000/pprof/growth

# If you are running a test cluster or similar with self-signed certs, you may need to pass an extra flag.
wget --no-check-certificate https://hostname:25000/pprof/heap

Installing Google Perf Tools

You will need pprof, which is part of the Google Perf Tools suite: https://github.com/gperftools/gperftools

This is available as an OS package on many distributions:

  • Ubuntu: sudo apt-get install google-perftools  (NOTE: pprof is renamed google-pprof on Ubuntu)
  • Centos/RHEL: yum install google-perftools

It is also included in the Impala toolchain under: gperftools-${IMPALA_GPERFTOOLS_VERSION}/

CPU Profile
To collect a CPU profile run:

# This defaults to sampling over a 30 second time period
pprof --text /path/to/impalad http://hostname:25000/pprof/profile

# To produce a PDF with the call graph:
pprof --pdf /path/to/impalad http://hostname:25000/pprof/profile > profile.pdf

# To sample over a longer/shorter period of time set the "seconds" argument.
pprof --text /path/to/impalad http://hostname:25000/pprof/profile?seconds=30

Heap Profile
To collect a heap profile run:

# This defaults to sampling over a 30 second time period
pprof --text /path/to/impalad http://hostname:25000/pprof/heap

# To sample over a longer/shorter period use the "seconds" argument:
pprof --text /path/to/impalad http://hostname:25000/pprof/heap?seconds=60

Heap Growth Profile
To collect a heap growth profile run:

pprof --text /path/to/impalad http://hostname:25000/pprof/growth

Example Output
This is an example of what each result should look like if everything is setup properly:

pprof  --text impalad http://localhost:25000/pprof/heap?seconds=5
  /home/lskuff/pprof/impalad.1367340295.localhost
Be patient...
Wrote profile to /home/lskuff/pprof/impalad.1367340295.localhost
Total: 35.9 MB
    32.0  89.2%  89.2%     32.0  89.2% impala::DiskIoMgr::GetFreeBuffer
     2.8   7.7%  96.9%      2.8   7.7% os::malloc
     1.0   2.8%  99.7%      1.0   2.8% NoBarrier_CompareAndSwap (inline)
     0.0   0.1%  99.8%      0.0   0.1% ObjectSynchronizer::inflate
     0.0   0.0%  99.8%      0.0   0.0% llvm::PassNameParser::passRegistered
     0.0   0.0%  99.9%      0.0   0.0% std::basic_string::_Rep::_S_create


pprof  --text impalad http://localhost:25000/pprof/profile?seconds=5
Wrote profile to /home/lskuff/pprof/impalad.1367340402.localhost
Removing _L_unlock_16 from all stack traces.
Total: 26 samples
       1   3.8%   3.8%        1   3.8% 0x00007f0a2faaf6b2
       1   3.8%   7.7%        1   3.8% 0x00007f0a2faaf6ce
       1   3.8%  11.5%        1   3.8% 0x00007f0a2fab9314
       1   3.8%  15.4%        1   3.8% 0x00007f0a2fabe301
       1   3.8%  19.2%        1   3.8% 0x00007f0a2fabe647
       1   3.8%  23.1%        1   3.8% 0x00007f0a2fad85e0
       1   3.8%  26.9%        1   3.8% 0x00007f0a2fae15b0
       1   3.8%  30.8%        1   3.8% ::ValueTable::create_expression
       1   3.8%  34.6%        1   3.8% ObjectMonitor::TrySpin_VaryDuration
       1   3.8%  38.5%        2   7.7% PointerIntPair
       1   3.8%  42.3%        1   3.8% SymbolTable::lookup@7e75e0
       1   3.8%  46.2%        3  11.5% ValueHandleBase
       1   3.8%  50.0%        1   3.8% __read_nocancel
       1   3.8%  53.8%        1   3.8% constantPoolOopDesc::klass_at_impl
       1   3.8%  57.7%        1   3.8% llvm::APInt::clearUnusedBits
       1   3.8%  61.5%        1   3.8% llvm::PassRegistry::getPassInfo
       1   3.8%  65.4%        1   3.8% llvm::PassRegistry::getPassRegistry
       1   3.8%  69.2%        1   3.8% llvm::PointerIntPair::setInt


pprof  --text impalad http://localhost:25000/pprof/growth
Wrote profile to /home/lskuff/pprof/impalad.1367340508.localhost.pprof.growth
Total: 102.0 MB
    50.9  49.9%  49.9%     50.9  49.9% os::malloc
    40.0  39.2%  89.2%     40.0  39.2% impala::DiskIoMgr::GetFreeBuffer
     5.0   4.9%  94.1%      5.0   4.9% zcalloc
     2.1   2.0%  96.1%      2.1   2.0% ProfileData::Start
     2.0   2.0%  98.0%      2.0   2.0% readCEN
     2.0   2.0% 100.0%      2.0   2.0% std::basic_string::_Rep::_S_create
     0.0   0.0% 100.0%      1.0   1.0% 0x00007fff49db0801
     0.0   0.0% 100.0%      8.1   8.0% Arena::Amalloc
     0.0   0.0% 100.0%      3.0   2.9% Arena::Arealloc
     0.0   0.0% 100.0%     40.9  40.1% Arena::grow
     0.0   0.0% 100.0%     40.9  40.1% C2Compiler::compile_method
     0.0   0.0% 100.0%     40.9  40.1% Chunk::operator new
     0.0   0.0% 100.0%      7.0   6.9% ChunkPool::allocate
     0.0   0.0% 100.0%     10.4  10.2% Compile::Code_Gen
  • No labels