To profile DAGAppMaster, we can attach Yourkit agent to it by following these steps:
- Download Yourkit tar on each node of your cluster (since you don't know where AM will be launched) and extract files.
- Add following property in tez-site.xml:
<property> <name>tez.am.launch.cmd-opts</name> <value>-agentpath:<absolute path of libyjpagent.so>=<yourkit options></value> </property>
libyjpagent.so can be found in <Yourkit-dir>/bin/<architecture>/libyjpagent.so.
Yourkit options can be found by running
java -agentpath:<path of libyjpagent.so>=help <some-java-class>
If you are capturing snapshot of DAGAppMaster, snapshot files can be found under your home dir in 'Snapshot' dir. If you submit YARN job as user yarn, you need go to /home/yarn/Snapshots.
Profiling task containers is similar to profiling DAGAppMaster, the only difference is you should use property 'tez.task.launch.cmd-opts' (for all tasks) or 'tez.task-specific.launch.cmd-opts' (for specific task) in tez-site.xml.