From time to time debugging a live Flink cluster may be a good choice to understand an issue, or simply to get a go line by line and get a better feeling of how a running Flink system works.
To do so, the Java platform allows developer to attach a debugger to a running JVM.
Passing the remote debugging JVM options
The JVM you wish to attach the debugger to has to be started with a certain set of parameters for the remote debugging capability to be enabled:
The command specifies to open the 5005 port (make sure your firewall poses no problem) and to suspend the process until a debugger is attached to the process. If you wish to allow remote debugging without having to attach a debugger before startup, simply set suspend=n.
Passing the JVM options via env.java.opts
In order to pass the remote debugging options to the JVM you can use Flink's env.java.opts configuration parameter. Simply set in the flink-conf.yaml the following line before starting your Flink cluster.
Passing the JVM options as a dynamic property
When starting Flink in YARN mode via the bin/yarn-session.sh script you can also provide the JVM options via a dynamic property. Simply add the following option to your yarn-session.sh command.
Editing the start-up scripts
Alternatively, you can edit the startup scripts bin/yarn-session.sh (for YARN) or bin/flink-daemon.sh (for standalone cluster) to include the JVM options.
Attach a remote debugger with IntelliJ IDEA
IntelliJ IDEA has a built-in debugger that is compatible with Java's remote debugging specification.
To use it you just have to edit the running configurations (Run > Edit configurations...) and add a new configuration (clicking the + sign on the upper left corner of the dialog) with the "Remote" template. You're free to name this configuration any way you want (perhaps the remote host and port you're connecting to can be a sensible name to use).
To attach to the process you now have to specify (in the "Settings" pane) the remote address and port on which the JVM will be listening for the remote debugger. Running this configuration will attach IntelliJ IDEA to the running JVM (and possibly unsuspend it if you specified suspend=y).
Attach a remote debugger with Eclipse
Similarly to IntelliJ, it is possible to debug a running Flink instance on a remote machine from Eclipse.
These are the required steps:
- Go to the menu 'Run' -> 'Debug configurations...'
- Create a new 'Remote Java Application'
- In the 'Connect' tab choose:
- the project to debug
- Connection type 'Standard (Socket Attach)'
- Connection properties, host (i.e. target hostname or IP address) and port (same port specified before, i.e. 5005)
- (Optional) If you have installed Scala IDE you could see the error 'Multiple launchers available'. Just click on 'Select one' and choose the proper debugger (e.g. Eclipse JDT Launcher)