NOTE: This Wiki is obsolete as of November 2016 and is retained for reference only.
Overview
(links: source code | pull requests | open JIRA issues)
Spark's web UIs are implemented using Java Servlets that are hosted inside of a Jetty web server. WebUI is the base class for web UIs and has four concrete implementations:
- The Master and Worker web UIs are used by Spark's standalone cluster manager.
- The Application web UI displays information for an individual Spark driver program. This UI is used in all Spark deployment modes, including
local, Mesos, YARN, and Spark Standalone. - The History Server displays the application UIs for completed applications.
In addition to hosting Spark-specific UIs, the WebUIs also host servlets for the metrics system.
The markup for pages is generated using the scala.xml package.
Application Web UI
The application web UI uses the SparkListenerBus to subscribe to events from the Spark scheduler. Each page registers a SparkListener that receives an event stream and uses it to update its internal state. The listeners' state is read by the WebUIPages when they're rendered.
To allow this UI to persist after applications finish, users can set spark.eventLog.enabled, which causes SparkListener events to be persisted to disk as JSON. The HistoryServer reads and replays this log in order to reconstitute the application web UI.
Links:
Security
The web UI uses Spark's SecurityManager to manage security (this class has good Scaladocs that provide an overview of its architecture). Pages on the web UI can be secured using javax.servlet filters; quoting from SecurityManager:
The Spark UI can also be secured by using javax servlet filters. A user may want to secure the UI if it has data that other users should not be allowed to see. The javax servlet filter specified by the user can authenticate the user and then once the user is logged in, Spark can compare that user versus the view acls to make sure they are authorized to view the UI. The configs 'spark.acls.enable' and 'spark.ui.view.acls' control the behavior of the acls. Note that the person who started the application always has view access to the UI.t
The web UI currently lacks HTTPS support, although there is an open JIRA ticket and pull request to fix this.
Testing
Spark includes a UIWorkloadGenerator tool that continuously runs a mixture of jobs in order to test different features of the web UI, including failure reporting. To run this tool:
./bin/spark-class org.apache.spark.ui.UIWorkloadGenerator [master] [FIFO|FAIR]