In addition to this guide, also see the latest Flink documentation on the website.
Requirements for developing and building Flink
- Unix-like environment (We use Linux, Mac OS X, and Cygwin)
- Maven (at least version 3.1.1)
- Java 8
Clone the repository
Apache Flink’s source code is stored in a git repository which is mirrored to GitHub. The common way to exchange code on GitHub is to fork the repository into your personal GitHub account. For that, you need to have a GitHub account or create one for free. Forking a repository means that GitHub creates a copy of the forked repository for you. This is done by clicking on the Fork button on the upper right of the repository website. Once you have a fork of Flink’s repository in your personal account, you can clone that repository to your local machine.
git clone https://github.com/<your-user-name>/flink.git
The code is downloaded into a directory called
Azure Pipelines for CI
See the Azure Pipelines Wiki page for details on setting up Azure for CI.
If you are behind a firewall you may need to provide Proxy settings to Maven and your IDE.
For example, the WikipediaEditsSourceTest communicates over IRC and need a SOCKS proxy server to pass.
Setup an IDE and import the source code of Java/Scala
The Flink committers use IntelliJ IDEA (recommended) and Eclipse IDE to develop the Flink Java/Scala code base.
Minimal requirements for an IDE are:
- Support for Java and Scala (also mixed projects)
- Support for Maven with Java and Scala
The IntelliJ IDE supports Maven out of the box and offers a plugin for Scala development.
- IntelliJ download: https://www.jetbrains.com/idea/
- IntelliJ Scala Plugin: http://plugins.jetbrains.com/plugin/?id=1347
Check out our Setting up IntelliJ guide for details.
Eclipse Scala IDE
For Eclipse users, we recommend using Scala IDE 3.0.3, based on Eclipse Kepler. While this is a slightly older version, we found it to be the version that works most robustly for a complex project like Flink.
Further details and a guide to newer Scala IDE versions can be found in the How to setup Eclipse docs.
Note: Before following this setup, make sure to run the build from the command line once (
mvn clean install -DskipTests; see below).
- Download the Scala IDE (preferred) or install the plugin to Eclipse Kepler. See How to setup Eclipse for download links and instructions.
- Add the “macroparadise” compiler plugin to the Scala compiler. Open “Window” -> “Preferences” -> “Scala” -> “Compiler” -> “Advanced” and put into the “Xplugin” field the path to the macroparadise jar file (typically “/home/-your-user-/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar”). Note: If you do not have the jar file, you probably did not run the command line build.
- Import the Flink Maven projects (“File” -> “Import” -> “Maven” -> “Existing Maven Projects”)
- During the import, Eclipse will ask to automatically install additional Maven build helper plugins.
- Close the “flink-java8” project. Since Eclipse Kepler does not support Java 8, you cannot develop this project.
Import the source code
Apache Flink uses Apache Maven as build tool. Most IDEs are capable of importing Maven projects.
Setup an IDE and import the source code of Python
The Flink committers use PyCharm(recommended) to develop Flink Python code base.
The PyCharm is a friendly IDE to supports python development.
- PyCharm download: https://www.jetbrains.com/pycharm/
Check out our Setting up PyCharm guide for details.
Import the source code
The Python code of Apache Flink can be opened by PyCharm directly. The open path should be pointed to the flink-python module.
Build the code
To build Flink from source code, open a terminal, navigate to the root directory of the Flink source code, and call:
mvn clean package
This will build Flink and run all tests(without python test case). Flink is now installed in
To build Flink without executing the tests you can call:
mvn -DskipTests clean package
Run a single/set of tests with maven
If you want to run a single/set of tests you can do this via maven:
mvn verify -Dtest=TestToRun
If you build multiple modules as part of the test run you might want to add
mvn verify -Dtest=TestToRun -pl flink-runtime -am -DfailIfNoTests=false
Note that Flink uses the surefire plugin to run unit and integration tests. That is why you can also specify integration tests via
Run Python Test
After building the Flink source code, you can run Python test in the flink-python module: