Skip to end of metadata
Go to start of metadata


How to contribute to Apache NiFi

We are always excited to have contributions from the community - especially from new contributors!
We are interested in accepting contributions of code, as well as documentation and even artwork that can be applied as icons or styling to the application.

Technologies

The back end of Apache NiFi is written in Java. The web tier makes use of JAX-RS, and JavaScript is extensively used to provide a user interface. We depend on several third-party JavaScript libraries, including D3 and JQuery among others. We make use of Apache Maven for our builds and Git for our version control system.

Documentation is created in AsciiDoc.

Running NiFi in Debug mode

While NiFi Mock library can greatly simplify your development and testing efforts, some times it is still necessary to be able to set breakpoints and debug your code within the running instance of NiFi. To do that NiFi needs to be started in Debug mode within your IDE. Please follow the instructions provided in the NiFi IDE Integration module

Where to Start?

Finding Issues and Extending the Project

NiFi's JIRA page can be used to find tickets that are tagged as "beginner", or you can dig into any of the tickets for creating Processors. Processors should be self-contained and not rely on other outside components (except for Controller Services), so they make for excellent starting points for new NiFi developers to get started. This exposes the developer to the NiFi API and is the most extensible part of the dataflow system. The NiFi Developer Guide provides details on how to get started with the NiFi API as well as best practices and common approaches to component design.

We have additionally created filters that show various types of issues available to be worked:  

Documentation Contributions

System-level and overview documentation is located in '<code checkout location>/nifi-docs/src/main/asciidoc'.
Tools available to facilitate documentation generation are available at Editing AsciiDoc with Live Preview.

 

Create a ticket for new bugs or features

Run into a bug or think there is something that would benefit the project?  Regardless if you have the time to provide the fix or implementation, we encourage any such items to be filed as an issue at the Apache NiFi JIRA.

 

Providing code or documentation contributions

The source is hosted at https://git-wip-us.apache.org/repos/asf/nifi.git

Like all Apache projects, a mirror of the git repository is also located on GitHub at https://github.com/apache/nifi which provides ease in forking and generating pull requests (PRs).

Configure your git client

Ensure your git user name and email are configured

 The following lines ensure your commits are appropriately annotated with your information

git config --global user.name "User Name"
git config --global user.email user.name@email.org

 

Windows Specific configuration

The following options provide handling of long file paths that can be troublesome as well as not using Windows style line returns.

git config --global core.longpaths true
git config --global core.autocrlf false

 

Clone a copy of the repository

From the Apache Hosted Repository

git clone https://git-wip-us.apache.org/repos/asf/nifi.git

From the GitHub Mirror

git clone https://github.com/apache/nifi.git

From your GitHub Fork

git clone git@github.com:<account name>/nifi.git

Retrieval of upstream changes

Additionally, it is beneficial to add a git remote for the mirror to allow the retrieval of upstream changes

git remote add upstream https://github.com/apache/nifi.git

 

Checkout the 'master' or '0.x' branch

git checkout -b master origin/master will create a local, tracking branch named master.

git checkout -b 0.x origin/0.x will create a local, tracking branch named 0.x.

The master branch currently represents the next major release line (1.x). Substantial work here is underway to improve the NiFi's core framework and UI. If you're interested in building the 0.x baseline which is still actively released to include new components and bug fixes, please checkout the 0.x branch. Ultimately contributions will need to be applied to both branches in order to be included in subsequent releases. 

The NiFi community uses a modified Gitflow development model.  A summary:

  • Use of a central repository
  • Branch per feature similar to the Feature Branch Workflow
  • Work is done locally and then pushed to the central repo
  • 'master' branch contains the official release history.  Code changes (not code formatting, administrative updates) require Review-Then-Commit (RTC) by another committer to get incorporated.

For a more in-depth guide, visit Gitflow Workflow by Atlassian as well as the original presentation of the development model by Vincent Driessen.

Perform your code changes

Create a feature branch

Typically, there is a 1-to-1 mapping of a branch to a backing ticket as specified in JIRA.

Create a local branch that relates the associated JIRA issue with the branch.  Such an example would be:

git checkout -b nifi-359 master

This provides instant traceability to the supporting issue and provides a means of linking discussion.

Test your changes

For code changes, ensure that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder.  Please write or update unit tests to verify your changes.

Did your change introduce new dependencies?  Are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?  Did you update the LICENSE file?

For documentation related changes, ensure that format looks appropriate for the output in which it is rendered.

Commit your changes

List the current files that have changes

$ git status
On branch nifi-359
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: nifi-docs/src/main/asciidoc/contributor-guide.adoc
no changes added to commit (use "git add" and/or "git commit -a")

Stage the file(s) to be committed with git add

$ git add nifi-docs/src/main/asciidoc/contributor-guide.adoc
$ git status
On branch nifi-359
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
modified: nifi-docs/src/main/asciidoc/contributor-guide.adoc

In the case of multiple files, it is also possible to stage all tracked files through git add .

Perform the commit

In the interest of providing traceability, it is helpful to lead off your commit with the associated JIRA issue number (case-sensitive, NIFI in uppercase) and a summary of the change this commit is providing. Such an example is:

$ git commit -m "NIFI-359 Adding a guide for procedure and best practices in contributing to the project."

The issue being listed first provides easy access to the supporting ticket and ensures it is not truncated from the varied means in which commits will be viewed.

 

Keeping your feature branch current 

If you are working on a branch over an extended period of time, it helps to keep your code current with the master branch to ensure your changes are applied as anticipated and to facilitate the merging process.

This is best accomplished through the git rebase functionality.  There are many ways in which git rebase can be utilized in this context of branching and rebasing.  Typically, the command to do so is performed from your feature branch.  Continuing on with the sample of NIFI-359, we will show one way of accomplishing this task.

Update your local copy of master

$ git checkout master
Switched to branch 'master'
$ git fetch upstream
remote: Counting objects: 52, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 16 (delta 7), reused 0 (delta 0)
Unpacking objects: 100% (16/16), done.
From https://git-wip-us.apache.org/repos/asf/nifi
a49a03d..8d745c2 master -> upstream/master
$ git merge upstream/master
Updating a49a03d..8d745c2
Fast-forward
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/src/main/java/org/apache/nifi/cluster/protocol/StandardDataFlow.java | 27 ++++++++++-----------------
1 file changed, 10 insertions(+), 17 deletions(-)

Perform the git rebase from your branch

$ git checkout nifi-359
Switched to branch 'nifi-359'
$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: NIFI-359: Removing the output to System.out and providing some simple checks to ensure extraction of attributes is consistent with the sample file used in the test.

Supplying a contribution

There are currently two paths from which we review and accept contributions, ASF JIRA and GitHub Pull Requests (PRs), regardless of whether your contribution is source code or documentation.


Supplying a contribution through patch and JIRA issue

Assuming you are still on the branch you want to make a patch from (NIFI-359) and that you've just committed the change and it is a single commit then you can run the following to generate a patch file

`git format-patch nifi-359~1 -o </desired/path/for/patch>/nifi-359.patch`

Now you can go to JIRA for ticket NIFI-359 and select 'more -> attach files' and upload the patch file.  Then select 'Submit Patch'.

TODO: Once auto-build verification comes into play this guidance will likely change.

Supplying a contribution through a pull request (PR) via GitHub

Push changes to your personal, GitHub repository remote

Typically, this remote is listed as origin. To confirm, perform a git remote -v from within your cloned repository directory.

$ git remote -v
origin git@github.com:apiri/nifi.git (push)
origin git@github.com:apiri/nifi.git (fetch)
upstream https://git-wip-us.apache.org/repos/asf/nifi.git (fetch)

 

Once the appropriate remote (remote name) is determined, it is possible to push the branch with your changes to GitHub through:

$ git push <remote name> <branch name>

As an example, based on the context of this guide, this would take the form of:

$ git push origin nifi-359

 

This will update your fork of NiFi and can be verified from your GitHub personal project page:

Opening a Pull Request (PR) to the NiFi project

From your personal fork, a Pull Request can be opened simply by selecting the Pull Request link highlighted below.

 

This will allow specification of the your branch which has the changes, and will open a PR for review by the committers of the project.

Code Review Process

Apache NIFI has a Review-Then-Commit (RTC) philosophy for handling all contributions.  Reviewers first ensure that the code applies and builds appropriately to the build, passes the code standards as established by the Maven profile "contrib-check."  From here, code is evaluated to ensure best practices within the NiFi framework are applied and, where applicable, that the user experience of interfacing with the contribution is consistent and any changes are backwards compatible.  This process may be iterative but works toward a final product that is then merged into the codebase. 

While only committers can actively promote contributions into the repository, feedback on issues, regardless of committer status, is appreciated and valued in the review process.

If you are interested in facilitating the review process, a listing of all code contributions with a patch are available via a JIRA filter, NIFI Patch Available.

A view of the filter is shown below:

Loading
T Key Summary Assignee Reporter P Status Resolution Created Updated Due
Bug NIFI-3523 WriteAheadProvenanceRepository fails to initialize properly if it contains an invalid Provenance Event Mark Payne Mark Payne Major Patch Available Unresolved Feb 22, 2017 Feb 24, 2017
Improvement NIFI-3497 ScanAttribute should support tagging a flowfile with metadata value from the supplied dictionary Joseph Witt Joseph Witt Major Patch Available Unresolved Feb 17, 2017 Feb 24, 2017
Improvement NIFI-3490 TLS Toolkit - define SAN in standalone mode Pierre Villard Pierre Villard Minor Patch Available Unresolved Feb 16, 2017 Feb 23, 2017
Bug NIFI-3442 FTPTransfer tries to evaluate PORT as expression language, although property doesn't support EL Andre F de Miranda Andre F de Miranda Major Patch Available Unresolved Feb 06, 2017 Feb 16, 2017
Improvement NIFI-3425 Cache prepared statements in PutCassandraQL Unassigned Matt Burgess Minor Patch Available Unresolved Feb 01, 2017 Feb 15, 2017
New Feature NIFI-3414 Implement an EnforceOrder processor Koji Kawamura Matt Burgess Major Patch Available Unresolved Jan 27, 2017 Feb 16, 2017
Improvement NIFI-3339 Add getDataSource() to DBCPService Toivo Adams Toivo Adams Minor Patch Available Unresolved Jan 13, 2017 Feb 03, 2017
Improvement NIFI-3293 Expose counter repository to reporting tasks Pierre Villard Pierre Villard Minor Patch Available Unresolved Jan 06, 2017 Jan 06, 2017
Improvement NIFI-3238 ListenLumberjack should support the *beat protocol Andre F de Miranda Andre F de Miranda Major Patch Available Unresolved Dec 20, 2016 Feb 08, 2017
New Feature NIFI-3196 Provide MiNiFi FlowStatus Insights processor that integrates MiNiFi Flow Status Report with Azure Application Insights Andrew Psaltis Andrew Psaltis Major Patch Available Unresolved Dec 14, 2016 Feb 19, 2017
Bug NIFI-3189 ConsumeKafka 0.9 and 0.10 can cause consumer rebalance when backpressure is engaged Koji Kawamura Joseph Witt Major Patch Available Unresolved Dec 12, 2016 Feb 24, 2017
Bug NIFI-3166 SocketRemoteSiteListener NPE when calling NifiProperties.getRemoteInputHttpPort Koji Kawamura Andre F de Miranda Major Patch Available Unresolved Dec 08, 2016 Feb 14, 2017
Bug NIFI-3163 Flow Fingerprint should include new RPG configurations Koji Kawamura Koji Kawamura Major Patch Available Unresolved Dec 07, 2016 Feb 20, 2017
Bug NIFI-3142 ExtractHL7Attributes processor does not route to REL_FAILURE for an exception other than an HL7Exception Unassigned Kirby Linvill Minor Patch Available Unresolved Dec 02, 2016 Feb 17, 2017
Bug NIFI-3122 Allow templates to be instantiated even if they contain missing processor types Mark Payne Mark Payne Major Patch Available Unresolved Nov 29, 2016 Dec 08, 2016
Sub-task NIFI-3109 Refactor TimerDrivenSchedulingAgent Oleg Zhurakousky Oleg Zhurakousky Minor Patch Available Unresolved Nov 28, 2016 Nov 29, 2016
Improvement NIFI-3093 HIVE Support for QueryDatabaseTable Peter Wicks Peter Wicks Major Patch Available Unresolved Nov 23, 2016 Feb 15, 2017
Bug NIFI-3039 Provenance Repository - Fix PurgeOldEvent and Rollover Size Limits Joe Skora Joe Skora Major Patch Available Unresolved Nov 15, 2016 Feb 21, 2017
Improvement NIFI-3038 Update layout of Cluster <component> Summary window Scott Aslan Rob Moran Minor Patch Available Unresolved Nov 15, 2016 Feb 23, 2017
Bug NIFI-2952 ReplaceText: Search String Validation Error Pierre Villard Justin M Cavinee Minor Patch Available Unresolved Oct 26, 2016 Feb 20, 2017
Showing 20 out of 50 issues Refresh

Additional notes on code contributions

Rebasing and Squashing while under review

Although you may be asked to rebase or squash your contribution as part of the review process, don't feel the need to do so speculatively. The committer working on merging the contribution may prefer to do these types of operations as part of the merge process, and the history of your patch or pull request may aid in the review process.

Steps to merge/close pull requests with two main branches

As NiFi now has a 1.0 (master) and 0.x (support) branch, pull requests (PR) must be applied to both. Here is a step-by-step guide for committers to ensure this occurs for all PRs.

 

  1. Check out the latest master
    1. $ git checkout master
    2. $ git pull upstream master
  1. Check out the PR (example #327). This will be in detached-HEAD state. (Note: You may need to edit the .git/config file to add the fetch lines below)
    1. $ git checkout github/pr/327
  1. Create a branch for the PR
    1. $ git checkout -b pr327
  1. Apply the changes and sign off. This could be through a commit --amend, rebase, etc.
    1. $ git commit --amend -s
    2. Edit the commit file to contain "This closes #327. "
    3. $ git log
    4. Copy commit id of last commit
  1. Switch back to the master branch
    1. $ git checkout master
  1. Merge the changes. You can use cherry-pick, merge, etc.
    1. $ git cherry-pick <commit id>
  1. (Optional) Ensure the commit was applied successfully
    1. $ git log
  1. Push to the Apache repository (master branch)
    1. $ git push apache master
  1. Switch to the support branch
    1. $ git checkout -t upstream/0.x
  1. (Optional) Check the status of the branch
    1. $ git log
  1. Apply the changes from the PR branch
    1. $ git cherry-pick <commit id>
  1. Push to the Apache repository (support branch)
    1. $ git push apache 0.x

Fetch Config

To ensure you are able to pull the PR directly, add the following lines to your .git/config file.

[remote "github"]
url = git@github.com:apache/nifi.git
fetch = +refs/heads/*:refs/remotes/github/*
fetch = +refs/pull/*/head:refs/remotes/github/pr/*

Code Style

This is a developing standard and will be updated to reflect an agreed upon format further supported by the maven-checkstyle-plugin.  Our current checkstyle rules can be found here.

TODO: Provide code formatters for the common IDEs

IntelliJ IDEA Users

nifi-checkstyle.xml is a Checkstyle configuration file created by extracting checkstyleRules from nifi/pom.xml. 

Steps for IntelliJ users:

  1.  Download Checkstyle-IDEA plugin
  2. Install plugin by loading the downloaded zip from: Preferences -> Plugins -> Install plugin from disk...
  3. After restarting IntelliJ, configure Checkstyle plugin to use NiFi Checkstyle configuration file from: Preferences -> Other Settings -> Checkstyle
  4. Add nifi-checkstyle.xml in Configuration File and activate it

You can now see Checkstyle violations in the editor in real-time.

Eclipse Users

Code formatter rules that adhere to our checkstyle rules can be found here.

Consider using the Eclipse Checkstyle Plugin with our checkstyle rules to see what portions of your code are conflicting with our coding standard.

Consider installing these templates that help generate common Apache NiFi code snippets.  TODO: provide link

Contact us!

The developer mailing list (dev@nifi.apache.org) is monitored pretty closely, and we tend to respond quickly.  If you have a question, don't hesitate to shoot us an e-mail - we're here to help! Unfortunately, though, e-mails can get lost in the shuffle, so if you do send an e-mail and don't get a response within a day or two, it's our fault - don't worry about bothering us. Just ping the mailing list again.