Versions Compared

Old Version 36

changes.mady.by.user Michael Hogue

Saved on

compared with

New Version Current

changes.mady.by.user Pierre Villard

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Documentation is created in AsciiDoc.

Running NiFi in Debug mode

While NiFi Mock library can greatly simplify your development and testing efforts, some times it is still necessary to be able to set breakpoints and debug your code within the running instance of NiFi. To do that NiFi needs to be started in Debug mode within your IDE. Please follow the instructions provided in the NiFi IDE Integration module. 

Where to Start?

Finding Issues and Extending the Project

NiFi's JIRA page can be used to find tickets that are tagged as "beginner", or you can dig into any of the tickets for creating Processors. Processors should be self-contained and not rely on other outside components (except for Controller Services), so they make for excellent starting points for new NiFi developers to get started. This exposes the developer to the NiFi API and is the most extensible part of the dataflow system. The NiFi Developer Guide provides details on how to get started with the NiFi API as well as best practices and common approaches to component design.

We have additionally created filters that show various types of issues available to be worked:  

Documentation Contributions

Supported Versions

NiFi automated builds run on the latest updated versions of the following Java Long Term Support releases:

  • Java 21

Minimum Required Versions

For each supported version of Java, NiFi requires the follow minimum update version:

  • Java 21

NiFi automated builds run on the latest updated version of Apache Maven and require the following minimum version, which can be executed using the Maven Wrapper scripts included in the repository:

  • Apache Maven 3.9.2

Running NiFi in Debug mode

While NiFi Mock library can greatly simplify your development and testing efforts, some times it is still necessary to be able to set breakpoints and debug your code within the running instance of NiFi. To do that NiFi needs to be started in Debug mode within your IDE. Please follow the instructions provided in the NiFi IDE Integration module. 

Where to Start?

Finding Issues and Extending the Project

NiFi's JIRA page can be used to find tickets that need effort. The first step to getting involved is to create an account on Jira, then sign up for the Developer Mailing List and send an email to dev@nifi.apache.org requesting "Jira contributor access" and noting your Jira username. This will allow you to assign Jira tickets to yourself to indicate active development efforts.  

Any tickets that are tagged as "beginner", or extension tickets (creating a new processor for interacting with a different system) are good places to start. Processors should be self-contained and not rely on other outside components (except for Controller Services), so they make for excellent starting points for new NiFi developers to get started. This exposes the developer to the NiFi API and is the most extensible part of the dataflow system. The NiFi Developer Guide provides details on how to get started with the NiFi API as well as best practices and common approaches to component design.

We have additionally created filters that show various types of issues available to be worked:  

Documentation Contributions

System-level documentation

...

Code Block
@DynamicProperty( description = "The description of the dynamic property would go here...")

 


and the description method of Component Properties and Relationships

Code Block
PropertyDescriptor.Builder()
        .name("Property Name")
        .description("The description of the Property would go here...")
 
Relationship.Builder()
        .name("name-of-the-relationship")
        .description("The description of the relationship would go here... ")
        .build();

...


When the documentation of a processor is particularly complex (e.g, may include very long descriptions or images), a contributor may chose choose to use NiFi's capability to automatically link additional documentation to a processor.
In this case the contributor may extend documentation by performing changes to 'additionalDetails.html'. As described in the Developer's Guide, this file should exist within a directory whose name is the fully-qualified name of the Processor, and this directory’s parent should be named docs and exist in the 'resources' directory of a particular processor bundle. 

Example of additionalDetails.htm 

As with other contributions, component level documentation should follow the process described in "Providing code or documentation contributions" section of this document.
...
Run into a bug or think there is something that would benefit the project?  Regardless if you have the time to provide the fix or implementation, we encourage any such items to be filed as an issue at the Apache NiFi JIRA.
 

 Anchor
providingCodeOrDocumentationContribution
providingCodeOrDocumentationContribution
Providing code or documentation contributions
...
git config --global user.email user.name@email.org

 
Windows Specific configuration
The following options provide handling of long file paths that can be troublesome as well as not using Windows style line returns.
git config --global core.longpaths true
git config --global core.autocrlf false
 

Clone a copy of the repository
...
git remote add upstream https://github.com/apache/nifi.git 

Checkout the '
...
main' or '
...
1.x.0' branch
git checkout -b master origin/master will main origin/main will create a local, tracking branch named master main.
git checkout -b 0.x origin/01.x.0 will create a local, tracking branch named 0named 1.x.0.
The master branch main branch currently represents the next major release line (1.x). Substantial work here is underway to improve the NiFi's core framework and UI. If you're interested in building the 0.x baseline which is still actively released to include new components and bug fixes, please checkout the 0.x branch. Ultimately contributions will need to be applied to both branches in order to be included in subsequent releasesline (1.x.0). 
The NiFi community uses a modified Gitflow development model.  A summary:
  • Use of a central repository
  • Branch per feature similar to the Feature Branch Workflow
  • Work is done locally and then pushed to the central repo
  • 'mastermain' branch contains the official release history.  Code changes (not code formatting, administrative updates) require Review-Then-Commit (RTC) by another committer to get incorporated.
...
Create a local branch that relates the associated JIRA issue with the branch.  Such an example would be:
git checkout -b nifi-359 mastermain
This provides instant traceability to the supporting issue and provides a means of linking discussion.
...
  • If you've changed a dependency at all (e.g. add, remove, change version), you must verify that the LICENSE and NOTICE from that dependency are contained the nar you're changing (if applicable) and in nifi-assembly. Be mindful to check if the dependency you're adding already exists in the LICENSE and NOTICE files you're editing. If it does, Consider the following guidelines:

      • LICENSE: If the dependency version already exists here and the licenses are identical, you may simply add the additional version (e.g. "The binary distribution of this product bundles 'Google Protocol
        Protocol Buffers Java 2.5.0 and 3.3.1"). If the licenses are different, list them separately.
      • NOTICE: If the dependency version already exists here and the notices are the same, there's no change needed. If the notices are different you need to determine how different they are. If one is a superset of the other, you may simply replace the subset with the superset. If they're vastly different from each other, you need to list them separately.
...
The issue being listed first provides easy access to the supporting ticket and ensures it is not truncated from the varied means in which commits will be viewed. 
Keeping your feature branch current 
If you are working on a branch over an extended period of time, it helps to keep your code current with the master branch main branch to ensure your changes are applied as anticipated and to facilitate the merging process.
This is best accomplished through the git rebase functionality.  There are many ways in which git rebase can be utilized in this context of branching and rebasing.  Typically, the command to do so is performed from your feature branch.  Continuing on with the sample of NIFI-359, we will show one way of accomplishing this task.
Update your local copy of 
...
main
$ git checkout mastermain
Switched to branch 'mastermain'
$ git fetch upstream
remote: Counting objects: 52, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 16 (delta 7), reused 0 (delta 0)
Unpacking objects: 100% (16/16), done.
From https://git-wip-us.apache.org/repos/asf/nifi
 a49a03d..8d745c2 mastermain -> upstream/mastermain
$ git merge upstream/mastermain
Updating a49a03d..8d745c2
Fast-forward
 nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/src/main/java/org/apache/nifi/cluster/protocol/StandardDataFlow.java | 27 ++++++++++-----------------
 1 file changed, 10 insertions(+), 17 deletions(-)
...
$ git checkout nifi-359
Switched to branch 'nifi-359'
$ git rebase mastermain
First, rewinding head to replay your work on top of it...
Applying: NIFI-359: Removing the output to System.out and providing some simple checks to ensure extraction of attributes is consistent with the sample file used in the test.
...
There are currently two paths from which we review and accept contributions, ASF JIRA and GitHub Pull Requests (PRs), regardless of whether your contribution is source code or documentation (note that submitting your contribution  is source code or documentationthrough a pull request via GitHub is the preferred option as it makes the reviews much easier).
Supplying a contribution through patch and JIRA issue
...
$ git remote -v
origin git@github.com:apiri/nifi.git (push)
origin git@github.com:apiri/nifi.git (fetch)
upstream https://git-wip-us.apache.org/repos/asf/nifi.git (fetch)
...

Once the appropriate remote (remote name) is determined, it is possible to push the branch with your changes to GitHub through:
$ git push <remote name> <branch name>
As an example, based on the context of this guide, this would take the form of:
$ git push origin nifi-359
 
This will update your fork of NiFi and can be verified from your GitHub personal project page:
Image Removed
Opening a Pull Request (PR) to the NiFi project
From your personal fork, a Pull Request can be opened simply by selecting the Pull Request link highlighted below.
Image Removed
 
This will allow specification of the your branch which has the changes, and will open a PR for review by the committers of the project.
an example, based on the context of this guide, this would take the form of:
$ git push origin nifi-359

This will update your fork of NiFi and can be verified from your GitHub personal project page:
Image Added
Opening a Pull Request (PR) to the NiFi project
From your personal fork, a Pull Request can be opened simply by selecting the Pull Request link highlighted below.
Image Added

This will allow specification of the your branch which has the changes, and will open a PR for review by the committers of the project.
Image Added
While we request that the PR begin with a single, squashed commit as the initial commit, if you continue to modify the code, please continue to make commits against the same branch. GitHub handles this process well and by pushing to the same remote branch you opened the PR from, your commits will show automatically. This way reviewers can track changes in accordance with their feedback cleanly. Please avoid using squash  or --force when making these changes, as it will overwrite the remote branch and reduce the capability to track these changes. See additional note about Rebasing and Squashing while under review below for more information.  Image Removed
Code Review Process
...
Steps to merge/close pull requests with two main branches
As NiFi now has a 1.0 x (mastermain) and 01.x.y (support) branchbranches, pull requests (PR) must may be applied to bothto both, but typically will only be targetted at main. Here is a step-by-step guide for committers to ensure this occurs for all PRs.
 

  1. Check out the latest master main
    1. $ git checkout mastermain
    2. $ git pull upstream mastermain
  2. Check out the PR (example #327). This will be in detached-HEAD state. (Note: You may need to edit the .git/config file to add the fetch lines below)
...
    1. $ git commit --amend -s
    2. Edit the commit file to contain "This closes #327. "
    3. $ git log
    4. Copy commit id of last commit
  2. Switch back to the master main branch
    1. $ git checkout mastermain
  2. Merge the changes. You can use cherry-pick, merge, etc.
...
    1. $ git log
  2. Push to the Apache repository (master main branch)
    1. $ git push apache mastermain
  2. (Optional) Switch to the support branch - the following steps are only required if the change is specifically being targetted for a patch release of an existing build (e.g. a bug fix to create 1.15.3 from the 1.15.2 build)
    1. $ git checkout -t upstreamsupport/01.x.y
  2. (Optional) Check the status of the branch
    1. $ git log
  2. (Optional) Apply the changes from the PR branch
    1. $ git cherry-pick <commit id>
  2. (Optional) Push to the Apache repository (support branch)
    1. $ git push apache 0support/nifi-1.x.y
 Anchor
fetch-config
fetch-config
Fetch Config
...
[remote "github"]
 url = git@github.com:apache/nifi.git
 fetch = +refs/heads/*:refs/remotes/github/*
 fetch = +refs/pull/*/head:refs/remotes/github/pr/*
Code Style
This is a developing standard and will be updated to reflect an agreed upon format further supported by the maven-checkstyle-plugin.  Our current checkstyle rules can be found here.
TODO: Provide code formatters for the common IDEs
IntelliJ IDEA Users
nifi-checkstyle.xml is a Checkstyle configuration file created by extracting checkstyleRules from nifi/pom.xml. 
Steps for IntelliJ users:
The maven-checkstyle-plugin enforces minimum code style requirements.
The checkstyle.xml configuration in the project root directory contains the current set of style rules.
IntelliJ IDEA Users
  1.  Download Checkstyle-IDEA plugin
  2. Install plugin by loading the downloaded zip from: Preferences -> Plugins -> Install plugin from disk...
  3. After restarting IntelliJ, configure Checkstyle plugin to use NiFi Checkstyle configuration file from: Preferences -> Other Settings -> Checkstyle
  4. Add nifi-checkstyle.xml in in Configuration File and activate it
...
  1. the configuration
Eclipse Users
Code formatter rules that adhere to our checkstyle rules can be found here.
Consider using the The Eclipse Checkstyle Plugin can be configured with our the project checkstyle rules to see what portions of your code are conflicting with our coding standard.
Consider installing these templates that help generate common Apache NiFi code snippets.  TODO: provide link.xml to evaluate conformance with project coding standards.
Contact us!
The developer mailing list (dev@nifi.apache.org) is monitored pretty closely, and we tend to respond quickly.  If you have a question, don't hesitate to shoot us an e-mail - we're here to help! Unfortunately, though, e-mails can get lost in the shuffle, so if you do send an e-mail and don't get a response within a day or two, it's our fault - don't worry about bothering us. Just ping the mailing list again.
...