Requirements

Docker Hub Write Access

In order to publish images to the official Apache NiFi Docker Hub repositories, you must:

If you are an Apache NiFi PMC member, you can request this from ASF Infra. Here is an example: INFRA-20726 - Getting issue details... STATUS

Then, you must be logged into the Docker Hub from the command line, which you can do using: docker login

Build Env Requirements

Any typical Linux/Unix/Mac NiFi dev environment should suffice. You will need:

  • git to clone the apache/nifi source code repository
  • docker command line tools for building and pushing images from the Dockerfiles.
  • bash to run the Docker image build & publish automation script

Note: multi-platform Docker images do not require the build machine to match the target architecture(s). Docker buildx (included with docker desktop) provides the emulation necessary to build multi-platform images from any host architecture. An x86_64 or arm64 development machine should be suitable for this process.

Apache NiFi

The following instructions are for building and publishing official Docker Hub images from the apache/nifi source code repository, which includes:

For instructions for building and publishing the Apache NiFi MiNiFi C++ image, see Apache NiFi MiNiFi C++ below.

  1. Clone the apache/nifi source code repository:

    git clone git@github.com:apache/nifi.git

  2. Checkout the release tag for the version you want to build. For example, to checkout Apache NiFi 1.21.0, you would do:

    git checkout rel/nifi-1.21.0

  3. Run the following script to build and push all the Docker images:

    publish-nifi-docker.sh
    #!/bin/sh

set -euxo pipefail

nifi_dir='/Users/jdoe/code/nifi'
version='1.21.0'
#        ^^^^^^
# MAKE SURE TO CHANGE THE VERSION ABOVE
# MAKE SURE TO CHECKOUT THE rel/nifi-${version} TAG PRIOR TO RUNNING THIS SCRIPT

dockerhub_org='apache'
download_location=https://downloads.apache.org
#download_location=https://archive.apache.org/dist

# NiFi
nifi_repo="${dockerhub_org}/nifi"
nifi_docker_dir="${nifi_dir}/nifi-docker/dockerhub"
nifi_dockerfile="${nifi_docker_dir}/Dockerfile"

# Toolkit
nifi_toolkit_repo="${dockerhub_org}/nifi-toolkit"
nifi_toolkit_docker_dir="${nifi_dir}/nifi-toolkit/nifi-toolkit-assembly/docker"
nifi_toolkit_dockerfile="${nifi_toolkit_docker_dir}/Dockerfile.hub"

# MiNiFi
minifi_repo="${dockerhub_org}/nifi-minifi"
minifi_docker_dir="${nifi_dir}/minifi/minifi-docker/dockerhub"
minifi_dockerfile="${minifi_docker_dir}/Dockerfile"

# MiNiFi C2
c2_repo="${dockerhub_org}/nifi-minifi-c2"
c2_docker_dir="${nifi_dir}/minifi/minifi-c2/minifi-c2-docker/dockerhub"
c2_dockerfile="${c2_docker_dir}/Dockerfile"

# Registry
registry_repo="${dockerhub_org}/nifi-registry"
registry_docker_dir="${nifi_dir}/nifi-registry/nifi-registry-core/nifi-registry-docker/dockerhub"
registry_dockerfile="${registry_docker_dir}/Dockerfile"

build_image () {
  dockerfile=$1
  build_context=$2
  docker_repo=$3
  build_args=$4
  platforms="linux/amd64,linux/arm64"

  docker buildx build --push --platform "${platforms}" \
    ${build_args} \
    --tag "${docker_repo}:${version}" \
    --tag "${docker_repo}:latest" \
    --file "${dockerfile}" \
    "${build_context}"
}

cleanup () {
  docker buildx rm nifi-builder
}

trap cleanup EXIT

docker buildx create --use --name nifi-builder --driver docker-container

echo 'Building & publishing NiFi'
build_image ${nifi_dockerfile} ${nifi_docker_dir} ${nifi_repo} "--build-arg NIFI_VERSION=${version} --build-arg BASE_URL=${download_location}"
echo 'Finished publishing NiFi'

echo 'Building & publishing NiFi Toolkit'
build_image ${nifi_toolkit_dockerfile} ${nifi_toolkit_docker_dir} ${nifi_toolkit_repo} "--build-arg NIFI_TOOLKIT_VERSION=${version} --build-arg MIRROR=${download_location}"
echo 'Finished publishing toolkit'

echo 'Building & publishing MiNiFi'
build_image ${minifi_dockerfile} ${minifi_docker_dir} ${minifi_repo} "--build-arg MINIFI_VERSION=${version} --build-arg MIRROR=${download_location}"
echo 'Finished publishing MiNiFi'

echo 'Building & publishing MiNiFi C2'
build_image ${c2_dockerfile} ${c2_docker_dir} ${c2_repo} "--build-arg MINIFI_C2_VERSION=${version} --build-arg MIRROR=${download_location}"
echo 'Finished publishing MiNiFi C2'

echo 'Building & publishing NiFi Registry'
build_image ${registry_dockerfile} ${registry_docker_dir} ${registry_repo} "--build-arg NIFI_REGISTRY_VERSION=${version} --build-arg MIRROR=${download_location}"
echo 'Finished publishing NiFi Registry'

echo "Don't forget to manually update the README files on docker hub!"

    Note: A copy of this script is also hosted (and updated) as a GitHub gist and may find a home in an Apache NiFi source code repository in the future:
    https://gist.github.com/kevdoran/0bcb26c2f7435d7d53cc93c4e4393b52

  4. Manually update the descriptions for each Docker Hub image repository to match the latest README contents for each image. These are typically in the same directory as the Dockerfile for the image. For example, the README file for the apache/nifi Dockerfile can be found at: https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/README.md. Note that these change infrequently, so this step is not required if there is nothing to update. It also might be possible to automate this process as part of the publish script in the future.

Apache NiFi MiNiFi C++

The following instructions are for building and publishing the official Apache NiFi MiNiFi C++ image from the apache/nifi-minifi-cpp source code repository to Docker Hub at apache/nifi-minifi-cpp.

TODO


  • No labels