...

Tag the pinot docs for the given version

Please ask permission for gitbook Apache Pinot organization access in #pinot-committers slack channel.

...

Check out the codebase (the release branch created while preparing for the release). Add the following dependencies to pinot-distribution/pom.xml (otherwise there will be missing lib dependencies) but don't check it in. (NOTE: Please double check if the following includes all packages in pinot-connectors and pinot-plugins. If not, please update the following code block.)

    <!-- pinot-plugins/pinot-batch-ingestion -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-common</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-hadoop</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-spark-2.4</artifactId>
      <version>${project.version}</version>
    </dependency> 
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-spark-3.2<3</artifactId>
      <version>${project.version}</version>
    </dependency>      
     <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-standalone</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-environment -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-azure</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-file-system -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-adls</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-gcs</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-hdfs</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-s3</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-input-format -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-avro-avro<base</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-avro-base<avro</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-csv<clp-log</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-confluent-avro</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-csv</artifactId>
     <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-json</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-orc</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-parquet</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-protobuf</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-thrift</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-metrics -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-dropwizard</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-yammer</artifactId>
      <version>${project.version}</version>
    </dependency>
 

     <<!-- pinot-plugins/pinot-minion-tasks -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-minion-builtin-tasks</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-segment-uploader -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-segment-uploader-default</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-segment-writer -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-segment-writer-file-based</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-stream-ingestion -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-02.9<0</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-2.0<base</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-base<kinesis</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kinesis<pulsar</artifactId>
      <version>${project.version}</version>
    </dependency>
 
   <!-- pinot-connectors -->
   <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-pulsar<flink-connector</artifactId>
      <version>${project.version}</version>
    </dependency> 

    <!-- pinot-connectors -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-spark-flink2-connector</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-spark-3-connector</artifactId>
      <version>${project.version}</version>
    </dependency> 
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>presto<artifactId>pinot-pinotspark-driver<common</artifactId>
      <version>${project.version}</version>
    </dependency> 

After that, run the commands:

mvn -Ddependency.locations.enabled=false project-info-reports:dependencies -Ppresto-driver -pl :pinot-distribution
# This will create a file in pinot-distribution/target/site/dependencies.html

Open the file dependencies.html (mentioned above) in a browser. You will need to get two parts of the screen onto raw files by copy-pasting it:

...

Open the file dependencies.html (mentioned above) in a browser. You will need to get two parts of the screen onto raw files by copy-pasting it:

1. Create a fresh working directory ($WORKING_DIR), cd to it.
2. Scroll down to the sections on the project's compile dependencies (direct and transitive). Grab the table using the mouse, and save the result to a text file. Name the file pkg-dependencies-raw.txt. Keep the browser page open, you will need it later.
3. Scroll down to the project's runtime dependencies. Grab the table using the mouse and save the result to a text file named runtime-dependencies-raw.txt.
   
   

Do NOT trust the License column in the browser. You will need to manually verify the license in a later step by following the package links in the page.

You can now work on this text file to generate the LICENSE-binary file for the next release

cd $WORKING_DIR
# First, get the new list of dependencies, sorted
cat pkg-dependencies-raw.txt | awk '{printf

...

Do NOT trust the License column in the browser. You will need to manually verify the license in a later step by following the package links in the page.

You can now work on this text file to generate the LICENSE-binary file for the next release

cd $WORKING_DIR
# First, get the new list of dependencies, sorted
cat pkg-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq > /tmp/x1
cat runtime-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq >>> /tmp/x1
sort /tmp/x1 > new-pkg-versions.txt

# Now cat runtime-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq >> /tmp/x1
sort /tmp/x1 > new-pkg-versions.txt

# Now pull out all package version lines from the existing LICENSE-binary file into another file, call it current-pkg-versions.txt.unsorted
cp /path/to/existing/source/LICENSE-binary ./LICENSE-binary
cp LICENSE-binary current-pkg-versions.txt.unsorted
vim current-pkg-versions.txt.unsorted # Now, remove all fluff in the file except for the lines that hold package name and versions.
sort current-pkg-versions.txt.unsorted | uniq > current-pkg-versions.txt

# Now you can compare (diff) the two files to see which packages have been removed, and which ones have been added.

# I use tkdiff, but you can use 'diff -y' or some of the more fancy options of diff that helps you get what you want.
tkdiff current-pkg-versions.txt new-pkg-versions.txt

...

Before executing the step below, this page is a useful read these pages:

http
• https://maven.apache.org/maven-release/maven-release-plugin/
examples
• usage/prepare-release.html
• https://maven.apache.org/maven-release/maven-release-plugin/prepare-mojo.html

...

$ mvn -Darguments=-DskipTests release:perform -Pscala-2.12,release-sign-artifacts -Ppresto-driver
# This step will update Nexus with your release bits.

After running the release:perform  command,

1. Log in to Nexus (https://repository.apache.org)
2. Select Staging Repositories and find the repository with the name "orgapachepinot-{number}".
3. Select that and click the "Close" button.

This will trigger to Nexus to do some checks on artifacts (checksum, signatures..etc). 

After running the release:perform  command,

1. Log in to Nexus (https://repository.apache.org)
2. Select Staging Repositories and find the repository with the name "orgapachepinot-{number}".
3. Select that and click the "Close" button.

This will trigger to Nexus to do some checks on artifacts (checksum, signatures..etc). 

1. Create a source/binary tarballs & Staging source and binary release tarballs

   Code Block
   language

   Create a source/binary tarballs & Staging source and binary release tarballs

   Code Block
   language bash
   $ cd <pinot_source_code_root_path> # Check out the release candidate tag $ git checkout tags/release-$VERSION-rc$RC # Check git hash for the official release $ git log # Create the package $ mvn install -DskipTests -Papache-release,bin-dist,src-dist -T1C # If there is a missing asc file, you can run # gpg -ab apache-pinot-0.11.0-src.tar.gz # to generate apache-pinot-0.11.0-src.tar.gz.asc $ cd pinot-distribution/target $ ls -l ... -rw-r--r-- 1 xiangfu staff 176867304 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz -rw-r--r-- 1 xiangfu staff 833 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz.asc -rw-r--r-- 1 xiangfu staff 128 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz.sha512 -rw-r--r-- 1 xiangfu staff 44435410 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz -rw-r--r-- 1 xiangfu staff 833 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz.asc -rw-r--r-- 1 xiangfu staff 128 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz.sha512 ... # Copy files to the pinot-dev-dist svn repo $ mkdir /path/to/pinot-dev-dist/apache-pinot-$VERSION-rc$RC $ cp apache-pinot-$VERSION-*.tar.gz* /pat/to/pinot-dev-dist/apache-pinot-$VERSION-rc$RC

   
   

2. Validate the release that you built, using Validating a release candidate
3. Commit the files to svn repository

# Commit the files to staging svn repository
$ cd /# Commit the files to staging svn repository
$ cd /path/to/pinot-dev-dist
$ svn add apache-pinot-$VERSION-rc$RC
$ svn commit -m "Update apache-pinot-$VERSION-rc$RC"

...

We need to go through the voting at dev@pinot mailing list. Send out the mail for voting. The example is present at the following links. The voting should be open for at least 72 hours and it requires a minimum of three +1 votes. Once the voting passes, you can send out the result mail. Examples:

...

https://lists.apache.org/thread.html/rdd8d44971e6b4539f40240005cf3a0af1a81e493f2ef2a7e57b360d1%40%3Cdev.pinot.apache.org%3E

out the result mail. Examples:

Once both votings passed, it is now officially approved by Apache and we just need to do some final steps.

Recall a Release Candidate

There could be some issues unveiled during the testing and voting phase. If it happens, we need to recall the release candidate and apply patches/changes then redo from step: Prepare the Release Candidate and increase <rc_num> by 1.

There are several things to clean up:

1. Delete git repo release draft: https://github.com/apache/pinot/releases
2. Delete the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Drop button.

3. Clean up tarballs from dev dist

   Code Block
   language bash
   $ svn rm https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC -m "Delete release-$VERSION-rc$RC" # NOTE: You will need your apache login and password for this step.

   
   

Finalize a Release

1. Move staging tarballs to the svn directory for official release. You need to be a PMC member to be able to do this step. If you are a committer, ask a PMC member to do this step for you.

   Code Block
   language bash
   $ svn mv https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC https://dist.apache.org/repos/dist/release/pinot/apache-pinot-$VERSION # NOTE: You will need your apache login and password for this step.

   Check https://downloads.apache.org/pinot
   After several hours later, check https://archive.apache.org/dist/pinot/apache-pinot-<version>/  to see if it shows up.
   (e.g. https://archive.apache.org/dist/pinot/apache-pinot-0.8.0/)

2. Delete previous release tarballs. Similar to step 1, you need to be PMC member to be able to perform this step.

   We only need to keep the most recent release, so feel free to delete any previous released distributions.

   Code Block
   language bash
   $ svn rm https://dist.apache.org/repos/dist/release/pinot/apache-pinot-0.8.0 # NOTE: You will need your apache login and password for this step.

   
   

3. Delete release branch from github repo if any

   Code Block
   language bash
   $ git push origin --delete release-$VERSION-rc

   
   

4. Update git tag for official release

   Code Block
   language bash
   $ git checkout tags/release-$VERSION-rc$RC $ git tag release-$VERSION -m "Apache Pinot $VERSION" $ git push origin release-$VERSION

   
   

5. Update Release note for the official release on Github https://github.com/apache/pinot/releases

   1. No need to draft a new release, just EDIT the previously drafted release

   2. Replace tag name to release-<version> tag

   3. Uncheck the pre-release checkbox, then publish

   4. Delete all other rc drafts if there is any.

6. Delete all RC tags from github repo

   Code Block
   language bash
   # Clean up the rc tag to not to show on the Github release page $ git tag -d release-$VERSION-rc$RC $ git push origin :refs/tags/release-$VERSION-rc$RC

   
   

7. Publish

Once both votings passed, it is now officially approved by Apache and we just need to do some final steps.

Recall a Release Candidate

There could be some issues unveiled during the testing and voting phase. If it happens, we need to recall the release candidate and apply patches/changes then redo from step: Prepare the Release Candidate and increase <rc_num> by 1.

There are several things to clean up:

1. Delete git repo release draft: https://github.com/apache/pinot/releases
2. Delete the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Drop button.

   Clean up tarballs from dev dist

   Code Block

   language	bash

   $ svn rm Release button. Several hours later, the new version should show up at https://distrepo.maven.apache.org/reposmaven2/dist/devorg/pinot/apache-pinot-$VERSION-rc$RC -m "Delete release-$VERSION-rc$RC" # NOTE: You will need your apache login and password for this step.

Finalize a Release

...

Move staging tarballs to the svn directory for official release. You need to be a PMC member to be able to do this step. If you are a committer, ask a PMC member to do this step for you.

$ svn mv https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC https://dist.apache.org/repos/dist/release/pinot/apache-pinot-$VERSION
# NOTE: You will need your apache login and password for this step.

Check https://downloads.apache.org/pinot
After several hours later, check https://archive.apache.org/dist/pinot/apache-pinot-<version>/  to see if it shows up.
(e.g. https://archive.apache.org/dist/pinot/apache-pinot-0.8.0/)

Delete previous release tarballs. Similar to step 1, you need to be PMC member to be able to perform this step.

We only need to keep the most recent release, so feel free to delete any previous released distributions.

$ svn rm https://dist.apache.org/repos/dist/release/pinot/apache-pinot-0.8.0
# NOTE: You will need your apache login and password for this step.

Delete release branch from github repo if any

$ git push origin --delete release-$VERSION-rc

Update git tag for official release

$ git checkout tags/release-$VERSION-rc$RC
$ git tag release-$VERSION -m "Apache Pinot $VERSION"
$ git push origin release-$VERSION

1. apache/pinot

2. Build docker image for the release:

   1. Please ask Xiang Fu to grant you permission to run the jobs on this repo: https://github.com/apachepinot/pinot-fork/actions.
   2. There are total 5 jobs related to different JDKs(Amazon Corretto 11/17, Microsoft OpenJDK 11/17 and OpenJDK 21)  to run:
      1. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-amazoncorretto-11-pinot-docker-image.yml
      2. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-amazoncorretto-17-pinot-docker-image.yml
      3. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-ms-openjdk-11-pinot-docker-image.yml
      4. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-ms-openjdk-17-pinot-docker-image.yml
      5. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-java21-pinot-docker-image.yml 

   3. Build images with release tag, and tag it with both tags: release-<version>  and <version> , e.g. release-1.2.0  and 1.2.0 . See below graph for the input reference.Image Added

   4. After all the tasks are done, you should find all the docker images here: https://hub.docker.com/r/apachepinot/pinot/tags?page=&page_size=&ordering=&name=release-1.2.0
   5. Tag official release and push, making sure you asked Xiang Fu for the dockerhub push permission for ApachePinot org.

docker manifest create apachepinot/pinot:1.2.0 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-arm64 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-amd64
docker manifest create apachepinot/pinot:release-1.2.0 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-arm64 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-amd64

docker manifest push apachepinot/pinot:1.2.0
docker manifest push apachepinot/pinot:release-1.2.0

...

Update Release note for the official release on Github https://github.com/apache/pinot/releases

1. No need to draft a new release, just EDIT the previously drafted release

2. Replace tag name to release-<version> tag

3. Uncheck the pre-release checkbox, then publish

4. Delete all other rc drafts if there is any.

Delete all RC tags from github repo

# Clean up the rc tag to not to show on the Github release page
$ git tag -d release-$VERSION-rc$RC
$ git push origin :refs/tags/release-$VERSION-rc$RC

...

1. Please ask Xiang Fu to grant you permission to run the job.

2. Build two images one for amd64 arch and one for arm64 arch. See below graph for the input reference.

   Image RemovedImage Removed

Announce to the world

Once you finalized the release, you now need to announce this to the world. 

1. Update https://pinot.apache.org/download page. Example pr: https://github.com/apache/pinot-site/pull/66131
2. Apache takes time to archive all the distributions in SVN release repo, so we don't need to keep all the distributions in SVN release repo. For download URL, we use https://downloads.apache.org for the most recent dist and https://archive.apache.org/dist for old dists.
   1. SVN release repo distribution can be downloaded from: https://downloads.apache.org e.g. https://downloads.apache.org/pinot/apache-pinot-0.9.1/apache-pinot-0.9.1-bin.tar.gz
   2. Old distributions are available through the archive urls, please make sure you update download links for old dist from https://downloads.apache.org to https://archive.apache.org/dist. E.g. https://archive.apache.org/dist/pinot/apache-pinot-0.9.0/apache-pinot-0.9.0-bin.tar.gz Sample PR: https://github.com/apache/pinot-site/pull/61
3. Update Pinot docs: github repo DOAP file to reflect the latest release on https://docs.pinotprojects.apache.org/basics/releasesGuidance on updating the page: project.html?pinot, sample PR: https://docsgithub.pinot.com/apache.org/pinot/pull/11827
4. Update Pinot docs: https://docs.pinot.apache.org/basics/releases
   1. Guidance on updating the page: developers/developers-and-contributors/update-document.Publish versioned docs:
   2. Go to gitbook, select the space previously created, on the right upper corner, click Publish  button and select In Collection.Image Removed
   Check the docs are available at https://docs.pinot.apache.org/developers/. You should be able to find it in the right upper side bar:Image Removeddevelopers-and-contributors/update-document.
   3. Raise a PR similar to (you can raise a single PR): pinot-docs/commit/d1849be pinot-docs/commit/0f035df against the pinot-docs repo. Use the release notes you had used and prepared before.
5. Send the mail to dev@pinot.apache.org about the release announcement
   dev@pinot https://lists.apache.org/thread/qw3chy5rvclymzwk0ros98nlk7mxwhos
6. Send the mail to announce@apache.org  about the release announcement.  Remember to use plain text mode:     

   
   

   

Note that emails to announce@apache.org may get delayed for up to a day, so please be patient

Reference

1. https://cwiki.apache.org/confluence/display/GOBBLIN/Release+Process

...