Prerequisites (One time setup to make sure you have the right permissions)

Before starting the process, you need to ensure a few things are set up correctly. We recommend that you do these things up front anytime (some of these steps may take several days), so that your actual release process takes a shorter time period.

Create PGP key and upload it to public servers

You need to create a PGP key and upload it to public servers. Apache sites pick up the certificate from public repositories once a day.  We strongly suggest that you read through this page first: https://www.apache.org/dev/release-signing.html. Here are the steps involved.

Creating the key

On Mac OS, use gpg and on Linux use gpg2 to generate the key. The GPG key has to be associated with the apache email address. One can create the GPG by using the following cmd. Note that the key length should be 4096 bits. See the example below:

$ gpg2 --full-generate-key
gpg (GnuPG) 2.1.22; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 1

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096

Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)

Key does not expire at all
Is this correct? (y/N) y

...

gpg: key 2C9B34D6A54AAFAE marked as ultimately trusted
...

After you generate the GPG key, copy the key into an environment variable. Look for the line in the output that says "marked as ultimately trusted". For example, in the above output:  "gpg: key 2C9B34D6A54AAFAE marked as ultimately trusted" (From here, I will use GPG_KEY to refer the key).

$ GPG_KEY=<your_gpg_key>  # fill in your key from the "ultimately trusted" line as above

Update the key preference, if necessary

Make sure the key has a preference of SHA512 (otherwise, the key does not show up in apache site). See https://www.apache.org/dev/openpgp.html#key-gen-avoid-sha1

Uploading the generated key

1. Upload GPG public key to a public key server (Use one of the following two: http://pool.sks-keyservers.net:11371/ and http://keyserver.ubuntu.com:11371 as they are used by Apache Nexus server). Copy the following command's output (between the lines that Begin and end certificate, including the BEGIN and END line) to "Submit a key" box and click the submit button. You can also upload it to other sites (See https://www.apache.org/dev/release-signing.html#keyserver for some servers, and also a command-line way to send key to public servers)

   $ gpg --export --armor $GPG_KEY
   
   -----BEGIN PGP PUBLIC KEY BLOCK-----
   ...
   -----END PGP PUBLIC KEY BLOCK-----

2. Update key fingerprint at https://id.apache.org. Type the command given below to display the fingerprint of the key.

   $ gpg --fingerprint $GPG_KEY

   Copy the hex numbers (e.g. "0D9E 9997 4D5A 59F1 7183 362F 2C9B 34D6 A54A AFAE") to "OpenPGP Public Key Primary Fingerprint" box and submit the change. 
   
   

3. Update KEYS file under pinot-release-dist. You need to be a PMC member to be able to run `svn commit`. If you are a committer, ask give the content of the KEYS file to a PMC member to run the command for you.

   $ svn co https://dist.apache.org/repos/dist/release/pinot pinot-release-dist
   $ cd pinot-release-dist
   # Append the public key to KEYS file
   gpg --list-sigs $GPG_KEY >> KEYS && gpg --armor --export $GPG_KEY >> KEYS
   
   # Commit the change
   svn commit -m "Update Pinot KEYS file for $GPG_KEY" 

After following 1-3, you should have the following (after at most 24 hours delay. Certificates are synced from public domain once a day) If your key or fingerprint of the key does not show up correctly, then go back and verify the steps above. Did the key satisfy the SHA512 requirement? Can you try uploading it to other public servers?:

• (Must have)Your public key is listed in https://dist.apache.org/repos/dist/release/pinot/KEYS
• (Optional)Your public key is viewable at https://people.apache.org/keys/committer/<apache_ldap_id>.asc (e.g. https://people.apache.org/keys/committer/snlee.asc)
• (Optional)Your public key is viewable at https://people.apache.org/keys/group/pinot.asc

Note: sometime you may not get the public key listed in the two optional page above. It might be due to Apache site errors and it should not affect the signing.

Check out staging and release distributions

You need to check out Apache staging & release distribution directories for Pinot.

svn co https://dist.apache.org/repos/dist/release/pinot pinot-release-dist
svn co https://dist.apache.org/repos/dist/dev/pinot pinot-dev-dist

Ensure access to Apache repository manager

You need to ensure that you have the access to Apache Repository Manager (Nexus). Try to log in with Apache user name and password.

Ensure your machine can get access to GitHub via SSH

You can test it by running the following command (mentioned on this page):

$ ssh -T git@github.com

Clone Pinot code base

Clone the latest pinot code base in your local machine. Create a branch for your release from the point at which you wish to cut a release. his should be a commit hash that has been tested and verified in reasonable settings (e.g  Pinot software has been tested to run in production with various use cases at a particular commit hash). For example:

$ git clone git@github.com:apache/pinot.git 
$ git checkout -b release-<version> <commit-hash>
$ git push origin release-<version>

# Example
$ git checkout -b release-0.9.0-rc 13c9ee9
$ git push origin release-0.9.0-rc

Run the test, and make sure there are no failures (this can take 2-3 hours)

mvn clean test

Run the compatibility test

cd compatibility-verifier

./checkoutAndBuild.sh -w /tmp/wd-0.7.1 -o release-0.7.1 -n release-0.9.0-rc  

./compCheck.sh -w /tmp/wd-0.7.1 -t sample-test-suite

rm -rf /tmp/wd-0.7.1

## Pinot 0.8.0 didn't shade and release pinot-pulsar correctly, so exclude it for pinot 0.8.0 build.
export PINOT_MAVEN_OPTS='-T1C -pl !:pinot-pulsar' 

./checkoutAndBuild.sh -w /tmp/wd-0.8.0 -o release-0.8.0 -n release-0.9.0-rc 

./compCheck.sh -w /tmp/wd-0.8.0 -t sample-test-suite

 rm -rf /tmp/wd-0.8.0

[ERROR] Failed to execute goal on project pinot-confluent-avro: Could not resolve dependencies for project org.apache.pinot:pinot-confluent-avro:jar:-compat-1628726375: Failed to collect dependencies at io.confluent:kafka-schema-registry-client:jar:5.3.1: Failed to read artifact descriptor for io.confluent:kafka-schema-registry-client:jar:5.3.1: Could not transfer artifact io.confluent:kafka-schema-registry-client:pom:5.3.1 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [confluent (http://packages.confluent.io/maven/, default, releases+snapshots)] -> [Help 1]

<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 http://maven.apache.org/xsd/settings-1.2.0.xsd">
     <mirrors>
          <mirror>
               <id>maven-default-http-blocker</id>
               <mirrorOf>dummy</mirrorOf>
               <name>Dummy mirror to override default blocking mirror that blocks http</name>
               <url>http://0.0.0.0/</url>
               <blocked>true</blocked>
         </mirror>
    </mirrors>
</settings>

Prepare the release notes

You need to prepare good release notes that highlight:

• Important features added in the new release.
• Any deprecated configuration or external methods, noting these these may be removed later.
• Any configuration/methods that were deprecated earlier, and have been removed from this release. 
• Any special notes for upgrading from previous releases (e.g. for the zero-downtime upgrade, please upgrade to release X before this one).
• Other special notes

Here is a command you can use to get all the check-ins since the last release

git log <PREVIOUS_RELEASE_COMMIT_HASH>..<CURRENT_RELEASE_COMMIT_HASH> --oneline | cut -c 11- |  sed 's/^/*/'

e.g.
git log 78152cdb2892cf8c2df5b8a4d04e2aa897333487..02d54b9d986a72e6155014b6e0245d10 --oneline | cut -c 11- |  sed 's/^/*/'
# Assuming 78152cdb2892cf8c2df5b8a4d04e2aa897333487 is the commit hash in the master where the previous release is branched off and 02d54b9d986a72e6155014b6e0245d10 is the commit hash in the master where the current release is branched off.

Once you have the commits, you are ready to come up with good material for release notes.

Draft the release notes in a text file (in markdown syntax) and keep it ready. You will need it in a later step.

Tag the pinot docs for the given version

Please ask permission for gitbook Apache Pinot organization access in #pinot-committers slack channel.

Once you have the access, please duplicate the existing latest docs workspace:

You can update the space name by clicking the title:

Updating LICENSE-binary file

This is a fairly manual process and somewhat time-consuming one. You may need a few hours to complete this step. Familiarize yourself with the contents of LICENSE-binary file and also the files in the licenses-binary directory

The LICENSE-binary file is at the top level of the pinot source code. It has one section for each type of license that a dependency may have. All the binary jars shipped with a license are listed within the section for that license (e.g. there is a section for Apache 2 License, and lists all packages that are shipped with the Apache 2.0 license). Each license type has a file in the licenses-binary/ folder that has the licensing terms (essentially, the text of the license).

Before everything, make sure you build the pinot project so your m2 cache is fulfilled：

Build the project along with presto dependencies:

mvn clean install -DskipTests -Ppresto-driver

Or you can do parallel build but in two commands

mvn clean install -DskipTests -Ppresto-driver -T1C -pl :presto-pinot-driver -am
mvn clean install -DskipTests -T1C

Check out the codebase (the release branch created while preparing for the release). Add the following dependencies to pinot-distribution/pom.xml (otherwise there will be missing lib dependencies) but don't check it in. (NOTE: Please double check if the following includes all packages in pinot-connectors and pinot-plugins. If not, please update the following code block.)

   <!-- pinot-plugins/pinot-batch-ingestion -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-batch-ingestion-common</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-batch-ingestion-hadoop</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-batch-ingestion-spark-2.4</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-batch-ingestion-spark-3</artifactId>
     <version>${project.version}</version>
   </dependency>      
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-batch-ingestion-standalone</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-plugins/pinot-environment -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-azure</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-plugins/pinot-file-system -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-adls</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-gcs</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-hdfs</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-s3</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-plugins/pinot-input-format -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-avro-base</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-avro</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-clp-log</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-confluent-avro</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-csv</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-json</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-orc</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-parquet</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-protobuf</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-thrift</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-plugins/pinot-metrics -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-dropwizard</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-yammer</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-plugins/pinot-minion-tasks -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-minion-builtin-tasks</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-plugins/pinot-segment-uploader -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-segment-uploader-default</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-plugins/pinot-segment-writer -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-segment-writer-file-based</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-plugins/pinot-stream-ingestion -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-kafka-2.0</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-kafka-base</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-kinesis</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-pulsar</artifactId>
     <version>${project.version}</version>
   </dependency>
 
   <!-- pinot-connectors -->
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-flink-connector</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-spark-2-connector</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-spark-3-connector</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-spark-common</artifactId>
     <version>${project.version}</version>
   </dependency>

After that, run the commands:

mvn -Ddependency.locations.enabled=false project-info-reports:dependencies -Ppresto-driver -pl :pinot-distribution
# This will create a file in pinot-distribution/target/site/dependencies.html

Open the file dependencies.html (mentioned above) in a browser. You will need to get two parts of the screen onto raw files by copy-pasting it:

1. Create a fresh working directory ($WORKING_DIR), cd to it.
2. Scroll down to the sections on the project's compile dependencies (direct and transitive). Grab the table using the mouse, and save the result to a text file. Name the file pkg-dependencies-raw.txt. Keep the browser page open, you will need it later.
3. Scroll down to the project's runtime dependencies. Grab the table using the mouse and save the result to a text file named runtime-dependencies-raw.txt.
   
   

Do NOT trust the License column in the browser. You will need to manually verify the license in a later step by following the package links in the page.

You can now work on this text file to generate the LICENSE-binary file for the next release

cd $WORKING_DIR
# First, get the new list of dependencies, sorted
cat pkg-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq > /tmp/x1
cat runtime-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq >> /tmp/x1
sort /tmp/x1 > new-pkg-versions.txt

# Now pull out all package version lines from the existing LICENSE-binary file into another file, call it current-pkg-versions.txt.unsorted
cp /path/to/existing/source/LICENSE-binary ./LICENSE-binary
cp LICENSE-binary current-pkg-versions.txt.unsorted
vim current-pkg-versions.txt.unsorted # Now, remove all fluff in the file except for the lines that hold package name and versions.
sort current-pkg-versions.txt.unsorted | uniq > current-pkg-versions.txt

# Now you can compare (diff) the two files to see which packages have been removed, and which ones have been added.

# I use tkdiff, but you can use 'diff -y' or some of the more fancy options of diff that helps you get what you want.
tkdiff current-pkg-versions.txt new-pkg-versions.txt

Once you have the old and new packages with version numbers, do the following:

1. Identify which packages are no longer in use. Remove them from LICENSE-binary file. If that license is no longer used, then remove the license file from licenses-binary folder.
   1. Tip: use one liner  diff -y current-pkg-versions.txt new-pkg-versions.txt |grep "<"|awk '{print $1}' |xargs -I {} sed -i '' '/{}/d' LICENSE-binary
2. Identify packages that have changed versions. Go the package's home page (as pointed by the browser page that has the dependencies), and verify that the license terms have not changed. If they have, then remove the package name from the  license section, and treat it as a new package dependency (in the next step)
3. Identify all new package dependencies (including those from step 2 above). Click on the package's home page (as pointed by the browser page that has dependencies), and examine the license. You may need to traverse other pages or to the code base in order to finally get the license.
   1. If the license is one of the categories that already exist in the licenses-binary folder, just add the package name and version to the LICENSE-binary file.
   2. If it is a new kind of license
      1. Make sure to create a file in licenses-binary directory copying the license terms from the package's LICENSE file or from its website as the case may be.
      2. Create a new section in LICENSE-binary file just like the sections for other licenses.
      3. copy the package name from new-pkg-versions.txt under the section for that license.
4. Sort the package names in each section of the LICENSE-binary file. This is useful step that makes it easier to identify the changes when building the next release.

Update NOTICE-binary file

This file is a concatenation of all the NOTICE files of packages we use.

In order to generate this file, we need to add a maven-shade-plugin at root pom.xml.

Note that, you need to search <artifactId>maven-shade-plugin</artifactId> in all the submodules pom.xml. Then remove all the found maven-shade-plugin plugins.

Edit pom.xml temporarily (DO NOT CHECKIN)

Change the top level pom.xml as follows:

diff --git a/pom.xml b/pom.xml
index e72be7717..a38f08915 100644
--- a/pom.xml
+++ b/pom.xml
@@ -889,6 +889,31 @@
     <defaultGoal>clean install</defaultGoal>
     <pluginManagement>
       <plugins>
+<plugin>
+  <groupId>org.apache.maven.plugins</groupId>
+  <artifactId>maven-shade-plugin</artifactId>
+  <configuration>
+    <shadedArtifactAttached>false</shadedArtifactAttached>
+    <artifactSet>
+      <includes>
+        <include>*:*</include>
+      </includes>
+    </artifactSet>
+  </configuration>
+  <executions>
+    <execution>
+      <phase>package</phase>
+      <goals>
+        <goal>shade</goal>
+      </goals>
+      <configuration>
+        <transformers>
+          <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer"/>
+        </transformers>
+      </configuration>
+    </execution>
+  </executions>
+</plugin>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-compiler-plugin</artifactId>

Change pinot-distribution/pom.xml to use jar packaging. And of course, you need to remove maven-shade-plugin.

diff --git a/pinot-distribution/pom.xml b/pinot-distribution/pom.xml
index 321a131de..3f7b12541 100644
--- a/pinot-distribution/pom.xml
+++ b/pinot-distribution/pom.xml
@@ -22,7 +22,7 @@
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
-  <packaging>pom</packaging>
+  <packaging>jar</packaging>
   <parent>
     <artifactId>pinot</artifactId>
     <groupId>org.apache.pinot</groupId>

--- a/pinot-distribution/pom.xml
+++ b/pinot-distribution/pom.xml
@@ -144,42 +144,6 @@
           </algorithms>
         </configuration>
       </plugin>
-      <plugin>
-        <artifactId>maven-shade-plugin</artifactId>
-        <executions>
-          <execution>
-            <goals>
-              <goal>shade</goal>
-            </goals>
-            <configuration>
-              <transformers>
-                <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer"/>
-              </transformers>
-              <!--
-              Usually in hadoop environment, there are multiple jars with different versions.
-              Most of the NoSuchMethodExceptions are caused by class loading conflicts.
-              Class relocation ensures the reference of certain packages/classes in Pinot code to
-              shaded libs, e.g. jackson or guava.
-              Ref: https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
-              -->
-              <relocations>
-                <relocation>
-                  <pattern>com.google.common</pattern>
-                  <shadedPattern>shaded.com.google.common</shadedPattern>
-                </relocation>
-                <relocation>
-                  <pattern>com.fasterxml.jackson</pattern>
-                  <shadedPattern>shaded.com.fasterxml.jackson</shadedPattern>
-                </relocation>
-                <relocation>
-                  <pattern>org.apache.http</pattern>
-                  <shadedPattern>shaded.org.apache.http</shadedPattern>
-                </relocation>
-              </relocations>
-            </configuration>
-          </execution>
-        </executions>
-      </plugin>
     </plugins>
   </build>
   <profiles>

Now enter the following commands:

# cd to the top level pinot directory
mvn clean -Ddependency.locations.enabled=false -DskipTests package -Ppresto-driver

cd pinot-distribution/target
unzip pinot-distribution-0.11.0-SNAPSHOT-shaded.jar META-INF/NOTICE
cd ../..
head -6 NOTICE-binary > /tmp/xx
cat pinot-distribution/target/META-INF/NOTICE >> /tmp/xx
cp /tmp/xx NOTICE-binary

git checkout  -- **/pom.xml pom.xml# Revert the changes made temporarily

Lastly, remove the following lines from NOTICE-binary

Apache Pinot
Copyright 2018-2021 Apache Software Foundation

Now, diff the NOTICE-binary file to make sure that it looks like we want, with no extra lines in place.

Submit a diff with LICENSE and NOTICE changes: https://github.com/apache/pinot/pull/5145/files. After merging it to master, pull the change into the release branch.

Maven Configuration Setup

As part of the release process, we will upload our artifacts to maven central staging repository and "maven-release-plugin" will do it for you automatically for the most of part. In order to use the plugin, we need to set up the maven config correctly.

$ vim ~/.m2/settings.xml
# Paste the below text
<settings>
  <servers>
    <server>
      <id>apache.snapshots.https</id>
      <username>{YOUR APACHE USER ID}</username>
      <password>{YOUR APACHE USER PASSWORD IN CLEAR TEXT}</password>
    </server>
    <server>
      <id>apache.releases.https</id>
      <username>{YOUR APACHE USER ID}</username>
      <password>{YOUR APACHE PASSWORD IN CLEAR TEXT}</password>
    </server>

  </servers>
</settings>

Also, We recommend to add the following line to your `~/.bashrc` file.

$ vim ~/.bashrc
# add the following line
export GPG_TTY=$(tty)
$ source ~/.bashrc

Adding above line make the terminal UI correctly show when you need to enter the key phrase for your key when signing the artifacts.

If you are uncomfortable placing your password in clear text in the settings.xml file, you can use http://maven.apache.org/guides/mini/guide-encryption.html. This seems to work fine, at least with maven 3.6.2 (might work with earlier releases as well)

Prepare a Release Candidate

Before we move on to the voting, we need to prepare the release along with the below information:

• GPG Key that the artifacts are signed with - From GPG Key Setup
• Git tag - Step 2
• Git hash for the release - Step 2 (Pick the git hash from the branch created during step 2)
• Release Note - Step 3
• Staging Repository link (Nexus) - Step 4
• Release Candidate under https://dist.apache.org/repos/dist/dev/pinot - Step 5

After following the steps below, you should have all information listed above.

Prepare the release candidate

After this step is successfully completed, you should see two extra entries if you issue a git log command on your release branch. One will be a new tag like release-0.9.0-rc2 and the other is a commit to all pom.xml files changing the working release number to the next version.

Please refer "Preparing a release" section in http://plc4x.apache.org/developers/release.html to see what actually happens with the command below - has a good explanation along with screenshots, our gitbox url for checking is https://gitbox.apache.org/repos/asf?p=pinot.git) 

Before executing the step below, this page is a useful read these pages:

• https://maven.apache.org/maven-release/maven-release-plugin/usage/prepare-release.html
• https://maven.apache.org/maven-release/maven-release-plugin/prepare-mojo.html

Set environment variables for version & release candidate number.

VERSION=0.9.0
RC=0

Note: the scala-maven-plugin in pinot-spark-connector requires us to add the -P option in mvn release command. It's due to the fact that maven-source-plugin and scala-maven-plugin have separate lifecycles, and the gpg files created by the former get deleted by the latter. 

$ mvn -Darguments=-DskipTests release:prepare -Pscala-2.12,release-sign-artifacts -Ppresto-driver
# Be sure to answer the questions as below (example):
[INFO] Checking dependencies and plugins for snapshots ...
What is the release version for "Pinot"? (org.apache.pinot:pinot) 0.9.0: : 
What is SCM release tag or label for "Pinot"? (org.apache.pinot:pinot) pinot-0.9.0: : release-0.9.0-rc0
What is the new development version for "Pinot"? (org.apache.pinot:pinot) 0.9.1-SNAPSHOT: : 0.10.0-SNAPSHOT

After completion of the command above , you should see two commits on your release branch that look like this (example for 0.3.0 release)

$ git log -n 2
commit e41ae3356c9b6891a179ef3bd674906dcd18408a (HEAD -> release-0.3.0-rc)
Author: Xiang Fu <fx19880617@gmail.com>
Date:   Sat Mar 14 21:00:07 2020 -0700

    [maven-release-plugin] prepare for next development iteration

commit 9b2dc20c07dec6cf33df08c4444d996e8202c3ba (tag: release-0.3.0-rc2)
Author: Xiang Fu <fx19880617@gmail.com>
Date:   Sat Mar 14 20:59:46 2020 -0700

    [maven-release-plugin] prepare release release-0.3.0-rc2

# The commit 9b2dc20c07dec6cf33df08c4444d996e8202c3ba is just a tag added, and the commit e41ae3356c9b6891a179ef3bd674906dcd18408a is the modification of all the pom files to be 0.4.0-SNAPSHOT.

You can also verify from https://github.com/apache/pinot that a new tag has appeared and that the commit of the pom files shows up.

Run Compatibility Regression Test

Please run the compatibility test against the previous Apache release and the current release candidate at the following page:

https://github.com/apache/pinot/actions/workflows/pinot_compatibility_tests.yml

Prepare the release notes

You will need to cut-paste the release notes you prepared in the earlier steps. (DO NOT list the entire commit log for release notes).

1. Pull up the site https://github.com/apache/pinot/releases/
2. The new tag created during the release-preparation step above will automatically show up on the page. Click "Draft a new release" button, put "tag version" as the one you used from the previous step (e.g. release-0.3.0-rc2). Change the branch field to your release branch.
3. Fill in the release note (cut/paste)
4. Be sure to check the "This is a pre-release" box.
5. Publish the release.

Update master branch with the next release pom changes

Recall the step in which you typed mvn release:prepare. That step introduced two commits in your release branch. Now is the time you need to take the second commit (the one with the comment [maven-release-plugin] prepare for next development iteration) and merge it back to the master.

Or you can run below command to directly update pom version.

$ mvn --batch-mode release:update-versions -DdevelopmentVersion=<next-version>

# example
$ mvn --batch-mode release:update-versions -DdevelopmentVersion=0.4.0-SNAPSHOT


## NOTE: the above approach using "maven-release-plugin" will break the formatting. If one simply want to change the versions for all pom.xml files, please run the following:

$ PREV_VERSION=0.8.0-SNAPSHOT
$ NEXT_VERSION=0.9.0-SNAPSHOT

# Mac OS
$ find . -name pom.xml | while read i; do cat $i | sed -i '' "s/$PREV_VERSION/$NEXT_VERSION/g" $i ; done

# Linux
$ find . -name pom.xml | while read i; do cat $i | sed -i "s/$PREV_VERSION/$NEXT_VERSION/g" $i ; done

Stage the release

In this step, you will publish the artifacts to "staging repository"

$ mvn -Darguments=-DskipTests release:perform -Pscala-2.12,release-sign-artifacts -Ppresto-driver
# This step will update Nexus with your release bits.

After running the release:perform  command,

1. Log in to Nexus (https://repository.apache.org)
2. Select Staging Repositories and find the repository with the name "orgapachepinot-{number}".
3. Select that and click the "Close" button.

This will trigger to Nexus to do some checks on artifacts (checksum, signatures..etc). 

1. Create a source/binary tarballs & Staging source and binary release tarballs

   $ cd <pinot_source_code_root_path>
   
   # Check out the release candidate tag
   $ git checkout tags/release-$VERSION-rc$RC
   
   # Check git hash for the official release
   $ git log
   
   # Create the package
   $ mvn install -DskipTests -Papache-release,bin-dist,src-dist -T1C
   
   # If there is a missing asc file, you can run
   # gpg -ab apache-pinot-0.11.0-src.tar.gz
   # to generate apache-pinot-0.11.0-src.tar.gz.asc
   
   $ cd pinot-distribution/target
   $ ls -l
   ...
   -rw-r--r--  1 xiangfu  staff  176867304 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz
   -rw-r--r--  1 xiangfu  staff        833 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz.asc
   -rw-r--r--  1 xiangfu  staff        128 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz.sha512
   -rw-r--r--  1 xiangfu  staff   44435410 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz
   -rw-r--r--  1 xiangfu  staff        833 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz.asc
   -rw-r--r--  1 xiangfu  staff        128 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz.sha512
   ...
   
   # Copy files to the pinot-dev-dist svn repo
   $ mkdir /path/to/pinot-dev-dist/apache-pinot-$VERSION-rc$RC
   $ cp apache-pinot-$VERSION-*.tar.gz* /pat/to/pinot-dev-dist/apache-pinot-$VERSION-rc$RC

2. Validate the release that you built, using Validating a release candidate
3. Commit the files to svn repository

# Commit the files to staging svn repository
$ cd /path/to/pinot-dev-dist
$ svn add apache-pinot-$VERSION-rc$RC
$ svn commit -m "Update apache-pinot-$VERSION-rc$RC"

Check https://dist.apache.org/repos/dist/dev/pinot/ to see if the files are uploaded correctly. requires -src/bin.tar.gz, tar.gz.asc, tar.gz.sha512 files.

Once you have all 6 information (gpg key, git tag, git hash, release note, staging repository, staged tarballs), release is now ready for voting process!

Call the Vote

Note that there are ONE Rounds of Approvals to be collected:

We need to go through the voting at dev@pinot mailing list. Send out the mail for voting. The example is present at the following links. The voting should be open for at least 72 hours and it requires a minimum of three +1 votes. Once the voting passes, you can send out the result mail. Examples:

Once both votings passed, it is now officially approved by Apache and we just need to do some final steps.

Recall a Release Candidate

There could be some issues unveiled during the testing and voting phase. If it happens, we need to recall the release candidate and apply patches/changes then redo from step: Prepare the Release Candidate and increase <rc_num> by 1.

There are several things to clean up:

1. Delete git repo release draft: https://github.com/apache/pinot/releases
2. Delete the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Drop button.

3. Clean up tarballs from dev dist

   $ svn rm https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC -m "Delete release-$VERSION-rc$RC"
   # NOTE: You will need your apache login and password for this step.

Finalize a Release

1. Move staging tarballs to the svn directory for official release. You need to be a PMC member to be able to do this step. If you are a committer, ask a PMC member to do this step for you.

   $ svn mv https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC https://dist.apache.org/repos/dist/release/pinot/apache-pinot-$VERSION
   # NOTE: You will need your apache login and password for this step.

   Check https://downloads.apache.org/pinot
   After several hours later, check https://archive.apache.org/dist/pinot/apache-pinot-<version>/  to see if it shows up.
   (e.g. https://archive.apache.org/dist/pinot/apache-pinot-0.8.0/)

2. Delete previous release tarballs. Similar to step 1, you need to be PMC member to be able to perform this step.

   We only need to keep the most recent release, so feel free to delete any previous released distributions.

   $ svn rm https://dist.apache.org/repos/dist/release/pinot/apache-pinot-0.8.0
   # NOTE: You will need your apache login and password for this step.

3. Delete release branch from github repo if any

   $ git push origin --delete release-$VERSION-rc

4. Update git tag for official release

   $ git checkout tags/release-$VERSION-rc$RC
   $ git tag release-$VERSION -m "Apache Pinot $VERSION"
   $ git push origin release-$VERSION

5. Update Release note for the official release on Github https://github.com/apache/pinot/releases

   1. No need to draft a new release, just EDIT the previously drafted release

   2. Replace tag name to release-<version> tag

   3. Uncheck the pre-release checkbox, then publish

   4. Delete all other rc drafts if there is any.

6. Delete all RC tags from github repo

   # Clean up the rc tag to not to show on the Github release page
   $ git tag -d release-$VERSION-rc$RC
   $ git push origin :refs/tags/release-$VERSION-rc$RC

7. Publish the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Release button. Several hours later, the new version should show up at https://repo.maven.apache.org/maven2/org/apache/pinot

8. Build docker image for the release:

   1. Please ask Xiang Fu to grant you permission to run the jobs on this repo: https://github.com/apachepinot/pinot-fork/actions.
   2. There are total 5 jobs related to different JDKs(Amazon Corretto 11/17, Microsoft OpenJDK 11/17 and OpenJDK 21)  to run:
      1. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-amazoncorretto-11-pinot-docker-image.yml
      2. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-amazoncorretto-17-pinot-docker-image.yml
      3. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-ms-openjdk-11-pinot-docker-image.yml
      4. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-ms-openjdk-17-pinot-docker-image.yml
      5. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-java21-pinot-docker-image.yml 

   3. Build images with release tag, and tag it with both tags: release-<version>  and <version> , e.g. release-1.2.0  and 1.2.0 . See below graph for the input reference.

   4. After all the tasks are done, you should find all the docker images here: https://hub.docker.com/r/apachepinot/pinot/tags?page=&page_size=&ordering=&name=release-1.2.0
   5. Tag official release and push, making sure you asked Xiang Fu for the dockerhub push permission for ApachePinot org.

docker manifest create apachepinot/pinot:1.2.0 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-arm64 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-amd64
docker manifest create apachepinot/pinot:release-1.2.0 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-arm64 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-amd64

docker manifest push apachepinot/pinot:1.2.0
docker manifest push apachepinot/pinot:release-1.2.0

Announce to the world

Once you finalized the release, you now need to announce this to the world. 

1. Update https://pinot.apache.org/download page. Example pr: https://github.com/apache/pinot-site/pull/131
2. Apache takes time to archive all the distributions in SVN release repo, so we don't need to keep all the distributions in SVN release repo. For download URL, we use https://downloads.apache.org for the most recent dist and https://archive.apache.org/dist for old dists.
   1. SVN release repo distribution can be downloaded from: https://downloads.apache.org e.g. https://downloads.apache.org/pinot/apache-pinot-0.9.1/apache-pinot-0.9.1-bin.tar.gz
   2. Old distributions are available through the archive urls, please make sure you update download links for old dist from https://downloads.apache.org to https://archive.apache.org/dist. E.g. https://archive.apache.org/dist/pinot/apache-pinot-0.9.0/apache-pinot-0.9.0-bin.tar.gz Sample PR: https://github.com/apache/pinot-site/pull/61
3. Update Pinot github repo DOAP file to reflect the latest release on https://projects.apache.org/project.html?pinot, sample PR: https://github.com/apache/pinot/pull/11827
4. Update Pinot docs: https://docs.pinot.apache.org/basics/releases
   1. Guidance on updating the page: https://docs.pinot.apache.org/developers/developers-and-contributors/update-document.
   2. Raise a PR similar to (you can raise a single PR): pinot-docs/commit/d1849be pinot-docs/commit/0f035df against the pinot-docs repo. Use the release notes you had used and prepared before.
5. Send the mail to dev@pinot.apache.org about the release announcement
   dev@pinot https://lists.apache.org/thread/qw3chy5rvclymzwk0ros98nlk7mxwhos
6. Send the mail to announce@apache.org  about the release announcement.  Remember to use plain text mode:     

   
   

   

Note that emails to announce@apache.org may get delayed for up to a day, so please be patient

Reference

1. https://cwiki.apache.org/confluence/display/GOBBLIN/Release+Process