Versions Compared

Old Version 114

changes.mady.by.user Xiang Fu

Saved on

compared with

New Version Current

changes.mady.by.user Xiang Fu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

pi

Table of Contents

Info

It is useful to go through this document fully before getting started with the release.7199AF89

Expect that the whole release process could take 10+ hours and span a few days.

...

On Mac OS, use gpg and on Linux use gpg2 to generate the key. The GPG key has to be associated with the apache email address. One can create the GPG by using the following cmd. Note that the key length should be 4096 bits. See the example below:

Code Block
languagebash

$ gpg2 --full-generate-key
gpg (GnuPG) 2.1.22; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 1

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096

Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)

Key does not expire at all
Is this correct? (y/N) y

...

gpg: key 2C9B34D6A54AAFAE marked as ultimately trusted
...

...

  1. Upload GPG public key to a public key server (Use one of the following two: http://pool.sks-keyservers.net:11371/ and http://keyserver.ubuntu.com:11371 as they are used by Apache Nexus server). Copy the following command's output (between the lines that Begin and end certificate, including the BEGIN and END line) to "Submit a key" box and click the submit button. You can also upload it to other sites (See https://www.apache.org/dev/release-signing.html#keyserver for some servers, and also a command-line way to send key to public servers)

    Code Block
    languagebash
    $ gpg --export --armor $GPG_KEY

-----BEGIN PGP PUBLIC KEY BLOCK-----
...
-----END PGP PUBLIC KEY BLOCK-----


  2. Update key fingerprint at https://id.apache.org. Type the command given below to display the fingerprint of the key.

    Code Block
    languagebash
    $ gpg --fingerprint $GPG_KEY

    Copy the hex numbers (e.g. "0D9E 9997 4D5A 59F1 7183 362F 2C9B 34D6 A54A AFAE") to "OpenPGP Public Key Primary Fingerprint" box and submit the change. 

  3. Update KEYS file under pinot-release-dist. You need to be a PMC member to be able to run `svn commit`. If you are a committer, ask give the content of the KEYS file to a PMC member to run the command for you.

    Code Block
    languagebash
    $ svn co https://dist.apache.org/repos/dist/release/pinot pinot-release-dist
$ cd pinot-release-dist
# Append the public key to KEYS file
gpg --list-sigs $GPG_KEY >> KEYS && gpg --armor --export $GPG_KEY >> KEYS

# Commit the change
svn commit -m "Update Pinot KEYS file for $GPG_KEY" 


...

Code Block
languagebash
cd compatibility-verifier

./checkoutAndBuild.sh -w /tmp/wd-0.7.1 -o release-0.7.1 -n release-0.9.0-rc  

./compCheck.sh -w /tmp/wd-0.7.1 -t sample-test-suite

rm -rf /tmp/wd-0.7.1

## Pinot 0.8.0 didn't shade and release pinot-pulsar correctly, so exclude it for pinot 0.8.0 build.
export PINOT_MAVEN_OPTS='-T1C -pl !:pinot-pulsar' 

./checkoutAndBuild.sh -w /tmp/wd-0.8.0 -o release-0.8.0 -n release-0.9.0-rc 

./compCheck.sh -w /tmp/wd-0.8.0 -t sample-test-suite

 rm -rf /tmp/wd-0.8.0

...

Draft the release notes in a text file (in markdown syntax) and keep it ready. You will need it in a later step.

Updating LICENSE-binary file

Tag the pinot docs for the given version

Warning
titleDeprecated

This step is deprecated. Instead, you can raise a PR for pinot-docs and get it merged, after the "Finalize the release" steps are done. No need to request access for or use Gitbook UI.

Please ask permission for gitbook Apache Pinot organization access in #pinot-committers slack channel.

Once you have the access, please duplicate the existing latest docs workspace:

Image Added

You can update the space name by clicking the title:

Image Added


Updating LICENSE-binary file

This is a fairly manual process and somewhat time-consuming This is a fairly manual process and somewhat time-consuming one. You may need a few hours to complete this step. Familiarize yourself with the contents of LICENSE-binary file and also the files in the licenses-binary directory

The LICENSE-binary file is at the top level of the pinot source code. It has one section for each type of license that a dependency may have. All the binary jars shipped with a license are listed within the section for that license (e.g. there is a section for Apache 2 License, and lists all packages that are shipped with the Apache 2.0 license). Each license type has a file in the licenses-binary/ folder that has the licensing terms (essentially, the text of the license).


Before everything, make sure you build the pinot project so your m2 cache is fulfilled：

Build the project along with presto dependencies:

Code Block
languagebash
mvn clean install -DskipTests -Ppresto-driver

Or you can do parallel build but in two commands

Code Block
languagebash
mvn clean install -DskipTests -Ppresto-driver -T1C -pl :presto-pinot-driver -am
mvn clean install -DskipTests -T1C


Check out the codebase (the release branch Check out the codebase (the release branch created while preparing for the release). Add the following dependencies to pinot-distribution/pom.xml (otherwise there will be missing lib dependencies) but don't check it in. (NOTE: Please double check if the following includes all packages in pinot-connectors and pinot-plugins. If not, please update the following code block.)

Code Block
    <!-- pinot-plugins/pinot-batch-ingestion -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-common</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-hadoop</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-spark-spark<2.4</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-batch-ingestion-spark-3</artifactId>
     <version>${project.version}</version>
    </dependency>      
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-standalone</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-environment -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-azure</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-file-system -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-adls</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-gcs</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-hdfs</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-s3</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-input-format -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-avro<avro-base</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-avro-base<avro</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-clp-csv<log</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-confluent-avro</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-csv</artifactId>
     <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-json</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-orc</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-parquet</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-protobuf</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-thrift</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-metrics -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-dropwizard</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-yammer</artifactId>
      <version>${project.version}</version>
    </dependency>
 

     <<!-- pinot-plugins/pinot-minion-tasks -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-minion-builtin-tasks</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-segment-uploader -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-segment-uploader-default</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-segment-writer -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-segment-writer-file-based</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-stream-ingestion -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-02.9<0</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-2.0<base</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-base<kinesis</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kinesis<pulsar</artifactId>
      <version>${project.version}</version>
    </dependency>
 
   <!-- pinot-connectors -->
   <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-pulsar<flink-connector</artifactId>
      <version>${project.version}</version>
    </dependency> 

    <!-- pinot-connectors -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-spark-2-connector</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>presto-pinot-driver<<artifactId>pinot-spark-3-connector</artifactId>
      <version>${project.version}</version>
    </dependency> 
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-spark-common</artifactId>
     <version>${project.version}</version>
   </dependency>


After After that, run the commands:

Code Block
languagebash
mvn -Ddependency.locations.enabled=false project-info-reports:dependencies -Ppresto-driver -pl :pinot-distribution
# This will create a file in pinot-distribution/target/site/dependencies.html

Open the file dependencies.html (mentioned above) in a browser. You will need to get two parts of the screen onto raw files by copy-pasting it:

...


 Warning
titleWatchout for Errors
If you get errors due to inability to get some confluent packages, you can add the confluent repository to your ~/.m2/settings.xml file.
    <repositories>
      <repository>
        <releases>
          <enabled>false</enabled>
        </releases>
        <id>apache.snapshots</id>
        <name>Apache Snapshot Repository</name>
        <url>https://repository.apache.org/snapshots</url>
      </repository>
      <repository>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>Central Repository</name>
        <url>https://repo.maven.apache.org/maven2</url>
      </repository>
      <repository>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
        <id>confluent</id>
        <name>Confluent Repository</name>
        <url>https://packages.confluent.io/maven/</url>
      </repository>
    </repositories>

Open the file dependencies.html (mentioned above) in a browser. You will need to get two parts of the screen onto raw files by copy-pasting it:
  1. Create a fresh working directory ($WORKING_DIR), cd to it.
  2. Scroll down to the sections on the project's compile dependencies (direct and transitive). Grab the table using the mouse, and save the result to a text file. Name the file pkg-dependencies-raw.txt. Keep the browser page open, you will need it later.
  3. Scroll down to the project's runtime dependencies. Grab the table using the mouse and save the result to a text file named runtime-dependencies-raw.txt.

 Note
Do NOT trust the License column in the browser. You will need to manually verify the license in a later step by following the package links in the page.

You can now work on this text file to generate the LICENSE-binary file for the next release
 Code Block
languagebash
titleProcessing license dependencies
cd $WORKING_DIR
# First, get the new list of dependencies, sorted
cat pkg
...
 Note
Do NOT trust the License column in the browser. You will need to manually verify the license in a later step by following the package links in the page.
You can now work on this text file to generate the LICENSE-binary file for the next release
 Code Block
languagebash
titleProcessing license dependencies
cd $WORKING_DIR
# First, get the new list of dependencies, sorted
cat pkg-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq > /tmp/x1
cat runtime-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq >>> /tmp/x1
sort /tmp/x1 > new-pkg-versions.txt

# Now pull cat runtime-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq >> /tmp/x1
sort /tmp/x1 > new-pkg-versions.txt

# Now pull out all package version lines from the existing LICENSE-binary file into another file, call it current-pkg-versions.txt.unsorted
cp /path/to/existing/source/LICENSE-binary ./LICENSE-binary
cp LICENSE-binary current-pkg-versions.txt.unsorted
vim current-pkg-versions.txt.unsorted # Now, remove all fluff in the file except for the lines that hold package name and versions.
sort current-pkg-versions.txt.unsorted | uniq > current-pkg-versions.txt

# Now you can compare (diff) the two files to see which packages have been removed, and which ones have been added.

# I use tkdiff, but you can use 'diff -y' or some of the more fancy options of diff that helps you get what you want.
tkdiff current-pkg-versions.txt new-pkg-versions.txt
...
This file is a concatenation of all the NOTICE files of packages we use.

In order to generate this file, we need to update pinot temporarily to make the following changes
Edit pom.xml temporarily (DO NOT CHECKIN)
Change the top level pom.xml as follows:
add a maven-shade-plugin at root pom.xml.

Note that, you need to search <artifactId>maven-shade-plugin</artifactId> in all the submodules pom.xml. Then remove all the found maven-shade-plugin plugins.

Edit pom.xml temporarily (DO NOT CHECKIN)
Change the top level pom.xml as follows:
 Code Block
languagebash
diff --git a/pom.xml b/pom.xml
index e72be7717..a38f08915 100644
--- a/pom.xml
+++ b/pom.xml
@@ -889,6 +889,31 @@
     <defaultGoal>clean install</defaultGoal>
     <pluginManagement>
       <plugins>
+<plugin>
+  <groupId>org.apache.maven.plugins</groupId>
+  <artifactId>maven-shade-plugin</artifactId>
+  <configuration>
+    <shadedArtifactAttached>false</shadedArtifactAttached>
+    <artifactSet>
+      <includes>
+        <include>*:*</include>
+      </includes>
+    </artifactSet>
+  </configuration>
+  <executions>
+    <execution>
+      <phase>package</phase>
+      <goals>
+        <goal>shade</goal>
+      </goals>
+      <configuration>
+        <transformers>
+          <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer"/>
+        </transformers>
+      </configuration>
+    </execution>
+  </executions>
+</plugin>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-compiler-plugin</artifactId>

Change pinot-distribution/pom.xml as follows: to use jar packaging. And of course, you need to remove maven-shade-plugin.
 Code Block
languagebash
diff --git a/pinot-distribution/pom.xml b/pinot-distribution/pom.xml
index 321a131de..3f7b12541 100644
--- a/pinot-distribution/pom.xml
+++ b/pinot-distribution/pom.xml
@@ -22,7 +22,7 @@
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
-  <packaging>pom</packaging>
+  <packaging>jar</packaging>
   <parent>
     <artifactId>pinot</artifactId>
     <groupId>org.apache.pinot</groupId>

--- a/pinot-distribution/pom.xml
+++ b/pinot-distribution/pom.xml
@@ -144,42 +144,6 @@
           </algorithms>
         </configuration>
       </plugin>
-      <plugin>
-        <artifactId>maven-shade-plugin</artifactId>
-        <executions>
-          <execution>
-            <goals>
-              <goal>shade</goal>
-            </goals>
-            <configuration>
-              <transformers>
-                <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer"/>
-              </transformers>
-              <!--
-              Usually in hadoop environment, there are multiple jars with different versions.
-              Most of the NoSuchMethodExceptions are caused by class loading conflicts.
-              Class relocation ensures the reference of certain packages/classes in Pinot code to
-              shaded libs, e.g. jackson or guava.
-              Ref: https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
-              -->
-              <relocations>
-                <relocation>
-                  <pattern>com.google.common</pattern>
-                  <shadedPattern>shaded.com.google.common</shadedPattern>
-                </relocation>
-                <relocation>
-                  <pattern>com.fasterxml.jackson</pattern>
-                  <shadedPattern>shaded.com.fasterxml.jackson</shadedPattern>
-                </relocation>
-                <relocation>
-                  <pattern>org.apache.http</pattern>
-                  <shadedPattern>shaded.org.apache.http</shadedPattern>
-                </relocation>
-              </relocations>
-            </configuration>
-          </execution>
-        </executions>
-      </plugin>
     </plugins>
   </build>
   <profiles>

...
 Code Block
languagebash
# cd to the top level pinot directory
mvn clean -Ddependency.locations.enabled=false -DskipTests package -Ppresto-driver -T1C

cd pinot-distribution/target
unzip pinot-distribution-0.911.0-SNAPSHOT-shaded.jar META-INF/NOTICE
cd ../..
head -6 NOTICE-binary > /tmp/xx
cat pinot-distribution/target/META-INF/NOTICE >> /tmp/xx
cp /tmp/xx NOTICE-binary

git checkout  -- pinot-distribution**/pom.xml pom.xml# Revert the changes made temporarily
...
Lastly, remove the following lines from NOTICE-binary
 Code Block
languagebash

Apache Pinot
Copyright 2018-2021 Apache Software Foundation


...
Before executing the step below, this page is a useful read these pages:
...
 Warning
Not checking the pre-release box in this step will violate Apache terms.
...
Update master branch with the next release pom changes
...
Recall the step in which you typed mvn release:prepare. That step introduced two commits in your release branch. Now is the time you need to take the second commit (the one with the comment [maven-release-plugin] prepare for next development iteration) and merge it back to the master.
Or you can run below command to directly update pom version.
 Code Block
languagebash
$ mvn -Darguments=-batch-DskipTestsmode release:perform -Pscala-2.12,release-sign-artifacts -Ppresto-driver
# This step will update Nexus with your release bits.
After running the release:perform  command,
  1. Log in to Nexus (https://repository.apache.org)
  2. Select Staging Repositories and find the repository with the name "orgapachepinot-{number}".
  3. Select that and click the "Close" button.
This will trigger to Nexus to do some checks on artifacts (checksum, signatures..etc). 
 Note
Create a source/binary tarballs & Staging source and binary release tarballs
 Code Block
languagebash
$ cd <pinot_source_code_root_path>

# Check out the release candidate tag
$ git checkout tags/release-$VERSION-rc$RC

# Check git hash for the official release
$ git log

# Create the package
$ mvn install -DskipTests -Papache-release,bin-dist,src-dist

$ cd pinot-distribution/target
$ ls -l
...
-rw-r--r--  1 xiangfu  staff  176867304 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz
-rw-r--r--  1 xiangfu  staff        833 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz.asc
-rw-r--r--  1 xiangfu  staff        128 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz.sha512
-rw-r--r--  1 xiangfu  staff   44435410 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz
-rw-r--r--  1 xiangfu  staff        833 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz.asc
-rw-r--r--  1 xiangfu  staff        128 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz.sha512
...

# Copy files to the pinot-dev-dist svn repo
$ mkdir /path/to/pinot-dev-dist/apache-pinot-$VERSION-rc$RC
$ cp apache-pinot-$VERSION-*.tar.gz* /pat/to/pinot-dev-dist/apache-pinot-$VERSION-rc$RC
update-versions -DdevelopmentVersion=<next-version>

# example
$ mvn --batch-mode release:update-versions -DdevelopmentVersion=0.4.0-SNAPSHOT


## NOTE: the above approach using "maven-release-plugin" will break the formatting. If one simply want to change the versions for all pom.xml files, please run the following:

$ PREV_VERSION=0.8.0-SNAPSHOT
$ NEXT_VERSION=0.9.0-SNAPSHOT

# Mac OS
$ find . -name pom.xml | while read i; do cat $i | sed -i '' "s/$PREV_VERSION/$NEXT_VERSION/g" $i ; done

# Linux
$ find . -name pom.xml | while read i; do cat $i | sed -i "s/$PREV_VERSION/$NEXT_VERSION/g" $i ; done
Stage the release
In this step, you will publish the artifacts to "staging repository"
 Code Block
languagebash
$ mvn -Darguments=-DskipTests release:perform -Pscala-2.12,release-sign-artifacts -Ppresto-driver
# This step will update Nexus with your release bits.

 Note
If the command above fails with the error "No SCM URL was provided to perform the release from during maven release", check if your release.properties file is intact. If not, repeat the steps below to prepare the release candidate. 
  • Run the command mvn release:clean
  • Delete the tag release-1.x.0-rc0 in your local and remote github branch
  • Remove the 2 commits added by the prepare release step. 
  • Repeat the process to prepare the release candidate. 

After running the release:perform  command,
  1. Log in to Nexus (https://repository.apache.org)
  2. Select Staging Repositories and find the repository with the name "orgapachepinot-{number}".
  3. Select that and click the "Close" button.
This will trigger to Nexus to do some checks on artifacts (checksum, signatures..etc). 

 Note

  1. Create a source/binary tarballs & Staging source and binary release tarballs
     Code Block
    languagebash
    $ cd <pinot_source_code_root_path>

# Check out the release candidate tag
$ git checkout tags/release-$VERSION-rc$RC

# Check git hash for the official release
$ git log

# Create the package
$ mvn install -DskipTests -Papache-release,bin-dist,src-dist -T1C

# If there is a missing asc file, you can run
# gpg -ab apache-pinot-0.11.0-src.tar.gz
# to generate apache-pinot-0.11.0-src.tar.gz.asc

$ cd pinot-distribution/target
$ ls -l
...
-rw-r--r--  1 xiangfu  staff  176867304 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz
-rw-r--r--  1 xiangfu  staff        833 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz.asc
-rw-r--r--  1 xiangfu  staff        128 Mar 16 01:01 apache-pinot-0.8.0-bin.tar.gz.sha512
-rw-r--r--  1 xiangfu  staff   44435410 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz
-rw-r--r--  1 xiangfu  staff        833 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz.asc
-rw-r--r--  1 xiangfu  staff        128 Mar 16 01:01 apache-pinot-0.8.0-src.tar.gz.sha512
...

# Copy files to the pinot-dev-dist svn repo
$ mkdir /path/to/pinot-dev-dist/apache-pinot-$VERSION-rc$RC
$ cp apache-pinot-$VERSION-*.tar.gz* /pat/to/pinot-dev-dist/apache-pinot-$VERSION-rc$RC

  2. Validate the release that you built, using Validating a release candidate
  3. Commit the files to svn repository
 Warning
titleSVN Commit Failure
If the SVN Commit fails with the error "E175002: Commit failed (details follow):", it is likely because we are hitting per-artifact limit on the Apache SVN Servers. See this issue for more details: https://github.com/apache/pinot/issues/13726

 Code Block
languagebash
# Commit the files to staging svn repository
$ cd /path/to/pinot-dev-dist
$ svn add apache-pinot-$VERSION-rc$RC
$ svn commit -m "Update apache-pinot-$VERSION-rc$RC"

Check https://dist.apache.org/repos/dist/dev/pinot/ to see if the files are uploaded correctly. requires -src/bin.tar.gz, tar.gz.asc, tar.gz.sha512 files.

Once you have all 6 information (gpg key, git tag, git hash, release note, staging repository, staged tarballs), release is now ready for voting process!
Call the Vote
Note that there are ONE Rounds of Approvals to be collected:
We need to go through the voting at dev@pinot mailing list. Send out the mail for voting. The example is present at the following links. The voting should be open for at least 72 hours and it requires a minimum of three +1 votes. Once the voting passes, you can send out the result mail. Examples:
dev@pinot votehttps://lists.apache.org/thread.html/r5f5f06126f39088b8b03a362bf39b5fd3557c07b5b7439e81746c752%40%3Cdev.pinot.apache.org%3E
dev@pinot vote result
https://lists.apache.org/thread.html/rdd8d44971e6b4539f40240005cf3a0af1a81e493f2ef2a7e57b360d1%40%3Cdev.pinot.apache.org%3E
Once both votings passed, it is now officially approved by Apache and we just need to do some final steps.
Recall a Release Candidate
There could be some issues unveiled during the testing and voting phase. If it happens, we need to recall the release candidate and apply patches/changes then redo from step: Prepare the Release Candidate and increase <rc_num> by 1.
There are several things to clean up:
  1. Delete git repo release draft: https://github.com/apache/pinot/releases
  2. Delete the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Drop button.
  3. Clean up tarballs from dev dist
     Code Block
    languagebash
    $ svn rm https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC -m "Delete release-$VERSION-rc$RC"
# NOTE: You will need your apache login and password for this step.

Finalize a Release
  1. Move staging tarballs to the svn directory for official release. You need to be a PMC member to be able to do this step. If you are a committer, ask a PMC member to do this step for you.
     Code Block
    languagebash
    $ svn mv https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC https://dist.apache.org/repos/dist/release/pinot/apache-pinot-$VERSION
# NOTE: You will need your apache login and password for this step.
    Check https://downloads.apache.org/pinot
    After several hours later, check https://archive.apache.org/dist/pinot/apache-pinot-<version>/  to see if it shows up.
    (e.g. https://archive.apache.org/dist/pinot/apache-pinot-0.8.0/)
  2. Delete previous release tarballs. Similar to step 1, you need to be PMC member to be able to perform this step.
    We only need to keep the most recent release, so feel free to delete any previous released distributions.
     Code Block
    languagebash
    $ svn rm 
...
 Code Block
languagebash
# Commit the files to staging svn repository
$ cd /path/to/pinot-dev-dist
$ svn add apache-pinot-$VERSION-rc$RC
$ svn commit -m "Update apache-pinot-$VERSION-rc$RC"
...
  1. https://dist.apache.org/repos/dist/
...
  1. release/pinot/
...
  1. apache-pinot-0.8.0
# NOTE: You will need your apache login and password for this step.

  2. Delete release branch from github repo if any
     Code Block
    languagebash
    $ git push origin --delete release-$VERSION-rc

  3. Update git tag for official release
     Code Block
    languagebash
    $ git checkout tags/release-$VERSION-rc$RC
$ git tag release-$VERSION -m "Apache Pinot $VERSION"
$ git push origin release-$VERSION

  4. Update Release note for the official release on Github https://github.com/apache/pinot/releases
    1. No need to draft a new release, just EDIT the previously drafted release
    2. Replace tag name to release-<version> tag
    3. Uncheck the pre-release checkbox, then publish
    4. Delete all other rc drafts if there is any.
  5. Delete all RC tags from github repo
     Code Block
    languagebash
    # Clean up the rc tag to not to show on the Github release page
$ git tag -d release-$VERSION-rc$RC
$ git push origin :refs/tags/release-$VERSION-rc$RC

  6. Publish 
Once you have all 6 information (gpg key, git tag, git hash, release note, staging repository, staged tarballs), release is now ready for voting process!
Call the Vote
Note that there are ONE Rounds of Approvals to be collected:
We need to go through the voting at dev@pinot mailing list. Send out the mail for voting. The example is present at the following links. The voting should be open for at least 72 hours and it requires a minimum of three +1 votes. Once the voting passes, you can send out the result mail. Examples:
...
https://lists.apache.org/thread.html/rdd8d44971e6b4539f40240005cf3a0af1a81e493f2ef2a7e57b360d1%40%3Cdev.pinot.apache.org%3E
Once both votings passed, it is now officially approved by Apache and we just need to do some final steps.
Recall a Release Candidate
There could be some issues unveiled during the testing and voting phase. If it happens, we need to recall the release candidate and apply patches/changes then redo from step: Prepare the Release Candidate and increase <rc_num> by 1.
There are several things to clean up:
  1. Delete git repo release draft: https://github.com/apache/pinot/releases
  2. Delete the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Drop button.
    Clean up tarballs from dev dist
     Code Block
    languagebash
    $ svn rm Release button. Several hours later, the new version should show up at https://distrepo.maven.apache.org/reposmaven2/distorg/dev/pinot/apache-pinot-$VERSION-rc$RC -m "Delete release-$VERSION-rc$RC"
# NOTE: You will need your apache login and password for this step.
Finalize a Release
...
Move staging tarballs to the svn directory for official release 
 Code Block
languagebash
$ svn mv https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC https://dist.apache.org/repos/dist/release/pinot/apache-pinot-$VERSION
# NOTE: You will need your apache login and password for this step.
Check https://downloads.apache.org/pinot
After several hours later, check http://www.apache.org/dyn/closer.cgi/pinot/apache-pinot-<version> to see if it shows up.
(e.g. http://www.apache.org/dyn/closer.cgi/pinot/apache-pinot-0.8.0)
Delete previous release tarballs
We only need to keep the most recent release, so feel free to delete any previous released distributions.
 Code Block
languagebash
$ svn rm https://dist.apache.org/repos/dist/release/pinot/apache-pinot-0.8.0
# NOTE: You will need your apache login and password for this step.
Delete release branch from github repo if any
 Code Block
languagebash
$ git push origin --delete release-$VERSION-rc
Update git tag for official release
 Code Block
languagebash
$ git checkout tags/release-$VERSION-rc$RC
$ git tag release-$VERSION -m "Apache Pinot $VERSION"
$ git push origin release-$VERSION
  1. apache/pinot
  2. Build docker image for the release:
    1. Please ask Xiang Fu to grant you permission to run the jobs on this repo: https://github.com/apachepinot/pinot-fork/actions.
    2. There are total 5 jobs related to different JDKs(Amazon Corretto 11/17, Microsoft OpenJDK 11/17 and OpenJDK 21)  to run:
      1. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-amazoncorretto-11-pinot-docker-image.yml
      2. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-amazoncorretto-17-pinot-docker-image.yml
      3. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-ms-openjdk-11-pinot-docker-image.yml
      4. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-ms-openjdk-17-pinot-docker-image.yml
      5. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-java21-pinot-docker-image.yml 
    3. Build images with release tag, and tag it with both tags: release-<version>  and <version> , e.g. release-1.2.0  and 1.2.0 . See below graph for the input reference.Image Added
    4. After all the tasks are done, you should find all the docker images here: https://hub.docker.com/r/apachepinot/pinot/tags?page=&page_size=&ordering=&name=release-1.2.0
    5. Tag official release and push, making sure you asked Xiang Fu for the dockerhub push permission for ApachePinot org.
 Code Block
languagebash
titledocker tag release manifest
docker manifest create apachepinot/pinot:1.2.0 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-arm64 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-amd64
docker manifest create apachepinot/pinot:release-1.2.0 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-arm64 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-amd64

docker manifest push apachepinot/pinot:1.2.0
docker manifest push apachepinot/pinot:release-1.2.0
...
Update Release note for the official release on Github https://github.com/apache/pinot/releases
  1. No need to draft a new release, just EDIT the previously drafted release
  2. Replace tag name to release-<version> tag
  3. Uncheck the pre-release checkbox, then publish
  4. Delete all other rc drafts if there is any.
Delete all RC tags from github repo
 Code Block
languagebash
# Clean up the rc tag to not to show on the Github release page
$ git tag -d release-$VERSION-rc$RC
$ git push origin :refs/tags/release-$VERSION-rc$RC
...


Announce to the world
Once you finalized the release, you now need to announce this to the world. 

  1. Update https://pinot.apache.org/download page page. Example pr:  https://github.com/apache/pinot-site/pull/57
  2. Update Pinot docs: https://docs.pinot.apache.org/basics/releases
    1. Guidance on updating the page: https://docs.pinot.apache.org/developers/developers-and-contributors/update-document.
  3. Send the mail to dev@pinot about the release announcement (We can keep them same )
    dev@pinot https://lists.apache.org/thread.html/r2008f31727da372fe8616d1f9456e07817e335a1f0e71d793ce7c752%40%3Cdev.pinot.apache.org%3E
  4. Notify #general channel on Slack about the release. https://apache-pinot.slack.com/archives/CDRCA57FC/p1618388015390500
Update master branch with the next release pom changes
Recall the step in which you typed mvn release:prepare. That step introduced two commits in your release branch. Now is the time you need to take the second commit (the one with the comment [maven-release-plugin] prepare for next development iteration) and merge it back to the master.
Or you can run below command to directly update pom version.
...
languagebash
...
  1. 131
  2. Apache takes time to archive all the distributions in SVN release repo, so we don't need to keep all the distributions in SVN release repo. For download URL, we use https://downloads.apache.org for the most recent dist and https://archive.apache.org/dist for old dists.
    1. SVN release repo distribution can be downloaded from: https://downloads.apache.org e.g. https://downloads.apache.org/pinot/apache-pinot-0.9.1/apache-pinot-0.9.1-bin.tar.gz
    2. Old distributions are available through the archive urls, please make sure you update download links for old dist from https://downloads.apache.org to https://archive.apache.org/dist. E.g. https://archive.apache.org/dist/pinot/apache-pinot-0.9.0/apache-pinot-0.9.0-bin.tar.gz Sample PR: https://github.com/apache/pinot-site/pull/61
  3. Update Pinot github repo DOAP file to reflect the latest release on https://projects.apache.org/project.html?pinot, sample PR: https://github.com/apache/pinot/pull/11827
  4. Update Pinot docs: https://docs.pinot.apache.org/basics/releases
    1. Guidance on updating the page: https://docs.pinot.apache.org/developers/developers-and-contributors/update-document.
    2. Raise a PR similar to (you can raise a single PR): pinot-docs/commit/d1849be pinot-docs/commit/0f035df against the pinot-docs repo. Use the release notes you had used and prepared before.
  5. Send the mail to dev@pinot.apache.org about the release announcement
    dev@pinot https://lists.apache.org/thread/qw3chy5rvclymzwk0ros98nlk7mxwhos
  6. Send the mail to announce@apache.org  about the release announcement.  Remember to use plain text mode:     

    Image Added
Note that emails to announce@apache.org may get delayed for up to a day, so please be patient (smile)
...
Reference
  1. https://cwiki.apache.org/confluence/display/GOBBLIN/Release+Process
...