Prerequisites
Before starting the process, you need to ensure several things before proceed to create a release.
- You need to clone the latest pinot code base in your local machine
- You need to ensure that you have the access to Apache Repository Manager. Try to log in with Apache LDAP credential.
You need to check out Apache staging & release distribution directories for Pinot.
svn co https://dist.apache.org/repos/dist/release/incubator/pinot pinot-release-dist svn co https://dist.apache.org/repos/dist/dev/incubator/pinot pinot-dev-dist
https://dist.apache.org/repos/dist/release/incubator/pinot: this is where the actual release (PMC approved) will be placed. Do not upload anything here until voting from both dev@pinot and general@incubator pass.
- https://dist.apache.org/repos/dist/dev/incubator/pinot: this is where we put the release candidates and link for this will be used for reference during the voting process.
Updating LICENSE-binary file
This is a fairly manual process, and somewhat time-consuming one. You may need a few hours to complete this step. Familiarize yourself with the contents of LICENSE-binary file and also the files in the licenses-binary directory
The LICENSE-binary file is at the top level of the pinot source code. It has one section for each type of license that a dependency may have. All the binary jars shipped with a license are listed within the section for that license (e.g. there is a section for Apache 2 License, and lists all packages that are shipped with the Apache 2.0 license). Each license type has a file in the licenses-binary/ folder that has the licensing terms (essentially, the text of the license).
Check out the code base and run the command and run the commands:
cd pinot-distribution mvn -Ddependency.locations.enabled=false project-info-reports:dependencies # This will create a file in pinot-distribution/target/site/dependencies.html
Open this file in a browser. You will need to get two parts of the screen onto raw files by copy-pasting it:
- Create a fresh workig directory ($WORKING_DIR), cd to it.
- Scroll down to the section on the project's compile dependencies (transitive). Grab the table using the mouse, and save the result to a text file. Name the file
pkg-dependencies-raw.txt. Keep the browser page open, you will need it later. - Scroll down to the project's runtime dependencies. Grab the table using the mouse and save the result to a text file named
runtime-dependencies-raw.txt.
Do NOT trust the License column in the browser. You will need to manually verify the license in a later step by following the package links in the page.
You can now work on this text file to generate the LICENSE-binary file for the next release
cd $WORKING_DIR
# First, get the new list of dependencies, sorted
cat pkg-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq > /tmp/x1
cat runtime-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq >> /tmp/x1
sort /tmp/x1 > new-pkg-versions.txt
# Now pull out all package version lines from the existing LICENSE-binary file into another file, call it current-pkg-versions.txt.unsorted
cp /path/to/existing/source/LICENSE-binary ./LICENSE-binary
cp LICENSE-binary current-pkg-versions.txt.unsorted
vim current-pkg-versions.txt.unsorted # Now, remove all fluff in the file except for the lines that hold package name and versions.
sort current-pkg-versions.txt.unsorted | uniq > current-pkg-versions.txt
# Now you can compare (diff) the two files to see which packages have been removed, and which ones have been added.
# I use tkdiff, but you can use 'diff -y' or some of the more fancy options of diff that helps you get what you want.
tkdiff current-pkg-versions.txt new-pkg-versions.txt
Once you have the old and new package with version numbers, do the following:
- Identify which packages are no longer in use. Remove them from LICENSE-binary file. If that license is no longer used, then remove the license file from licenses-binary folder.
- Identify packages that have changed versions. Go the package's home page (as pointed by the browser page that has the dependencies), and verify that the license terms have not changed. If they have, then remove the package name from the license section, and treat it as a new package dependency (in the next step)
- Identify all new package dependencies (including those from step 2 above). Click on the package's home page (as pointed by the browser page that has dependencies), and examine the license. You may need to traverse other pages or to the code base in order to finally get the license.
- If the license is one of the categories that already exist in the licenses-binary folder, just add the package name and version to the LICENSE-binary file.
- If it is a new kind of license
- Make sure to create a file in licenses-binary directory copying the license terms from the package's LICENSE file or from its website as the case may be.
- Create a new section in LICENSE-binary file just like the sections for other licenses.
- copy the package name from
new-pkg-versions.txtunder the section for that license.
- Sort the package names in each section of the LICENSE-binary file. This is useful step that makes it easier to identify the changes when building the next release.
Note on BSD Licenses
The Copyright line in BSD licenses differ between packages. In most cases, we are having to create a separate license file the licenses-binary directory. See LICENSE-scala.txt, LICENSE-xmlenc.txt, etc. For a full list, try the command:
grep -l "Redistributions in binary form must reproduce the above copyright notice" licenses-binary/*.txt
Update NOTICE-binary file
This file is a concatenation of all the NOTICE files of packages we use. In order to generate this file, we need to update pinot temporarily to make the following changes
Edit pom.xml temporarily (DO NOT CHECKIN)
Change the top level pom.xml as follows:
diff --git a/pom.xml b/pom.xml
index e72be7717..a38f08915 100644
--- a/pom.xml
+++ b/pom.xml
@@ -889,6 +889,31 @@
<defaultGoal>clean install</defaultGoal>
<pluginManagement>
<plugins>
+<plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <artifactSet>
+ <includes>
+ <include>*:*</include>
+ </includes>
+ </artifactSet>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <transformers>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer"/>
+ </transformers>
+ </configuration>
+ </execution>
+ </executions>
+</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
Change pinot-distribiution/pom.xml as follows:
diff --git a/pinot-distribution/pom.xml b/pinot-distribution/pom.xml
index 321a131de..3f7b12541 100644
--- a/pinot-distribution/pom.xml
+++ b/pinot-distribution/pom.xml
@@ -22,7 +22,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
- <packaging>pom</packaging>
+ <packaging>jar</packaging>
<parent>
<artifactId>pinot</artifactId>
<groupId>org.apache.pinot</groupId>
Now enter the following commands:
# cd to the top level pinot directory mvn clean -Ddependency.locations.enabled=false -DskipTests package cd pinot-distribution/target unzip pinot-distribution-0.2.0-SNAPSHOT-shaded.jar META-INF/NOTICE cd ../.. head -6 NOTICE-binary > /tmp/xx cat pinot-distribution/target/META-INF/NOTICE >> /tmp/xx cp /tmp/xx NOTICE-binary git checkout -- pinot-distribution/pom.xml pom.xml# Revert the changes made temporarily
Lastly, remove the following lines from NOTICE-binary
Apache Pinot (incubating) Copyright 2018-2019 Apache Software Foundation
Now, diff the NOTICE-binary file to make sure that it looks like we want, with no extra lines in place.
GPG Key Setup
It is strongly suggested that you read through this page first: https://www.apache.org/dev/release-signing.html
The release manager must hold a GPG key for signing the release artifacts. GPG key has to be associated with the apache email address. One can create the GPG by using the following cmd. Note that the key length should be 4096 bits. Please refer to the example below:
$ gpg2 --full-generate-key gpg (GnuPG) 2.1.22; Copyright (C) 2017 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 4096 Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) Key does not expire at all Is this correct? (y/N) y ... gpg: key 2C9B34D6A54AAFAE marked as ultimately trusted ...
After you generate the GPG key, copy the key into an environment variable. Look for the line in the output that says "marked as ultimately trusted". For example, in the above output: "gpg: key 2C9B34D6A54AAFAE marked as ultimately trusted" (From here, I will use GPG_KEY to refer the key).
Make sure the key has a preference of SHA512. See https://www.apache.org/dev/openpgp.html#key-gen-avoid-sha1
$ GPG_KEY=<your_gpg_key> # fill in your key from the "ultimately trusted" line as above
Once you generate the gpg key, you need to go through the following steps:
Upload GPG public key to a public key server. Copy the following command's output to "Submit a key" box and click the submit button. http://pgp.mit.edu. You can also upload it to other sites (See https://www.apache.org/dev/release-signing.html#keyserver for some servers, and also a command-line way to send key to public servers)
$ gpg --export --armor $GPG_KEY -----BEGIN PGP PUBLIC KEY BLOCK----- ... -----END PGP PUBLIC KEY BLOCK-----
Update key fingerprint at https://id.apache.org
$ gpg --fingerprint $GPG_KEY
Copy the hex numbers (e.g. "0D9E 9997 4D5A 59F1 7183 362F 2C9B 34D6 A54A AFAE") to "OpenPGP Public Key Primary Fingerprint" box and submit the change.
Update KEYS file under pinot-release-dist
$ cd pinot-release-dist # Append the public key to KEYS file gpg --list-sigs $GPG_KEY && gpg --armor --export $GPG_KEY >> KEYS # Commit the change svn commit -m "Update Pinot KEYS file for $GPG_KEY"
After following 1-3, you should have the following (after at most 24 hours delay. Certificates are synced from public domain once a day):
- Your public key is listed in https://dist.apache.org/repos/dist/release/incubator/pinot/KEYS
- Your public key is viewable at https://people.apache.org/keys/committer/<apache_ldap_id>.asc (e.g. https://people.apache.org/keys/committer/snlee.asc)
- Your public key is viewable at https://people.apache.org/keys/group/pinot.asc
Maven Configuration Setup
As part of the release process, we will upload our artifacts to maven central staging repository and "maven-release-plugin" will do it for you automatically for the most of part. In order to use the plugin, we need to set up the maven config correctly.
$ vim ~/.m2/settings.xml
# Paste the below text
<settings>
<servers>
<server>
<id>apache.snapshots.https</id>
<username>{apache LDAP id}</username>
<password>{apache user password}</password>
</server>
<server>
<id>apache.releases.https</id>
<username>{apache LDAP id}</username>
<password>{apache user password}</password>
</server>
<server>
<id>gpg.passphrase</id>
<passphrase>{gpg key phrase}</passphrase>
</server>
</servers>
<profiles>
<profile>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<gpg.keyname>{GPG_KEY}</gpg.keyname>
</properties>
</profile>
</profiles>
</settings>
Also, We recommend to add the following line to your `~/.bashrc` file.
$ vim ~/.bashrc # add the following line export GPG_TTY=$(tty) $ source ~/.bashrc
Adding above line make the terminal UI correctly show when you need to enter the key phrase for your key when signing the artifacts.
Creating a Release
Before we move on to the voting, we need to prepare the release along with the below information:
- GPG Key that the artifacts are signed with - From GPG Key Setup
- Git tag - Step 2
- Git hash for the release - Step 2 (Pick the git hash from the tag created during step 2)
- Release Note - Step 3
- Staging Repository link (Nexus) - Step 4
- Release Candidate under https://dist.apache.org/repos/dist/dev/incubator/pinot - Step 5
After following the steps below, you should have all information listed above.
Prepare a release branch
$ git checkout -b release-<version>-rc<rc_num> $ git push origin release-<version>-rc<rc_num> # Example $ git checkout -b release-0.1.0-rc0 $ git push origin release-0.1.0-rc0
Prepare the release (Please refer "Preparing a release" section in http://plc4x.apache.org/developers/release.html to see what actually happens with the command below - has a good explanation along with screenshots, our gitbox url for checking is https://gitbox.apache.org/repos/asf?p=incubator-pinot.git) . Before entering this command, make sure that you have permissions to push to the remote git branch without entering password or passphrase. This command invokes git push at the end, and will fail if you don't have permissions to push to the remote tree.
$ mvn release:prepare # Be sure to answer the questions as below (example): [INFO] Checking dependencies and plugins for snapshots ... What is the release version for "Pinot"? (org.apache.pinot:pinot) 0.2.0: : What is SCM release tag or label for "Pinot"? (org.apache.pinot:pinot) pinot-0.2.0: : release-0.2.0-rc0 What is the new development version for "Pinot"? (org.apache.pinot:pinot) 0.2.1-SNAPSHOT: : 0.3.0-SNAPSHOT
- Prepare the release note. Step 2 will generate the new tag and it will automatically show up on https://github.com/apache/incubator-pinot/releases
- Click "Draft a new release" button, put "tag version" as the one you used from the previous step (e.g. release-0.1.0-rc0).
- Fill in the release note and publish the release. NOTE: you need to check "This is a pre-release" box in order not to violate Apache terms.
Perform the release (publishing artifacts to "staging repository")
$ mvn release:perform
After run this, log in to Nexus (https://repository.apache.org) and select Staging Repositories and find the repository with the name "orgapachepinot-{number}". Select that and click the "Close" button. This will trigger to Nexus to do some checks on artifacts (checksum, signatures..etc). Check https://repository.apache.org/content/repositories/orgapachepinot-{number} shows correctly.
Create a source/binary tarballs & Staging source and binary release tarballs
$ cd <pinot_source_code_root_path># Check out the release candidate tag$ git checkout tags/release-<version>-rc<rc_num># Check git hash for the official release $ git log# Create the package $ mvn install -DskipTests -Papache-release,bin-dist $ cd pinot-distribution/target $ ls ... -rw-r--r-- 1 snlee eng 115915316 Mar 21 14:25 apache-pinot-incubating-0.1.0-bin.tar.gz -rw-r--r-- 1 snlee eng 490 Mar 21 14:25 apache-pinot-incubating-0.1.0-bin.tar.gz.asc -rw-r--r-- 1 snlee eng 128 Mar 21 14:26 apache-pinot-incubating-0.1.0-bin.tar.gz.sha512 -rw-r--r-- 1 snlee eng 37404419 Mar 21 14:25 apache-pinot-incubating-0.1.0-src.tar.gz -rw-r--r-- 1 snlee eng 490 Mar 21 14:25 apache-pinot-incubating-0.1.0-src.tar.gz.asc -rw-r--r-- 1 snlee eng 128 Mar 21 14:26 apache-pinot-incubating-0.1.0-src.tar.gz.sha512 ... # Copy files to the pinot-dev-dist $ mkdir /path/to/pinot-dev-dist/apache-pinot-incubating-<version>-rc<rc_num> $ cp apache-pinot-incubating-0.1.0-SNAPSHOT-*.tar.gz* /pat/to/pinot-dev-dist/apache-pinot-incubating-<version>-rc<rc_num> # Commit the files to staging svn repository $ cd /path/to/pinot-dev-dist $ svn add apache-pinot-incubating-<version>-rc<rc_num> $ svn commit -m "Update apache-pinot-incubating-<version>-rc<rc_num>"
After the above step, check https://dist.apache.org/repos/dist/dev/incubator/pinot/ to see if the files are uploaded correctly. requires -src/bin.tar.gz, tar.gz.asc, tar.gz.sha512 files.
Once you have all 6 information (gpg key, git tag, git hash, release note, staging repository, staged tarballs), release is now ready for voting process!
Call the Vote
As a first step, we need to go through the voting at dev@pinot mailing list. Send out the mail for voting. The example is present at the following links. The voting should be open for at least 72 hours and it requires minimum 3 +1 vote. Once the voting passes, you can send out the result mail.
dev@pinot vote https://lists.apache.org/thread.html/f136d3eaa9dfbab6e17b262a5542813099f2b128465d9d17ef69defd@%3Cdev.pinot.apache.org%3E
dev@pinot vote result https://lists.apache.org/thread.html/ce58034678349f82afc5f8ed2edd435875301183554f964778dffb7a@%3Cdev.pinot.apache.org%3E
Once the vote is passed from Pinot community, we can open the vote in general@incubating channel. The process & requirements are similar (3 +1 vote). Once the voting passes, you can close the voting by sending a voting result. The examples are shown at the following.
general@incubator vote https://lists.apache.org/thread.html/1e1e57cdd5361371f813b67f8caa0de9bc6bf81559bf4cd0b6de7dba@%3Cgeneral.incubator.apache.org%3E
general@incubator vote result https://lists.apache.org/thread.html/7255544ff38ea30d364ef49a2f3ee0a731e5fd26df48ddf63f63ba25@%3Cgeneral.incubator.apache.org%3E
Once the voting passes, it is now officially approved by Apache and we just need to do some final steps.
Finalize a Release
Move staging tarballs to the svn directory for official release
$ svn mv https://dist.apache.org/repos/dist/dev/incubator/pinot/apache-pinot-incubating-<version>-rc<rc_num> https://dist.apache.org/repos/dist/release/incubator/pinot/apache-pinot-incubating-<version>
Check https://dist.apache.org/repos/dist/release/incubator/pinot
After several hours later, check http://www.apache.org/dyn/closer.cgi/incubator/pinot/apache-pinot-incubating-<version> to see if it shows up.
(e.g. http://www.apache.org/dyn/closer.cgi/incubator/pinot/apache-pinot-incubating-0.1.0)Update git tag for official release
$ git checkout tags/release-<version>-rc<rc_num>$ git tag release-<version> -m "Apache Pinot (Incubating) <version>" $ git push origin release-<version>
Update Release note for the official release on Github https://github.com/apache/incubator-pinot/releases (Draft a new release, this time you don't need to check "pre-release" box)
# In case you want to clean up the rc tag to not to show on the Github release page, $ git tag -d release-<version>-rc<rc_num>$ git push origin :refs/tags/release-<version>-rc<rc_num>
- Publish the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Release button. Several hours later, the new version should show up at https://repo.maven.apache.org/maven2/org/apache/pinot
Announce to the world
Once you finalized the release, you now need to announce this to the world.
- Send the mail to dev@pinot, general@incubator about the release announcement
dev@pinot https://lists.apache.org/thread.html/1b58323b12666bdd855e66cdb3cd72462ab2e86b7cba32757bdf7393@%3Cdev.pinot.apache.org%3E
general@incubator https://lists.apache.org/thread.html/3c1778095a992f8df08bf4774c1e1aea8e0e1914bdab02a90f582ac4@%3Cgeneral.incubator.apache.org%3E - Update the Announcements on website
- Update https://pinot.apache.org/download page. Example pr: https://github.com/apache/incubator-pinot-site/pull/7
Reference
- http://plc4x.apache.org/developers/release.html
- https://cwiki.apache.org/confluence/display/GOBBLIN/Release+Process