...

Tag the pinot docs for the given version

Please ask permission for gitbook Apache Pinot organization access in #pinot-committers slack channel.

...

Check out the codebase (the release branch created while preparing for the release). Add the following dependencies to pinot-distribution/pom.xml (otherwise there will be missing lib dependencies) but don't check it in. (NOTE: Please double check if the following includes all packages in pinot-connectors and pinot-plugins. If not, please update the following code block.)

    <!-- pinot-plugins/pinot-batch-ingestion -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-common</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-hadoop</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-spark-2.4</artifactId>
      <version>${project.version}</version>
    </dependency> 
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-spark-3.2<3</artifactId>
      <version>${project.version}</version>
    </dependency>      
     <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-batch-ingestion-standalone</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-environment -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-azure</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-file-system -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-adls</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-gcs</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-hdfs</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-s3</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-input-format -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-avro-avro<base</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-avro-base<avro</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-clp-csv<log</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-confluent-avro</artifactId>
     <version>${project.version}</version>
   </dependency>
   <dependency>
     <groupId>org.apache.pinot</groupId>
     <artifactId>pinot-csv</artifactId>
     <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-json</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-orc</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-parquet</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-protobuf</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-thrift</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-metrics -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-dropwizard</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-yammer</artifactId>
      <version>${project.version}</version>
    </dependency>
 

     <<!-- pinot-plugins/pinot-minion-tasks -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-minion-builtin-tasks</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-segment-uploader -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-segment-uploader-default</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-segment-writer -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-segment-writer-file-based</artifactId>
      <version>${project.version}</version>
    </dependency>
 
    <!-- pinot-plugins/pinot-stream-ingestion -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-02.9<0</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-2.0<base</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kafka-base<kinesis</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-kinesis<pulsar</artifactId>
      <version>${project.version}</version>
    </dependency>
 
   <!-- pinot-connectors -->
   <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-flink-pulsar<connector</artifactId>
      <version>${project.version}</version>
    </dependency> 

    <!-- pinot-connectors -->
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-spark-flink2-connector</artifactId>
      <version>${project.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>pinot-spark-3-connector</artifactId>
      <version>${project.version}</version>
    </dependency> 
    <dependency>
      <groupId>org.apache.pinot</groupId>
      <artifactId>presto<artifactId>pinot-pinotspark-driver<common</artifactId>
      <version>${project.version}</version>
    </dependency> 

After that, run the commands:

mvn -Ddependency.locations.enabled=false project-info-reports:dependencies -Ppresto-driver -pl :pinot-distribution
# This will create a file in pinot-distribution/target/site/dependencies.html

Open the file dependencies.html (mentioned above) in a browser. You will need to get two parts of the screen onto raw files by copy-pasting it:

...

Open the file dependencies.html (mentioned above) in a browser. You will need to get two parts of the screen onto raw files by copy-pasting it:

1. Create a fresh working directory ($WORKING_DIR), cd to it.
2. Scroll down to the sections on the project's compile dependencies (direct and transitive). Grab the table using the mouse, and save the result to a text file. Name the file pkg-dependencies-raw.txt. Keep the browser page open, you will need it later.
3. Scroll down to the project's runtime dependencies. Grab the table using the mouse and save the result to a text file named runtime-dependencies-raw.txt.
   
   

Do NOT trust the License column in the browser. You will need to manually verify the license in a later step by following the package links in the page.

You can now work on this text file to generate the LICENSE-binary file for the next release

cd $WORKING_DIR
# First, get the new list of dependencies, sorted
cat pkg-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq > /tmp/x1
cat runtime-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq >> /tmp/x1
sort /tmp/x1 > new-pkg-versions.txt

# Now pull out all package version lines from the existing LICENSE-binary file into another file, call it current-pkg-versions.txt.unsorted
cp /path/to/existing/source/LICENSE-binary ./LICENSE-binary
cp LICENSE-binary current-pkg-versions.txt.unsorted
vim current-pkg-versions.txt.unsorted # Now, remove all fluff in the file except for the lines that hold package name and versions.
sort current-pkg-versions.txt.unsorted | uniq > current-pkg-versions.txt

# Now you can compare (diff) the two files to see which packages have been removed, and which ones have been added.

# I use tkdiff, but you can use 'diff -y' or some of the more fancy options of diff that helps you get what you want.
tkdiff current-pkg-versions.txt new-pkg-versions.txt

Once you have the old and new packages with version numbers, do the following:

1. Identify which packages are no longer in use. Remove them from LICENSE-binary file. If that license is no longer used, then remove the license file from licenses-binary folder.
   1. Tip: use one liner  diff -y current-pkg-versions.txt new-pkg-versions.txt |grep "<"|awk '{print $1}' |xargs -I {} sed -i '' '/{}/d' LICENSE-binary
2. Identify packages that have changed versions. Go the package's home page (as pointed by the browser page that has the dependencies), and verify that the license terms have not changed. If they have, then remove the package name from the  license section, and treat it as a new package dependency (in the next step)
3. Identify all new package dependencies (including those from step 2 above). Click on the package's home page (as pointed by the browser page that has dependencies), and examine the license. You may need to traverse other pages or to the code base in order to finally get the license.
   1. If the license is one of the categories that already exist in the licenses-binary folder, just add the package name and version to the LICENSE-binary file.
   2. If it is a new kind of license
      1. Make sure to create a file in licenses-binary directory copying the license terms from the package's LICENSE file or from its website as the case may be.
      2. Create a new section in LICENSE-binary file just like the sections for other licenses.
      3. copy the package name from new-pkg-versions.txt under the section for that license.
4. Sort the package names in each section of the LICENSE-binary file. This is useful step that makes it easier to identify the changes when building the next release.

Update NOTICE-binary file

This file is a concatenation of all the NOTICE files of packages we use.

In order to generate this file, we need to add a maven-shade-plugin at root pom.xml.

Note that, you need to search <artifactId>maven-shade-plugin</artifactId> in all the submodules pom.xml. Then remove all the found maven-shade-plugin plugins.

Edit pom.xml temporarily (DO NOT CHECKIN)

Change the top level pom.xml as follows:

diff --git a/pom.xml b/pom.xml
index e72be7717..a38f08915 100644
--- a/pom.xml
+++ b/pom.xml
@@ -889,6 +889

...

Do NOT trust the License column in the browser. You will need to manually verify the license in a later step by following the package links in the page.

You can now work on this text file to generate the LICENSE-binary file for the next release

cd $WORKING_DIR
# First, get the new list of dependencies, sorted
cat pkg-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq > /tmp/x1
cat runtime-dependencies-raw.txt | awk '{printf("%s:%s:%s\n", $1, $2, $3);}' | grep -v org.apache.pinot | sort | uniq >> /tmp/x1
sort /tmp/x1 > new-pkg-versions.txt

# Now pull out all package version lines from the existing LICENSE-binary file into another file, call it current-pkg-versions.txt.unsorted
cp /path/to/existing/source/LICENSE-binary ./LICENSE-binary
cp LICENSE-binary current-pkg-versions.txt.unsorted
vim current-pkg-versions.txt.unsorted # Now, remove all fluff in the file except for the lines that hold package name and versions.
sort current-pkg-versions.txt.unsorted | uniq > current-pkg-versions.txt

# Now you can compare (diff) the two files to see which packages have been removed, and which ones have been added.

# I use tkdiff, but you can use 'diff -y' or some of the more fancy options of diff that helps you get what you want.
tkdiff current-pkg-versions.txt new-pkg-versions.txt

Once you have the old and new packages with version numbers, do the following:

1. Identify which packages are no longer in use. Remove them from LICENSE-binary file. If that license is no longer used, then remove the license file from licenses-binary folder.
   1. Tip: use one liner  diff -y current-pkg-versions.txt new-pkg-versions.txt |grep "<"|awk '{print $1}' |xargs -I {} sed -i '' '/{}/d' LICENSE-binary
2. Identify packages that have changed versions. Go the package's home page (as pointed by the browser page that has the dependencies), and verify that the license terms have not changed. If they have, then remove the package name from the  license section, and treat it as a new package dependency (in the next step)
3. Identify all new package dependencies (including those from step 2 above). Click on the package's home page (as pointed by the browser page that has dependencies), and examine the license. You may need to traverse other pages or to the code base in order to finally get the license.
   1. If the license is one of the categories that already exist in the licenses-binary folder, just add the package name and version to the LICENSE-binary file.
   2. If it is a new kind of license
      1. Make sure to create a file in licenses-binary directory copying the license terms from the package's LICENSE file or from its website as the case may be.
      2. Create a new section in LICENSE-binary file just like the sections for other licenses.
      3. copy the package name from new-pkg-versions.txt under the section for that license.
4. Sort the package names in each section of the LICENSE-binary file. This is useful step that makes it easier to identify the changes when building the next release.

Update NOTICE-binary file

This file is a concatenation of all the NOTICE files of packages we use.

In order to generate this file, we need to add a maven-shade-plugin at root pom.xml.

Note that, you need to search <artifactId>maven-shade-plugin</artifactId> in all the submodules pom.xml. Then remove all the found maven-shade-plugin plugins.

Edit pom.xml temporarily (DO NOT CHECKIN)

Change the top level pom.xml as follows:

diff --git a/pom.xml b/pom.xml
index e72be7717..a38f08915 100644
--- a/pom.xml
+++ b/pom.xml
@@ -889,6 +889,31 @@
     <defaultGoal>clean install</defaultGoal>
     <pluginManagement>
       <plugins>
+<plugin>
+  <groupId>org.apache.maven.plugins</groupId>
+  <artifactId>maven-shade-plugin</artifactId>
+  <configuration>
+    <shadedArtifactAttached>false</shadedArtifactAttached>
+    <artifactSet>
+      <includes>
+        <include>*:*</include>
+      </includes>
+    </artifactSet>
+  </configuration>
+  <executions>
+    <execution>
+      <phase>package</phase>
+      <goals>
+        <goal>shade</goal>
+      </goals>
+      <configuration>
+        <transformers>
+          <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer"/>
+        </transformers>
+      </configuration>
+    </execution>
+  </executions>
+</plugin>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-compiler-plugin</artifactId>

...

We need to go through the voting at dev@pinot mailing list. Send out the mail for voting. The example is present at the following links. The voting should be open for at least 72 hours and it requires a minimum of three +1 votes. Once the voting passes, you can send out the result mail. Examples:

...

https://lists.apache.org/thread.html/rdd8d44971e6b4539f40240005cf3a0af1a81e493f2ef2a7e57b360d1%40%3Cdev.pinot.apache.org%3E

the result mail. Examples:

Once both votings passed, it is now officially approved by Apache and we just need to do some final steps.

Recall a Release Candidate

There could be some issues unveiled during the testing and voting phase. If it happens, we need to recall the release candidate and apply patches/changes then redo from step: Prepare the Release Candidate and increase <rc_num> by 1.

There are several things to clean up:

1. Delete git repo release draft: https://github.com/apache/pinot/releases
2. Delete the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Drop button.

3. Clean up tarballs from dev dist

   Code Block
   language bash
   $ svn rm https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC -m "Delete release-$VERSION-rc$RC" # NOTE: You will need your apache login and password for this step.

   
   

Finalize a Release

1. Move staging tarballs to the svn directory for official release. You need to be a PMC member to be able to do this step. If you are a committer, ask a PMC member to do this step for you.

   Code Block
   language bash
   $ svn mv https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC https://dist.apache.org/repos/dist/release/pinot/apache-pinot-$VERSION # NOTE: You will need your apache login and password for this step.

   Check https://downloads.apache.org/pinot
   After several hours later, check https://archive.apache.org/dist/pinot/apache-pinot-<version>/  to see if it shows up.
   (e.g. https://archive.apache.org/dist/pinot/apache-pinot-0.8.0/)

2. Delete previous release tarballs. Similar to step 1, you need to be PMC member to be able to perform this step.

   We only need to keep the most recent release, so feel free to delete any previous released distributions.

   Code Block
   language bash
   $ svn rm https://dist.apache.org/repos/dist/release/pinot/apache-pinot-0.8.0 # NOTE: You will need your apache login and password for this step.

   
   

3. Delete release branch from github repo if any

   Code Block
   language bash
   $ git push origin --delete release-$VERSION-rc

   
   

4. Update git tag for official release

   Code Block
   language bash
   $ git checkout tags/release-$VERSION-rc$RC $ git tag release-$VERSION -m "Apache Pinot $VERSION" $ git push origin release-$VERSION

   
   

5. Update Release note for the official release on Github https://github.com/apache/pinot/releases

   1. No need to draft a new release, just EDIT the previously drafted release

   2. Replace tag name to release-<version> tag

   3. Uncheck the pre-release checkbox, then publish

   4. Delete all other rc drafts if there is any.

6. Delete all RC tags from github repo

   Code Block
   language bash
   # Clean up the rc tag to not to show on the Github release page $ git tag -d release-$VERSION-rc$RC $ git push origin :refs/tags/release-$VERSION-rc$RC

   
   

7. Publish

Once both votings passed, it is now officially approved by Apache and we just need to do some final steps.

Recall a Release Candidate

There could be some issues unveiled during the testing and voting phase. If it happens, we need to recall the release candidate and apply patches/changes then redo from step: Prepare the Release Candidate and increase <rc_num> by 1.

There are several things to clean up:

1. Delete git repo release draft: https://github.com/apache/pinot/releases
2. Delete the maven artifacts. Log in to Nexus, select the staging repository (orgapachepinot-<number>) and click Drop button.

   Clean up tarballs from dev dist

   Code Block

   language	bash

   $ svn rm Release button. Several hours later, the new version should show up at https://distrepo.maven.apache.org/reposmaven2/distorg/dev/pinot/apache-pinot-$VERSION-rc$RC -m "Delete release-$VERSION-rc$RC" # NOTE: You will need your apache login and password for this step.

Finalize a Release

...

Move staging tarballs to the svn directory for official release. You need to be a PMC member to be able to do this step. If you are a committer, ask a PMC member to do this step for you.

$ svn mv https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-$VERSION-rc$RC https://dist.apache.org/repos/dist/release/pinot/apache-pinot-$VERSION
# NOTE: You will need your apache login and password for this step.

Check https://downloads.apache.org/pinot
After several hours later, check https://archive.apache.org/dist/pinot/apache-pinot-<version>/  to see if it shows up.
(e.g. https://archive.apache.org/dist/pinot/apache-pinot-0.8.0/)

Delete previous release tarballs. Similar to step 1, you need to be PMC member to be able to perform this step.

We only need to keep the most recent release, so feel free to delete any previous released distributions.

$ svn rm https://dist.apache.org/repos/dist/release/pinot/apache-pinot-0.8.0
# NOTE: You will need your apache login and password for this step.

Delete release branch from github repo if any

$ git push origin --delete release-$VERSION-rc

Update git tag for official release

$ git checkout tags/release-$VERSION-rc$RC
$ git tag release-$VERSION -m "Apache Pinot $VERSION"
$ git push origin release-$VERSION

1. apache/pinot

2. Build docker image for the release:

   1. Please ask Xiang Fu to grant you permission to run the jobs on this repo: https://github.com/apachepinot/pinot-fork/actions.
   2. There are total 5 jobs related to different JDKs(Amazon Corretto 11/17, Microsoft OpenJDK 11/17 and OpenJDK 21)  to run:
      1. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-amazoncorretto-11-pinot-docker-image.yml
      2. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-amazoncorretto-17-pinot-docker-image.yml
      3. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-ms-openjdk-11-pinot-docker-image.yml
      4. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-ms-openjdk-17-pinot-docker-image.yml
      5. https://github.com/apachepinot/pinot-fork/actions/workflows/build-multi-arch-java21-pinot-docker-image.yml 

   3. Build images with release tag, and tag it with both tags: release-<version>  and <version> , e.g. release-1.2.0  and 1.2.0 . See below graph for the input reference.Image Added

   4. After all the tasks are done, you should find all the docker images here: https://hub.docker.com/r/apachepinot/pinot/tags?page=&page_size=&ordering=&name=release-1.2.0
   5. Tag official release and push, making sure you asked Xiang Fu for the dockerhub push permission for ApachePinot org.

docker manifest create apachepinot/pinot:1.2.0 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-arm64 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-amd64
docker manifest create apachepinot/pinot:release-1.2.0 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-arm64 --amend apachepinot/pinot:release-1.2.0-17-amazoncorretto-linux-amd64

docker manifest push apachepinot/pinot:1.2.0
docker manifest push apachepinot/pinot:release-1.2.0

...

Update Release note for the official release on Github https://github.com/apache/pinot/releases

1. No need to draft a new release, just EDIT the previously drafted release

2. Replace tag name to release-<version> tag

3. Uncheck the pre-release checkbox, then publish

4. Delete all other rc drafts if there is any.

Delete all RC tags from github repo

# Clean up the rc tag to not to show on the Github release page
$ git tag -d release-$VERSION-rc$RC
$ git push origin :refs/tags/release-$VERSION-rc$RC

...

Build images with release tag, and tag it with both tags: release-<version>  and <version> , e.g. release-0.13.0  and 0.13.0 . See below graph for the input reference.

...

Announce to the world

Once you finalized the release, you now need to announce this to the world. 

1. Update https://pinot.apache.org/download page. Example pr: https://github.com/apache/pinot-site/pull/66131
2. Apache takes time to archive all the distributions in SVN release repo, so we don't need to keep all the distributions in SVN release repo. For download URL, we use https://downloads.apache.org for the most recent dist and https://archive.apache.org/dist for old dists.
   1. SVN release repo distribution can be downloaded from: https://downloads.apache.org e.g. https://downloads.apache.org/pinot/apache-pinot-0.9.1/apache-pinot-0.9.1-bin.tar.gz
   2. Old distributions are available through the archive urls, please make sure you update download links for old dist from https://downloads.apache.org to https://archive.apache.org/dist. E.g. https://archive.apache.org/dist/pinot/apache-pinot-0.9.0/apache-pinot-0.9.0-bin.tar.gz Sample PR: https://github.com/apache/pinot-site/pull/61
3. Update Pinot github repo DOAP file to reflect the latest release on https://projects.apache.org/project.html?pinot, sample PR: https://github.com/apache/pinot/pull/11827
4. Update Pinot docs: https://docs.pinot.apache.org/basics/releases
   1. Guidance on updating the page: https://docs.pinot.apache.org/developers/developers-and-contributors/update-document.Publish versioned docs:
   2. Go to gitbook, select the space previously created, on the right upper corner, click Publish  button and select In Collection.Image Removed
   Check the docs are available at https://docs.pinot.apache.org/. You should be able to find it in the right upper side bar:Image Removed
   3. Raise a PR similar to (you can raise a single PR): pinot-docs/commit/d1849be pinot-docs/commit/0f035df against the pinot-docs repo. Use the release notes you had used and prepared before.
5. Send the mail to dev@pinot.apache.org about the release announcement
   dev@pinot https://lists.apache.org/thread/qw3chy5rvclymzwk0ros98nlk7mxwhos
6. Send the mail to announce@apache.org  about the release announcement.  Remember to use plain text mode:     

   
   

   

...