Versions of podman known to work include 3.4.1 (on RedHat) and 3.4.2 (on Ubuntu)
This is the workflow to create a release for either the Daffodil or the Daffodil VS Code Extension project.
You must be a Daffodil "Committer" to execute some of the steps of this workflow.
The following steps must only be performed only once:
Release files must be signed with an OpenPGP compatible key. If you do not already have a key for signing Apache releases, follow the developer instructions in the Daffodil KEYS file to generate a key and add it to the KEYS file. Follow the contributor workflow and create a review branch and pull request to commit your changes to the KEYS file.
Once merged, perform the following steps:
Clone the Apache Dist Daffodil release directory, copy the KEYS file, and commit it:
$ svn checkout https://dist.apache.org/repos/dist/release/daffodil/ daffodil-dist $ cp daffodil.git/KEYS daffodil-dist $ cd daffodil-dist $ svn ci -m "Update Apache Daffodil KEYS" |
Add your key fingerprint to https://id.apache.org. To get your fingerprint, run the following:
$ gpg --fingerprint KEYID |
Send your key to a keyserver via the command:
$ gpg --send-keys KEYID |
It is important for your keys to be signed by other trusted developers to create a web of trust. See Signing Keys and Getting Your Keys Signed.
For more information on signing keys, visit How to OpenPGP and Signing Releases.
To improve reproducibility and minimize the effects and variability of the user's environment, release candidates should be created using the "Daffodil release candidate container". Note that although commands to use this container have been tested with podman, you should be able to replace podman with docker if you would rather use it instead. Install the container software of choice using your systems package manager or from the container software website. For example:
$ sudo dnf install podman |
Versions of podman known to work include 3.4.1 (on RedHat) and 3.4.2 (on Ubuntu)
Versions of docker known to work include 20.10.11 (on Ubuntu 20.04).
The following steps should be performed prior to creating a release candidate:
containers/release-candidate/Dockerfile for dnf/npm packages with out of date versions (especially sbt and yarn installs). Dependabot does not automatically update these. If out of date, create and merge a pull request to update the versions.Prior to creating the release candidate, the version of the project still contains the -SNAPSHOT keyword. Create and merge a pull request to remove this -SNAPSHOT keyword in preparation for a non-snapshot release. The new version should not contain an -rcX suffix--the suffix is automatically added to release candidate artifact file names where necessary by the release candidate container.
Below are the steps one should follow to create a new release candidate.
Build the Daffodil release candidate image:
$ podman build -t daffodil-release-candidate /path/to/daffodil.git/containers/release-candidate/ |
This may take 10-15 minutes the first time, but should be significantly faster in subsequent runs unless something changes in the image where a full rebuild is needed.
This requires substantial free disk space. If you need to control where podman puts this storage, you can specify the --tmpdir, --root, and --runroot directories as ABSOLUTE paths. (Relative paths are known to cause issues.). Note that if you specify these additional flags on this podman build command you must also specify them for the podman run command below.Run the daffodil release candidate container. The command is here, but before running it, prepare the necessary inputs (below) that you will need to provide it when prompted. There are timeouts, and if you wait to dig these things up when prompted they will time out and you'll need to start over. Also, you must run this command in a standard terminal window that supports cursor-positioning via ANSI escape sequences, which it uses heavily.
$ podman run -it --privileged --group-add keep-groups --rm \
-v ~/.gitconfig:/root/.gitconfig \
-v ~/.gnupg/:/root/.gnupg/ \
-v ~/.ssh/:/root/.ssh/ \
--hostname daffodil.build \
daffodil-release-candidate |
Note: This must be run from an ordinary xterm/terminal window, as it uses escape sequences to position the cursor and text. It will not work in an Emacs shell buffer. Note that the -v option is used to bind mount files/directories from the host system in the container so the container has access to git configurations and gpg/ssh keys. The --dry-run option can be provided to the very end of the above podman command (e.g. ... daffodil-release-candidate --dry-run ) to perform a test without publishing any artifacts. An optional parameter can also be provided to perform a dry run with a different GitHub repository and branch (e.g. ... daffodil-release-candidate --dry-run user/repo@branch). The --entrypoint /bin/bash option can be provided before the last argument in the above podman command (e.g. ... --entrypoint /bin/bash daffodil-release-candidate ) to interactively view the configuration settings and manually run the daffodil-release-candidate script. This can be useful for debugging the container or testing changes.
rc1" if this is the first release candidate. This should not include the release version number (e.g. 2.0.0-rc1 ) – it should only contain the rcX part..gitconfig file. This is not your GitHub or Apache credentials--simply the name and email address you use for Daffodil commits. If you have Github's Email Privacy setting on, be sure to use the private email provided in your Github Settings, otherwise the complete-release script will fail.After entering the necessary information the script will run. It may output a few error messages that are not true errors about gpg using your private key as the default key for signing, and a few others also. The script will perform the actions listed below.
For Daffodil only, stage jars/poms to https://repository.apache.org along with their GPG signatures (".asc"), MD5 checksums (".md5"), and SHA1 checksums (".sha1").
Once the script completes, at that point the terminal window has a shell prompt where commands you type are being given to the running container.
You must complete the remaining steps without exiting this shell.
Verify the checksums and signatures are created in the Apache dist directories and are ready for commit, for example:
$ cd /root/daffodil-dist/ $ ls -R $ svn status |
A script that can be adapted to verify the signatures and checksums is in the comments of this page.
Verify the git tag is attached to the correct commit in the project repo, for example when releasing the Daffodil project:
$ git -C /root/daffodil/ log -n 1 |
For Daffodil only, verify the javadoc and scala docs for the version to be released exist in the daffodil site repository.
This can be done with git log:
$ git -C /root/daffodil-site/ log -n 1 -p |
or via the file system:
$ cd /root/daffodil-site/ $ ls -R |
A specific release of Daffodil may or may not be cross-compiled for more than one version of Scala. (As of Daffodil 3.3.0, only Scala 2.12 is supported, but future releases may support more than one version of Scala.) orgapachedaffodil-XXXX repository (there should be only one). Inspect the "Content" tab to make sure the appropriate jars are uploaded and appear valid. A script which can be adapted to download and verify the signatures and checksums for the contents of repository.apache.org is in a comment on this page. exit to close the container. All files/commits created in the container will be deleted.Run the command:
$ /root/complete-release |
All the previous commands have prepared commits and tags in the three repositories (Apache Dist, Daffodil, and Site). Running this command will push those commits and tag to the remote repositories.
exit to close the container.The previous steps have pushed the new scaladoc and javadoc docs to the Daffodil Site Repository repository. In order for your local sandbox copy of that repository to have this same content, you must fetch those updates, which is typically done via:
$ cd daffodil-site $ git checkout main $ git fetch --prune asf $ git rebase asf/main |
The most recent commit should then show the addition of new scaladoc and javadoc docs to the site.
Create a new release file in the site/_releases/ directory, updating the page to include a summary of the changes and links/descriptions of the bugs that were fixed in this release. The procedure for gathering the bugs fixed is as follows:
site/_releases/ directory, change all "[DAFFODIL-XXX]" to "{% jira XXXX %}", change all indentation and category lines to markdown format, and save your changes.For the section on Dependency updates, the following command shows the changes or new dependencies that should be mentioned.
$ git diff v1.0.0 -- project/Dependencies.scala |
You will of course change "v1.0.0" in that command to the tag of the prior release to the one you are preparing.
Parameters of the release file (near the top) that must be set because this is a release candidate include:
release: rc1 |
Look through the git log for any commits with the "Deprecation/Compatibility" keyword and copy that section of the commit message to the deprecation/compatibility section in the release notes.
Additionally, update the Unsupported Features page if any features/errata are now supported in this release.
Follow the steps in the README in the daffodil-site repository to test and publish the new release page.
With the release files published for staging and a website created, you may now start a vote on these files. To do so, send an email to dev@daffodil.apache.org based on the following example, making sure to update all links and version numbers. Make sure to update all links and use "plain text" editing for the email. HTML editing often leads to broken links or incorrect formatting.
If releasing the Daffodil VSCode Extension, make sure to use the correct project name, remove the "staging artifacts" wording and link, and reference GitHub Issues instead of JIRA.
Note that the link to daffodil-issues-2.0.0 should be created at https://s.apache.org, linking to the unreleased version in JIRA's Releases page showing the list of issues fixed by this release.
Note the https://dist.apache.org link instead https://downloads.apache.org because this is a pre-release.
Hi all, I'd like to call a vote to release Apache Daffodil 2.0.0-rc1. All distribution packages, including signatures, digests, etc. can be found at: https://dist.apache.org/repos/dist/dev/daffodil/2.0.0-rc1/ Staging artifacts can be found at: https://repository.apache.org/content/repositories/orgapachedaffodil-1000/ This release has been signed with PGP key 36F3494B033AE661, corresponding to slawrence@apache.org, which is included in the KEYS file here: https://downloads.apache.org/daffodil/KEYS The release candidate has been tagged in git with v2.0.0-rc1. For reference, here is a list of all resolved JIRA issues tagged with 2.0.0: https://s.apache.org/daffodil-issues-2.0.0 For a summary of the changes in this release, see: https://daffodil.apache.org/releases/2.0.0/ Please review and vote. The vote will be open for at least 72 hours (Sunday, 11 February 2018, 12 Noon EST). [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and reason why) Thanks, - Steve |
When committers test a new release candidate and vote for approving (or not approving it), they usually post a checklist showing what they verified to explain their vote. A very thorough person might check off an exhaustive checklist like the below one, although many use shorter checklists:
+1 [OK] verified signature of git tag [OK] verified hashes and signatures of source and helper binaries [OK] verified signatures use key in KEYS with apache email address [OK] verified source has no unexpected binary files [OK] verified source and git tag are same minus KEYS file [OK] verified source and helper binaries include LICENSE/NOTICE/README [OK] verified LICENSE/NOTICE/README look correct [OK] verified online JavaDoc and ScalaDoc docs look correct [OK] compiled source and ran all tests & ratCheck (LANG set to both en_US and de_DE) [OK] verified jars built from source have same content as helper binary jars [OK] verified dependencies in helper binaries are same as in maven poms [OK] tested bin & msi & rpm installers and checked "daffodil --version" output from each [OK] verified some public and private DFDL schema projects pass tests calling new release |
If any issues are discovered during the vote, the vote can be canceled and an rc2 created after the issues have been fixed and merged. After the message below has been sent, follow the same procedures as if the vote didn't pass.
Officially canceling the VOTE for 2.0.0-rc1. I'll create an rc2 and start a new VOTE. VOTE thread: https://lists.apache.org/thread.html/ra2fcf855251ed00c354abf29dfce73177fa3bf3a3705c5714f1aaabd%40%3Cdev.daffodil.apache.org%3E Thanks, - Steve |
After at least 72 hours, if the VOTE passes (at least 3 binding +1 votes and more positive than negative votes), create a RESULT thread announcing the passage and listing the binding and non-binding vote totals (you may omit those with zero totals) and vote breakdown. Binding votes are votes from PMC members and non-binding votes are votes from non-PMC members, which includes committers and community members. For example:
The VOTE to release Apache Daffodil 2.0.0-rc1 is now closed. The vote passes with: 3 binding +1 1 non-binding +1 The VOTE thread: https://lists.apache.org/thread.html/c8df54668fbcb7b8285f3e2cc524eac7cb82a721fa823ea5ae7edbe3%40%3Cdev.daffodil.apache.org%3E The vote breakdown is: +1 First Last (binding) +1 First Last (binding) +1 First Last (binding) +1 First Last Thanks to everyone who voted! |
If the VOTE does not pass, fix the issues, "Drop" the release at https://repository.apache.org (for Daffodil only), and repeat the "Create Release Candidate" process from the beginning with a new rcX number. Note that creating a new release candidate will automatically delete the files from the previous release candidate.
The following steps should be taken once the above vote passes.
Move the release candidate files to the release directory. (This prompts for your apache account password. It may prompt you about plaintext password storage. If so, it works even if you answer "no".)
For Daffodil:
$ svn mv -m "Release Apache Daffodil 2.0.0" \
https://dist.apache.org/repos/dist/dev/daffodil/2.0.0-rc1/ \
https://dist.apache.org/repos/dist/release/daffodil/2.0.0/ |
For Daffodil VS Code Extension:
$ svn mv -m "Release Apache Daffodil VSCode Extension 2.0.0" \
https://dist.apache.org/repos/dist/dev/daffodil/daffodil-vscode/2.0.0-rc1/ \
https://dist.apache.org/repos/dist/release/daffodil/daffodil-vscode/2.0.0/ |
Make a new clean clone (for good measure) and create a signed git tag based on the release candidate tag. You need your KEYID (See step 3b above) for your Apache signing key, and you will be prompted for the pass-phrase for your Apache signing key. Make sure to use daffodil-vscode.git when releasing Daffodil VS Code Extension.
$ cd /tmp
$ git clone git@github.com:apache/daffodil.git daffodil-2.0.0-rc1
$ cd daffodil-2.0.0-rc1
$ git tag -as -u KEYID -m "Release v2.0.0" v2.0.0 v2.0.0-rc1^{}
$ git push origin v2.0.0 |
You can delete the /tmp clone directory at this point.
https://dist.apache.org/repos/dist/release/daffodil/2.0.0/)Wait approximately 24 hours for the release files to sync to mirrors and Maven Central (for Daffodil only). To verify, check here
https://search.maven.org/ (ensure you can see daffodil libraries with versions matching all supported Scala versions; select one and verify the release versions exist in at least the sbt version)
https://www.apache.org/dyn/closer.lua/daffodil/2.0.0 (change 2.0.0 to the new version, pick a mirror, and ensure the files exist)
For Daffodil only, once the mirrors have synced, make the following changes to the daffodil site repository and publish them:
Modify the release page to have the following parameters:
release: final date: <date of release> |
Modify the site/doap.rdf file to include the release date and version, for example:
<release>
<Version>
<name>Apache Daffodil</name>
<created>2018-02-18</created>
<revision>2.0.0</revision>
</Version>
</release> |
Update the symlink to the latest Javadoc and Scaladoc docs in the site/docs directory (note that there is no forward slash at the end of latest).
$ cd site/docs/ $ ln -sfn 2.0.0 latest |
Now that the download URLS of the previous release point to the archive, remove that release from Apache dist to free up space on mirrors:
For Daffodil:
$ svn delete -m "Archive Apache Daffodil 1.0.0" \
https://dist.apache.org/repos/dist/release/daffodil/1.0.0/ |
For Daffodil VS Code Extension:
$ svn delete -m "Archive Apache Daffodil VS Code Extension 1.0.0" \
https://dist.apache.org/repos/dist/release/daffodil/daffodil-vscode/1.0.0/ |
For this access the to ASF VS Code publisher via your Visual Studio Marketplace account. This can be done by making a INFRA ticket in JIRA and tag Gavin McDonald. After you have been added you should be able to follow the steps below.
Send an announcement email from your apache.org email address to announce@apache.org, dev@daffodil.apache.org, and users@daffodil.apache.org, (note: send three separate emails instead of one email with multiple TO/CC's), with the below template. To send to announce@apache.org, your email app needs to be configured. Settings can be found on Apache's Committer Email page. For Gmail, you can request to "Send Email as" and with Thunderbird, you can add a new Outgoing SMTP Server and create a new Identity (Manage Identities) to send from the relay. Make sure to update links and use "plain text" editing for the email. HTML editing often leads to broken links or incorrect formatting. Make sure to update the project name and description when releasing the Daffodil VS Code Extension
The Apache Daffodil community is pleased to announce the release of version 2.0.0. Notable changes in this release include <short summary of changes>. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/2.0.0/ Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team |
Send a tweet from the @ApacheDaffodil twitter account, mentioning the release version, highlights of changes, and a link to the release page. You will need to be invited/accept, via Tweetdeck, permission to tweet from the account. At which point you may compose the below style of message from Tweetdeck.
The @ApacheDaffodil team is excited to announce the release of version 2.0.0! Notable changes include <short summary of changes>. Details and downloads at https://daffodil.apache.org/releases/2.0.0/ |
For some Daffodil releases DFDL schemas must also have updated library versions. (E.g., for release 3.3.0, in the build.sbt the scalaVersion must be "2.12.15" to enable Java 17 to work.) This is generally a good time to update other library versions used by the DFDL schema to newer versions consistent with those used by Daffodil, which are in daffodil/project/Dependencies.scala (e.g., junit and junit-interface).