AOO 3.4.1 was our second release and this page is intended to reflect and review the development and release process towards this release to identify areas where we can improve things for future releases. We are still learning and trying to improve and to streamline our development and release process. The goal is to get more and more routine and to reduce the overhead for all people who are involved.
Initial brainstorming of things that were recognized by people:
What we did good:
- Good triage process that ensured the most critical defects were caught in 3.4.1.
- Clear test report that give people an overview of the 3.4.1 quality status, and which scenario/component/configuration was covered.
- Early preparation of the release notes and announcement, so that we can announce the release as soon as the voting passed.
- Use Bugzilla query instead of wiki to trace must fix defects which is more accurate and in time.
what can be improved:
- non working build bots, and not building the release branch
- too short time-frame between developer snapshots
- too less testing/review by the community on dev snapshots, start on RC only -> Solution: organize interim QA sessions, even far from RC phase, to detect problems early.
- RC were not communicated clearly. It would be far better for testers to name them "RC1", "RC2" and so on, on the download page at least.
- do we need a more formal written down release check list that we can easy reuse for every release. What is important here?
- RAT scan should be run earlier, or even with all buildbot builds
- We should do complete license/notice review even in a minor release where only a small number of items have changed. Why? Because we may have missed something in the previous release's review.
- There are tradeoff's in how frequently we issue new Release Candidates. Doing it frequently helps get the latest code out faster for final testing. But frequent RC's increases workload and churn on those who are building and uploading the RC's. Perhaps the way to make everyone happy would be to trigger RC's from a buildbot?
- Some of the tests were not completed in time for the final RC. We should try to coordinate this better, so all anticipated/required testing is done before RC is voted on.
- Need more detailed test plan/test cases to be published in advance. So that more volunteers can participate.
- For RC testing, need to update the test progress daily or every other day, since usually the voting will happen before the RC testing complete.
- A webpage/wiki to indicate RC clearly, not only on email thread.
- reposition Bugzilla as an opportunity backlog
- repository serves as a collection of thing we might do - no wrong answers
- position opportunity backlog as a source of release candidates
- establish process to identify release candidates
- establish process to map release candidates into release plan
- restore the ability to query by votes in Bugzilla
1 Comment
Kay Schenk
From my perspective, having working buildbots (with appropriate caretakers) is critical to the credibility of our binary releases. This is not to diminish the efforts of volunteer developers on their individual systems. I guess I am feeling, given mentor comments on general@incubator for the 3.4.1 release, we need to reinforce both to the ASF and our user base WHAT steps we take to produce these binary releases and why what we do is safe and will work for them. This should be documented via a page on the project site as well I think.
Which meshes well with this item--
"do we need a more formal written down release check list that we can easy reuse for every release. What is important here?"
I would answer yes, but I don't know what to include. This should be documented as well as part of a QA process available for public consumption.
I know many opensource projects will not undergo the kind of scrutiny OpenOffice will/does. But this product has a history in commercial supply/support. So, I feel we need to shore up any items that speak to credibility/assurance.