Read Ahead material for Software Security Meeting pre-call, January, 2022. Context for this meeting: White House national security adviser asks software companies to discuss cybersecurity.

Executive Summary

  • The Apache Software Foundation (ASF) is one of the first and largest open source foundations, home to many high profile projects.
  • Background on the ASF (statistics, videos, and links) can be found at: https://apache.org/.
  • Software developed at the ASF is made available at no cost and without warranty, under a license permitting commercial modification and reuse without notification. Many commercial products include ASF software.
  • Projects act independently and set their own technical policies and directions and have their own communities of developers and management
  • Required policies exist for important supply-chain functions such as security vulnerability handling and notification, release management and distribution.

Foundation organization and governance structure (https://www.apache.org/foundation/governance/)

The ASF is a US 501(c)3 non-profit charity that acts as a steward for several hundred open source projects. The ASF is one of the largest open source organizations in the world, and is home to prolific projects such as Apache Hadoop, Apache Tomcat, Apache Cassandra, and scores of others. The ASF provides a known and vetted set of governance, intellectual property, and release processes as well as providing a vendor-neutral place for contributors to collaborate. However, within those processes there is a high degree of autonomy for each project. Projects decisions are driven by the people doing the work. The ASF Board of Directors is responsible for project oversight, ensuring the health and vitality of each project. 

In brief numbers the ASF has: 

  • 227 Million lines of code
  • 630,000+ contributors
  • 8,370+ committers (individuals who have earned the status of being able to directly change source code). 
  • 200+ projects

How project oversight works (https://www.apache.org/foundation/how-it-works.html)

ASF projects are run by Project Management Committees (PMCs). PMC members nominate and vote on new PMC members and collectively manage community, security, trademarks, or other problems. The PMC plays a key role in technical project oversight, but not the only role.  The full project community contributes by working, creating and reviewing patches and contributing to discussion.

The ASF regards all contributors as individuals and volunteers as part of its vendor neutrality. We do this despite the fact that many, if not most, of our contributors are paid by their employers to contribute to projects at the ASF. As part of that neutrality, we expect that all ASF contributors act in the best interest of the project. Additionally, who a contributor works for has no bearing on their standing in any particular project community. Contributors earn different levels of project standing based on their contributions to the project. 

How development works (https://www.apache.org/dev/pmc.html)

The ASF doesn't pick technologies or set technical direction for our projects.  Projects to have to adhere to various foundation-wide policies including trademarks, release policy, security vulnerability handling policy. However the number of mandated policies is kept to a minimum to allow projects to manage their own processes. For example there is no software development life cycle (SDLC) policy mandated by the foundation, each project is expected to set its own. 

We also require that everything happens in the open:

  • All bug reports are visible to all interested parties (with exception of reported security vulnerabilities, which are kept private until patched)
  • All commits to project source code are tracked and visible to all interested parties
  • All project-related discussion that can be public is public

ASF projects differ in how they do things, but most have documented process for working bug reports, introducing new features and planning and executing releases. As new features are developed and bugs are addressed, code changes are made that will go into releases.  Typically, proposed changes are submitted as “pull requests” (PRs) that explicitly describe how the source code will change if they are applied.  The original source, the PR and any changes that result from applying it are all publicly visible.  Any interested party can comment on proposed changes and it frequently happens that performance, security or other issues are discovered in the open development process that creates and modifies ASF software.

Although we use the term "volunteer" extensively in our materials, our contributors tend to be seasoned software professionals whose employers include ASF releases in their commercial products.

  • Our communities are healthy, open, and transparent.
  • As a result of vulnerability reports, more attention is focused on the specific product, at least for a period of time.  Open source enables a more diverse group of people to participate in these efforts.
  • Companies and government agencies that want to help don't need money or formal contracts to do so.  Join our mailing lists, review our code, contribute fixes.

How releases are versioned, validated and distributed (https://www.apache.org/dev/#releases)

PMCs follow ASF policies which require PMC reviews with minimum levels of voting required for new releases.

ASF software is meant to be consumed in the form of versioned releases.  ASF PMCs have various ways of packaging software for distribution, but the core asset being released is always source code.  Most projects release only source code and are not producing builds of their releases.  Most PMCs also make release candidates available to the public prior to or during release votes. Library or infrastructure projects sometimes make backward-incompatible changes as they develop new versions of their software.  Sometimes, a backward-compatible version of the software is added and maintained in parallel, but it often happens that the actively supported version is not backward compatible with an older version.   When library or infrastructure components make backward incompatible changes, downstream users may have to make code changes to upgrade their applications.

How ASF code is incorporated in other software

  • Software developed at the ASF is made available at no cost and without warranty

  • Commercial products may include ASF software without entering into any form of contract with the ASF, or even notifying us

    • Vendors of said commercial products though, must inform end users as indicated in section 4a of the Apache License, Version 2.0
    • Once informed, it is the responsibility of end users to keep track of whether and where an ASF software is used and which version(s), possibly with the help of a Software Asset Management (SAM)
    • End users may purchase these products but may have little interest or ability to apply fixes
  • There is a natural nesting that happens as software dependencies propagate through applications.  Addressing vulnerabilities in base level components (e.g. log4j) has a cascading impact.  Unless and until all downstream systems have effectively automated build, test and deployment systems, this creates systemic risk which has nothing to do with OSS per se.
  • ASF software is used in running systems in basically 2 ways:
    • A distributed system includes a running ASF product (e.g. Apache web server)
    • ASF software is integrated with other software to build applications.  When applications created are themselves used to create other applications, nesting happens, causing the impact of a vulnerability or incompatible change at the lowest level to have a cascading impact.   When a new release is made available by an ASF project, downstream users need to minimally make configuration changes and test their systems.  They typically have to rebuild and redeploy their applications.  In nested situations, this often has to happen sequentially.

How security reports are handled (https://www.apache.org/security/)

The ASF has a security team and a VP of security and acts with board authority.

The ASF encourages responsible disclosure of security vulnerabilities discovered in software managed by ASF projects.  The ASF security team sets a common policy, maintains security contacts for PMCs, and provides support for projects responding to security issues. Reporters are encouraged to use the designated security contacts to report vulnerabilities privately.  PMCs are required to respond to security reports promptly, working with reporters to investigate and if necessary develop patches.  

The ASF Security team is a CVE Project Candidate Naming Authority (CNA). CVE names are issued to vulnerabilities regardless if they are found by the project committers, members, PMCs, other ASF members, or third-parties.  The ASF Security team and PMCs work from time to time with third parties who wish to perform security functions such as code audits and bug bounties.  

The ASF Security team assist the PMCs in publishing a CVE once the vulnerability has been patched and a release containing the patch has been made available.  Vulnerability reports are sent to common and consistent locations including security lists, project development lists, and announce@apache.org. The security team oversees all reported issues across all ASF projects.  The security team reports monthly to the board and produces other public reports:

How end of support works

End of life is an industry standard concept that we apply at the ASF.  End of life means no more patches. Like all software products, ASF code lines go through a lifecycle that terminates in end of support.  Typically, ASF PMCs communicate end of support dates for products or code lines with ample time (one year plus) for users to plan upgrades or find alternatives.  Once communicated end of life dates arrive, the PMC is no longer expected or required to accept or apply patches to the end of life versions. This includes security patches against identified vulnerabilities. Therefore in some cases, when users who are depending on no longer supported versions of ASF software,  a version upgrade will be required on the users side to obtain a fix of an identified vulnerability which may also require code changes to be integrated.  The ASF security team may still issue CVE names against EOL software to help users quantify the risk. 

End of life software running in critical systems is a systemic risk which is not unique to open source software or the ASF.

Notes

The content above was developed on the security-discuss@community.apache.org mailing list, and perusing the archives there from late December 2021 to early January 2022 may provide important context.  Everyone is invited to participate there, but be aware that posts to that mailing list are publicly archived.  Background reading used to construct this page:


  • No labels

1 Comment

  1. Mark J. Cox

    Note that a snapshot of the page was sent earlier today, thanks to everyone who gave input or edited.  

    This page may not be required for the main meeting (or have different requirements), so future edits are useful but not essential and note they may not be used.