This version contains edits not present in the version sent to the White House

Executive Summary / Key Take-Aways

  • We can't fix open source supply chain issues by focusing exclusively on the upstream producer

    • Even perfect releases can take years to be adopted and deployed by downstream providers.

  • Community is defined by those who show up and do the work

    • Companies that build open source into their products rarely participate in their continued maintenance.

    • The ASF provides a neutral forum for collaborative maintenance and development that empowers the individuals doing the work (often supported by their employers), which has been shown to effectively sustain software projects for over two decades.

    • However, only a tiny percentage of downstream companies (reusing the same code within their own products) choose to participate.

    • Security directives MUST avoid placing additional unfunded burdens on the few maintainers who are already doing the work.

  • Vulnerabilities are rarely isolated to a single component

    • The recent Apache Log4j vulnerability was an unfortunate combination of independently designed features within the Java platform.

    • Disabling antiquated and unnecessary features, at least within a default configuration, would have prevented the vulnerability.

    • Improving platform security to prevent indirect activation of unsafe operations could be a very big win.

How the ASF addresses project security

  • Each ASF project is governed by a Project Management Committee (PMC) consisting of individuals who have earned merit by working on the project.
    • PMC members collectively manage their own community decisions regarding technical development and releases.
    • Projects adhere to various foundation-wide policies, including for trademarks, making releases, and handling reported security vulnerabilities.
    • ASF policies require PMC reviews with minimum levels of voting required for new releases.
    • Changes to project source code are tracked and visible to all interested parties.
  • The ASF has a security team and a VP of security who acts with board authority.
    • The ASF security team sets a common policy, maintains security contacts for PMCs, and provides support for projects responding to security issues.
    • We publish metadata about all vulnerabilities across all our projects in a consistent and machine-readable way.
  • When faced with a critical vulnerability, we aim to mitigate the threat before disclosure.
    • Initial reports are processed in private under rules for responsible disclosure.
    • Once an issue becomes public, the focused attention often leads to additional issues being detected.
    • We aim to be fast to release and fast to fix, even if that requires multiple releases.

Recommendations

Our experiences in dealing with Open Source Software and vulnerability handling over more than 25 years inform our positions on a number of items:

  • Businesses using open source need to understand what components are in use and be prepared to act quickly when vulnerabilities are discovered. 

    Log4j and HeartBleed are being used as examples of open source vulnerability risks but it must be remembered that once these issues were reported to their respective projects they were dealt with quickly and efficiently.  An update correcting HeartBleed was released     within a week of notification of the issue to OpenSSL and prior to any exploitation.  An initial update mitigating the Log4J vulnerability was released within two weeks after notification to the Apache Software Foundation (with the release escalated and published a day after it was found to be being exploited in the wild). Both of these vulnerabilities were also examples which required skilled human researchers to discover and were not detected by automated tools.

    What caused these, and other vulnerabilities, such as the Apache Struts issue in 2017, to be widely exploited was a failure of businesses to mitigate in a timely manner: either by updating to a new release or applying mitigations.

    While part of the solution may be to ensure companies know what they’ve included in their supply chain, they will also need to have processes for rapidly handling and disclosing vulnerabilities in their dependencies.

    Users of open source software also need to keep track of lifecycles and ensure the projects they are using are still getting security updates.

  • One of the most valuable things businesses that use open source can do is contribute back.

    Help fix bugs. Conduct security audits and feed back the results.  Cash, while welcome and useful, isn't sufficient.      We eagerly welcome audits and fixes from any source.  We have a process defined for doing so: https://www.apache.org/security/.

    Our one caution is that we have prior experiences with audits and it is not productive when those audits produce a number of false positives.  See      Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA).  It is worth noting that many automated means of performing an audit are more suited to detecting bugs inside a single component rather than identifying a vulnerability that is composed of combination of intentional features such as the one we saw with Log4j.  It is better when these audits are performed by skilled researchers who are able to accurately identify actual issues, produce fixes, and therefore reduce the burden on each project.

  • Open Source projects should prioritize publishing machine readable security metadata quickly after a critical security issue is public. 

    For example, many end users and tool vendors look to information published in the MITRE CVE database and in the downstream NIST NVD database. During 2021 MITRE have given CVE naming authorities (such as the ASF) the ability to publish information rapidly and     for Log4j, the CVE was live on cve.org with full machine-readable metadata including affected versions in under 30 minutes from release.

  • We support the notion of identifying critical components. 

    As our software is without cost and can be downloaded by anybody, we have no way of knowing how widely our projects are used. This means that, by necessity, determining if a component is critical can only either be determined by automated means which is always going to be incomplete and imperfect, or more manually such as by identifying critical products first, and then getting an inventory of what components those products embed.  

    This problem is not unique to the ASF, so it makes sense for there to be industry standards and best practices.  We are prepared to participate in these activities. 
    • As noted above, vulnerabilities can be a result of a how a number of components are combined.

    • The owner of a system is best placed to determine the criticality of that system and the degree to which individual components contribute to that criticality.

    • Criticality depends on usage. We need to avoid solutions that create burdens for all system owners using a software component when the component is only critical for some systems.

Appendix: About the recent issues in Log4j

Some of our recommendations are based on recent experience with the Log4j vulnerability.  Some of the contributing factors:

  • The IETF, W3C, and WHATWG are examples of standards organizations. Companies and invited experts collaborate there to produce specifications of standards, formats, and protocols. Along the way, those companies and individuals contribute their intellectual property to make this happen. Those standards are then implemented by commercial and open source products.
  • LDAP is an example of one such standard. It is defined at the IETF.  It's original use was for things like phone books and organization charts.
  • Java is a programming language, and OpenJDK is the open source implementation of that language. Java contains a component, named JNDI, that implements LDAP as well as a number of other standards such as CORBA and RMI.  Those other standards provide the ability to do things like remote code execution which was popular in the 1990s, but now isn't quite so important and may open the door to security issues.
  • The Apache Software Foundation is an example of an open source organization. Self managed groups of experts (e.g. academics, people seconded by their company or otherwise voluntary) collaborate there to produce releases of software components and products.  In this way the industry as a whole (e.g. companies, academia, the public sector and individuals) contribute their intellectual property to make this happen.
  • Log4j is one example of a software component produced by the ASF. It does the mundane job of producing an audit trail of important activities. Augmenting the audit trail with organization data from LDAP seemed like a good idea at the time. As Log4j is written in Java, it made use of JNDI to access that data.
  • Put together, and you have a mundane and almost invisible function which can be tricked into doing remote code execution. There have been prior efforts to identify critical functions, yet somehow Log4j never ranked high on any list. Furthermore, this vulnerability was not a bug (like a buffer overflow), but a combination of intentionally designed features when put together can be exploited. This went unnoticed for many years. A mundane and nearly invisible function that can be exploited is in no way unique to open source.
  • Since the vulnerability was disclosed, we had a number of inquiries from people trying to determine if they were affected by this vulnerability. It turns out many were running Log4j version 1, which was discontinued in 2015. The good news is that this particular vulnerability does not affect them. The bad news is that the code they are running has not benefited from any security fixes in many years.
  • The handling of the issues in Log4j followed ASF processes and the fixes were timely.

Appendix: About the Apache Software Foundation

The Apache Software Foundation (ASF) is a US 501(c)3 non-profit charity that acts as a steward for several hundred open source projects. The ASF is one of the largest open source organizations in the world, and is home to prolific projects such as Apache Hadoop, Apache Tomcat, Apache Cassandra, and scores of others. The ASF provides a known and vetted set of governance, intellectual property, and release processes as well as providing a vendor-neutral place for contributors to collaborate. However, within those processes there is a high degree of autonomy for each project. Projects decisions are driven by the people doing the work. The ASF Board of Directors is responsible for project oversight, ensuring the health and vitality of each project. 

In brief numbers the ASF has: 

  • 237 Million lines of code
  • 650,000+ contributors
  • 8,370+ committers (individuals who have earned the status of being able to directly change source code). 
  • 350+ projects

Appendix: Further reading

  • No labels