*** Unsent and unapproved draft ***
Executive Summary / Key Take-Aways
You don't fix supply chain issues by focusing exclusively on the means of production.
If we could somehow make perfect releases of all ASF projects tomorrow, it would literally take years before we would see the effects. And plenty of cases where the update would never be applied.
Community is defined by those who show up.
We have plenty of components which are widely used, yet many of those who use them either don't review the code or chose not to contribute back their fixes.
Vulnerabilities aren't always isolated to a single component.
In the case of the recent log4j vulnerability, JNDI (a part of the Java runtime library) is a critical part of the exploit. We tend to treat concepts of "JNDI injection" as if it were a known problem, but just like "SQL injection" and other injection problems, this is demonstrably not the case. If it were possible to improve those common components, that could be a very big win.
How the ASF keeps our projects secure
- ASF projects are run by Project Management Committees (PMCs). PMC members nominate and vote on new PMC members and collectively manage community, security, trademarks, or other problems.
- Projects to have to adhere to various foundation-wide policies including trademarks, release policy, security vulnerability handling policy. ASF policies require PMC reviews with minimum levels of voting required for new releases.
- All changes to project source code are tracked and visible to all interested parties.
- The ASF has a security team and a VP of security who acts with board authority. The ASF security team sets a common policy, maintains security contacts for PMCs, and provides support for projects responding to security issues.
- When faced with a critical vulnerability we aim to get it mitigated fast. Once an issue is public and has increased attention, it sometimes leads to additional issues being detected or workarounds not being complete and we aim to be fast to release and fast to fix.
- We publish metadata about all vulnerabilities across all our projects in a consistent and machine-readable way.
Recommendations
Our experiences in dealing with Open Source Software and vulnerability handling over more than 25 years inform our positions on a number of items:
Businesses using open source need to react faster. Log4J and HeartBleed are being used as examples of open source vulnerability risks but it must be remembered that once these issues were reported to their respective projects they were dealt with quickly and efficiently. An update correcting HeartBleed was released within a week of notification of the issue to OpenSSL and prior to any exploitation. An update mitigating the Log4J vulnerability was released two weeks after notification to the Apache Software Foundation (with the release escalated and published a day after it was found to be being exploited in the wild). Both of these vulnerabilities were also examples which required skilled human researchers to discover and were not detected by automated tools.
What caused these, and other vulnerabilities, such as the Apache Struts issue in 2017, to be widely exploited was a failure of businesses mitigate in a timely manner: either by updating to a new release or applying mitigations.
While part of the solution may be to ensure companies know what they’ve included in their supply chain, they will also need to have processes for rapidly handling and disclosing vulnerabilities in their dependencies.
Users of open source software also need to keep track of lifecycles and ensure the projects they are using are still getting security updates.
- One of the most valuable things businesses that use open source can do is contribute back. Help fix bugs. Conduct security audits and feed back the results. Cash, while welcome and useful, isn't sufficient. We eagerly welcome audits and fixes from any source. We have a process defined for doing so: https://www.apache.org/security/. Our one caution is that we have prior experiences with audits and it is not productive when those audits produce a number of false positives. See Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA). It is worth noting that many automated means of performing an audit are more suited to detecting bugs inside a single component rather than identifying a vulnerability that is composed of combination of intentional features such as the one we saw with log4j. It is better when these audits are performed by skilled resources who are able to accurately identify actual issues, produce fixes, and therefore reduce the burden on each project.
Open Source projects should prioritise publishing security metadata quickly after a critical security issue is public. Many end users and tool vendors look to information published in the MITRE CVE database and in the downstream NIST NVD database. During 2021 MITRE have given CVE naming authorities (such as the ASF) the ability to publish information rapidly. Future changes are being made to that format and process in 2022 which evolve and enhance that capability.
For example for Log4J, the CVE was live on cve.org with full machine-readable metadata including affected versions in under 30 minutes from release.
For projects outside of the ASF, initiatives from platforms such as GitHub give projects the ability to obtain CVE names and publish vulnerability metadata. Other Open source projects which are deemed critical should work with the CVE project to become CVE naming authorities, allowing them to provide rapid publication and consumable updates in machine readable formats.
- We support the notion of identifying critical components. As our software is without cost and can be downloaded by anybody, we have no way of knowing how widely our projects are used. This means that by necessity determining if a component is critical can only either be determined by automated means which is always going to be incomplete and imperfect, or more manually such as by identifying critical products first, and then getting an inventory of what components those products embed. This problem is not unique to the ASF, so it makes sense for there to be industry standards and best practices. We are prepared to participate in these activities.
- As noted above, vulnerabilities can be a result of a how a number of components are combined.
The owner of a system is best placed to determine the criticality of that system and the degree to which individual components contribute to that criticality.
Criticality depends on usage. We need to avoid solutions that create burdens for all system owners using a software component when the component is only critical for some systems.
Appendix: About the recent issues in log4j
Some of our recommendations are based on recent experience with the resent log4j vulnerability. Some of the contributing factors:
- The IETF, W3C, and WHATWG are examples of standards organizations. Companies and invited experts collaborate there to produce specifications of standards, formats, and protocols. Along the way, those companies and individuals contribute their intellectual property to make this happen. Those standards are then implemented by commercial and open source products.
- LDAP is an example of one such standard. It is defined at the IETF. It's original use was for things like phone books and organization charts.
- Java is a programming language, and OpenJDK is the open source implementation of that language. Java contains a component, named JNDI ,that implements LDAP as well as a number of other standards such as CORBA and RMI. Those other standards provide the ability to do things like remote code execution which was popular in the 1990s, but now isn't quite so important.
- The Apache Software Foundation is an example of an open source organization. Companies and invited experts collaborate there to produce releases of software components and products. Along the way, those companies and individuals contribute their intellectual property to make this happen. Those releases are then incorporated into commercial and other open source products.
- Log4J is one such example of a software component produced by the ASF. It does the mundane job of producing an audit trail of important activities. Augmenting the audit trail with organization data from LDAP seemed like a good idea at the time. As Log4J is written in Java, it made use of JNDI to access that data.
- Put together, and you have a mundane and almost invisible function which can be tricked into doing remote code execution. There have been prior efforts to identify critical functions, yet somehow log4j never ranked high on any list. Furthermore, this vulnerability was not a bug (like a buffer overflow), but a combination of intentionally designed features when put together can be exploited. This went unnoticed for many years.
- Since the vulnerability was disclosed, we had a number of inquiries from people trying to determine out if they were affected by this vulnerability. It turns out many were running log4j version 1, which was discontinued in 2015. The good news is that this particular vulnerability does not affect them. The bad news is that the code they are running has not benefited from any security fixes in many years.
- The handling of the issues in log4j followed ASF processes and the fixes were timely.
Appendix: About the Apache Software Foundation
The Apache Software Foundation (ASF) is a US 501(c)3 non-profit charity that acts as a steward for several hundred open source projects. The ASF is one of the largest open source organizations in the world, and is home to prolific projects such as Apache Hadoop, Apache Tomcat, Apache Cassandra, and scores of others. The ASF provides a known and vetted set of governance, intellectual property, and release processes as well as providing a vendor-neutral place for contributors to collaborate. However, within those processes there is a high degree of autonomy for each project. Projects decisions are driven by the people doing the work. The ASF Board of Directors is responsible for project oversight, ensuring the health and vitality of each project.
In brief numbers the ASF has:
- 227 Million lines of code
- 630,000+ contributors
- 8,370+ committers (individuals who have earned the status of being able to directly change source code).
- 200+ projects