...
This version contains edits not present in the version sent to the White House
Executive Summary / Key Take-Aways
...
Businesses using open source need to understand what components are in use and be prepared to act quickly when vulnerabilities are discovered.
Log4j and HeartBleed are being used as examples of open source vulnerability risks but it must be remembered that once these issues were reported to their respective projects they were dealt with quickly and efficiently. An update correcting HeartBleed was released within a week of notification of the issue to OpenSSL and prior to any exploitation. An initial update mitigating the Log4J vulnerability was released within two weeks after notification to the Apache Software Foundation (with the release escalated and published a day after it was found to be being exploited in the wild). Both of these vulnerabilities were also examples which required skilled human researchers to discover and were not detected by automated tools.What caused these, and other vulnerabilities, such as the Apache Struts issue in 2017, to be widely exploited was a failure of businesses to mitigate in a timely manner: either by updating to a new release or applying mitigations.
While part of the solution may be to ensure companies know what they’ve included in their supply chain, they will also need to have processes for rapidly handling and disclosing vulnerabilities in their dependencies.
Users of open source software also need to keep track of lifecycles and ensure the projects they are using are still getting security updates.
- One of the most valuable things businesses that use open source can do is contribute back.
Help fix bugs. Conduct security audits and feed back the results. Cash, while welcome and useful, isn't sufficient. We eagerly welcome audits and fixes from any source. We have a process defined for doing so: https://www.apache.org/security/.
Our one caution is that we have prior experiences with audits and it is not productive when those audits produce a number of false positives. See Feedback from The Apache Software Foundation on the Free and Open Source Security Audit (FOSSA). It is worth noting that many automated means of performing an audit are more suited to detecting bugs inside a single component rather than identifying a vulnerability that is composed of combination of intentional features such as the one we saw with Log4j. It is better when these audits are performed by skilled researchers who are able to accurately identify actual issues, produce fixes, and therefore reduce the burden on each project. Open Source projects should prioritize publishing machine readable security metadata quickly after a critical security issue is public.
For example, many end users and tool vendors look to information published in the MITRE CVE database and in the downstream NIST NVD database. During 2021 MITRE have given CVE naming authorities (such as the ASF) the ability to publish information rapidly and for Log4j, the CVE was live on cve.org with full machine-readable metadata including affected versions in under 30 minutes from release.- We support the notion of identifying critical components.
As our software is without cost and can be downloaded by anybody, we have no way of knowing how widely our projects are used. This means that, by necessity, determining if a component is critical can only either be determined by automated means which is always going to be incomplete and imperfect, or more manually such as by identifying critical products first, and then getting an inventory of what components those products embed.
This problem is not unique to the ASF, so it makes sense for there to be industry standards and best practices. We are prepared to participate in these activities.- As noted above, vulnerabilities can be a result of a how a number of components are combined.
The owner of a system is best placed to determine the criticality of that system and the degree to which individual components contribute to that criticality.
Criticality depends on usage. We need to avoid solutions that create burdens for all system owners using a software component when the component is only critical for some systems.
...
- The IETF, W3C, and WHATWG are examples of standards organizations. Companies and invited experts collaborate there to produce specifications of standards, formats, and protocols. Along the way, those companies and individuals contribute their intellectual property to make this happen. Those standards are then implemented by commercial and open source products.
- LDAP is an example of one such standard. It is defined at the IETF. It's original use was for things like phone books and organization charts.
- Java is a programming language, and OpenJDK is the open source implementation of that language. Java contains a component, named JNDI, that implements LDAP as well as a number of other standards such as CORBA and RMI. Those other standards provide the ability to do things like remote code execution which was popular in the 1990s, but now isn't quite so important and may open the door to security issues.
- The Apache Software Foundation is an example of an open source organization. Self managed groups of experts (e.g. academics, people seconded by their company or otherwise voluntary) collaborate there to produce releases of software components and products. In this way the industry as a whole (e.g. companies, academia, the public sector and individuals) contribute their intellectual property to make this happen.
- Log4j is one example of a software component produced by the ASF. It does the mundane job of producing an audit trail of important activities. Augmenting the audit trail with organization data from LDAP seemed like a good idea at the time. As Log4j is written in Java, it made use of JNDI to access that data.
- Put together, and you have a mundane and almost invisible function which can be tricked into doing remote code execution. There have been prior efforts to identify critical functions, yet somehow Log4j never ranked high on any list. Furthermore, this vulnerability was not a bug (like a buffer overflow), but a combination of intentionally designed features when put together can be exploited. This went unnoticed for many years. A mundane and nearly invisible function that can be exploited is in no way unique to open source.
- Since the vulnerability was disclosed, we had a number of inquiries from people trying to determine out if they were affected by this vulnerability. It turns out many were running Log4j version 1, which was discontinued in 2015. The good news is that this particular vulnerability does not affect them. The bad news is that the code they are running has not benefited from any security fixes in many years.
- The handling of the issues in Log4j followed ASF processes and the fixes were timely.
...