*** Unsent and unapproved draft ***
How the ASF keeps our projects secure
(add 3-4 bullet points about our controls and processes here and how they help, lead to these as recommended practices)
About the recent issues in log4j
Before jumping directly into recommendations, it is worth describing the factors contributing to the recent log4j vulnerability. Our recommendations are based on those experiences as well as over 20 years of dealing with vulnerabilities in open source software.
- The IETF, W3C, and WHATWG are examples of standards organizations. Companies and invited experts collaborate there to produce specifications of standards, formats, and protocols. Along the way, those companies and individuals contribute their intellectual property to make this happen. Those standards are then implemented by commercial and open source products.
- LDAP is an example of one such standard. It is defined at the IETF. It's original use was for things like phone books and organization charts.
- Java is a programming language, and OpenJDK is the open source implementation of that language. Java contains a component, named JNDI ,that implements LDAP as well as a number of other standards such as CORBA and RMI. Those other standards provide the ability to do things like remote code execution which was popular in the 1990s, but now isn't quite so important.
- The Apache Software Foundation is an example of an open source organization. Companies and invited experts collaborate there to produce releases of software components and products. Along the way, those companies and individuals contribute their intellectual property to make this happen. Those releases are then incorporated into commercial and other open source products.
- Log4J is one such example of a software component produced by the ASF. It does the mundane job of producing an audit trail of important activities. Augmenting the audit trail with organization data from LDAP seemed like a good idea at the time. As Log4J is written in Java, it made use of JNDI to access that data.
- Put together, and you have a mundane and almost invisible function which can be tricked into doing remote code execution. Despite prior efforts to list critical functions, log4j never ranked high on any list. Furthermore, this vulnerability was not a bug (like a buffer overflow), but a combination of intentionally designed features when put together can be exploited. This went unnoticed for many years.
- Since the vulnerability was disclosed, we had a number of inquiries from people trying to determine out if they were affected by this vulnerability. It turns out many were running log4j version 1, which was discontinued in 2015. The good news is that this particular vulnerability does not affect them. The bad news is that the code they are running has not benefited from any security fixes in many years.
Recommendations
These experiences inform our positions on a number of items.
- We support the notion of identifying critical components. As our software is without cost and can be downloaded by anybody, we have no way of knowing how widely our projects are used. This means that by necessity determining if a component is critical can only either be determined by automated means which is always going to be incomplete and imperfect, or more manually such as by identifying critical products first, and then getting an inventory of what components those products embed. This problem is not unique to the ASF, so it makes sense for there to be industry standards and best practices. We are prepared to participate in these activities.
- We eagerly welcome audits and fixes from any source. We have a process defined for doing so: https://www.apache.org/security/. Our one caution is that we have prior experiences with audits and it is not productive when those audits produce a number of false positives. See https://s.apache.org/fhoji. It is worth noting that many automated means of performing an audit are more suited to detecting bugs inside a single component rather than identifying a vulnerability that is composed of combination of intentional features such as the one we saw with log4j. It is better when these audits are performed by skilled resources who are able to accurately identify actual issues, produce fixes, and therefore reduce the burden on each project.
- It is particularly frustrating to us that we can produces fixes for critical issues in a matter of days or weeks and not to have those fixes be picked up for months or years, despite having a common and industry standard process for disclosing vulnerabilities and fixes. This is the problem that we see as most pressing.