With the current architecture, FOP uses quite some memory in certain situations, such as larger fo:page-sequences. A number of users have expressed the desire to see this changed.
XSL-FO compliance & features
table-layout="auto". It's the only feature I am missing, and makes me looking to commercial alternatives.
fo:footnotes in fo:tables and fo:list-blocks are not implemented yet. There is already a patch available here, that should be reviewed and preferably applied ASAP. (Patch was applied to FOP Trunk; for more info, see the Bugzilla report)
fo:floats are currently also unimplemented. Code branch has already been created during Google Summer of Code 2006, so not all is quiet on that front.
support for integer values for keep-together, keep-with-next and keep-with-previous (minimal support available; not a lot of user-feedback yet (no news = good news?))
the table-continuation label extension has so far not been ported to 0.9x, which still keeps certain users from upgrading (the extension will probably never be ported; fo:retrieve-table-marker should cover this requirement, and more)