Work in progress

This page is work in progress

Java Broker Logging

We plan to reimplement logging within the Java Broker, bringing logging configuration into the Configured Object Model

Requirements

  1. Ability to produce a broker wide log which may or may not include the activity of virtual hosts
  2. Ability to produce separate logs for each virtualhost
  3. Ability to assign/change the level of any logger at runtime.  It will be possible to assign the logging level on a logger related to a class e.g. o.a.q.X or hierarchy e.g. o.a.q.
  4. Logger levels can be assigned durably or non-durable.  Those durable will survive a restart.
  5. Ability to assign different logger levels for different virtual hosts.
  6. Ability to logs to different destinations, e.g.
    1. to a plain file on the filesystem
    2. to a stream
    3. to a bounded buffer
  7. Logging configuration will be persisted to the Broker/Virtualhost configuration stores.  As such in the HA use-case, the same logging configuration will be applied to all nodes within a HA group.
  8. Ability to configure file rolling and compression
  9. Ability to configure logging and view logs from Web Management Console, REST API and later AMQP management.
  10. Ability to limit who may configure logging or who may view logs via the Management interfaces.

Open Questions

 

  1. Do we want the ability to view a log file that is on disk from Management, or is the ability to view the log messages within the bounded buffer recorder sufficient?  Assuming the answer is yes, and we need to retain the ability to stream the contents of an existing log from disk in a secure manner.
  2. As logging configuration is to be unified into the store and substantial code is required to open/upgrade/recover the store, there is a possibility that an unexpected problem may occur before logging is configure.   It would seem sensible to have a emergency mode (command line switch) where we fall back on a simpler logging configuration that can be configured early.
  3. Related to 2), in normal operation logging won't be ready until after the store is recovered.   Do we want to use a temporary appender to cache the logging the events then replay them once logging is ready???

High Level Design

The Broker and Virtualhost will have one or more LogRecorders and zero or more LogFilters.

  • LogRecorders are responsible for writing the log itself. There will be different implementations of LogRecorder that will be responsible for writing to files, streams and maintain a bounded buffer of log messages for inspection on-line.
  • LogFilters determine if a particular log event is to appear in the log.

Proposed Configured Object Model

ConfigModelShowingProposedLoggingObjects

Class Diagram

 

Broker-Virtualhost

LogRecorders

LogRecorders are responsible for writing the log itself. We will have different implementations of LogRecorder.

  • RollingFileRecorder - a log recorder with the ability to log to a file and roll the file according to a time and/or size schedule.  The rolled files may be compressed.  The maximum number of rolled files retained is configurable.
  • StreamRecorder - a log record that writes to standard out or standard error.
  • BoundedCachedRecorder - a log the logs to an internal bounded cache.

Internally, each LogRecorders will create a LogBack Appender.  On receipt of each logging event, the LogRecorder implementation must decide if the log event is to be logged by the recorder.  It makes this determination in two ways:

  1. It casts the LogRecorder's parent to ConfiguedObjectLogTarget and invokes #shouldLog().  If this method returns true if continues to the next stage.
  2. It now looks for a nearest LogFilter matching the log event.  It does this by first looking for a LogFilter matching the log event's logger, and then if not successful it pops off hierarchy portions from the RHS of the logger name until a matching filter is found.  Once a filter is found it compares the logging level of the event with the logging level if the filter.  If no filter was found, the log event is compared with the rootLevel of the LogRecorder itself.

LogRecorder implementations

RollingFileRecorder

Internally, uses a LogBack RollingFileAppender.  We we leverage directly its features for time and/or size based rolling, compression and history retention.

To facilitate access to the log files through web management, it exposes a derived attribute fileList which is a list of the current file and any rolled files currently on the filesystem.  Access to the file itself will be delivery via the LogFileServlet (non-REST) 

StreamRecorder

Internally, uses a LogBack ConsoleAppender.

(Might we use an instance of the class to produce the startup messages to standard out?). 

BoundedCachedRecorder

Internally implements its own Appender that will append events to a circular buffer.

To facilitate access to the cached log, Range headers will be used and a multi-part response.   Not quite sure how this will work yet.

Other implementations

This approach is extensible.  It would be simple to provide implementations against for Syslog, MailAppenders etc.  We might also define test recorders for use during system testing.

Broker/Virtualhost implementation

The Broker/Virtualhost implementation classes must implement ConfiguredObjectLogTarget.

For BrokerAdapter#shouldLog() will consider the combinedLog attribute.   If this set true, the method must return true for all events so that the log contains log events for the entire Broker.  If combinedLog is set false, the shouldLog method must only return if the running principal does not have a virtual host.

For AbstractVirtualHost, shouldLog() will consider the principal on the thread performing the logging.  If the principal's virtual host matches the LogRecorder's parent, it will return true, or false otherwise.

LogFilter

Associates a logger with a logging level.

Default Configuration

The Broker will be shipped with a RollingFileRecorder and BoundedCacheRecorder configured at the Broker.  There will be no Virtualhost level LogRecorders.  The Broker log will include all logging including those for virtual hosts.

The RollingFileRecorder will be configured to suit a typical production environment:

  • Daily rotation with automatic compression of rolled files.
  • Rolled file names include a ISO 8601 date stamp
  • 28 days history
  • Root level FATAL

The BoundedCacheRecorder

  • 1MB
  • Root level FATAL

The LogFilters will be

  • org.apache.qpid => WARN
  • qpid.message => INFO
  • qpid.message.subscription.state => OFF
  • No labels