Status

Current state: Done.

Release: 4.5.0

Problem

Using BookKeeper as a low-latency distributed data storage achiving read-your-writes consistency outside the LAC protocol

The primary purpose of BookKeeper is to be used as distributed transaction log, but for its internal design it can be used as a distributed storage of binary data.

The idea is to store binary objects (BLOBS) as entries inside ledgers. In this case there is no need for "tailing reads" or "fencing".

This way of working can be drafted with this every simple scenario:

• Writers create ledgers, writes BLOBS to them and store to an external database the ID of each BLOB a pair (ledgerid, entryid) as BLOB_ID
• Once a write as been acked by the configured quorum of bookies the BLOB can be considered to be stored in a durable storage
• Readers can use the BLOB_IDs to access data reading directly from any of the the bookies which contain a copy of the BLOB

Design

In BookKeeper 4.4.0 the asyncReadEntries method checks that the range of entries to be retrieved is within the safe range of entry ids according to the LAC protocol:

public void asyncReadEntries(long firstEntry, long lastEntry, ReadCallback cb, Object ctx) {
        // Little sanity check
        if (firstEntry < 0 || firstEntry > lastEntry) {
            LOG.error("IncorrectParameterException on ledgerId:{} firstEntry:{} lastEntry:{}",
                    new Object[] { ledgerId, firstEntry, lastEntry });
            cb.readComplete(BKException.Code.IncorrectParameterException, this, null, ctx);
            return;
        }
        if (lastEntry > lastAddConfirmed) {
            LOG.error("ReadException on ledgerId:{} firstEntry:{} lastEntry:{}",
                    new Object[] { ledgerId, firstEntry, lastEntry });
            cb.readComplete(BKException.Code.ReadException, this, null, ctx);
            return;
        }
        asyncReadEntriesInternal(firstEntry, lastEntry, cb, ctx);
    }

The Idea is to add a new BookKeeper client side  method "readUnconfirmedEntries" (and the async counterpart) which does not check for the LastAddConfirmed range

public void asyncReadUnconfirmedEntries(long firstEntry, long lastEntry, ReadCallback cb, Object ctx) {
        // Little sanity check
        if (firstEntry < 0 || firstEntry > lastEntry) {
            LOG.error("IncorrectParameterException on ledgerId:{} firstEntry:{} lastEntry:{}",
                    new Object[] { ledgerId, firstEntry, lastEntry });
            cb.readComplete(BKException.Code.IncorrectParameterException, this, null, ctx);
            return;
        }
        asyncReadEntriesInternal(firstEntry, lastEntry, cb, ctx);
    }

We can say that there is an out-of-band contract between the writer and the readers, we only want to achieve "read-your-writes consistency"

Actions

• Related JIRA:
• Related PR: https://github.com/apache/bookkeeper/pull/121

Rejected Alternatives:

• Add a ClientConfiguration option 'readEntriesAllowReadAfterLastAddConfirmed' to disable the check for the method readEntries.
  • This has been rejected (Sijie Guo) because the usage of a new method will be clearer and because if a client is shared among different reader procedure this option will introduce uncontrolled side-effects

Related works:

• In Yahoo private fork the a similar work has been done in the BookKeeperAdmin class (see https://github.com/yahoo/bookkeeper/commit/92e2f906f83d997b5cf5ffd2bcfcdbe53ca3bd04) (Matteo Merli)
• In Salesforce private fork the 'ExplicitLAC' feature (contributed to 4.5.0 Apache version) is used to decrease overall latency of writes, without violating the LAC protocol (JV Jujjuri)