2018-09-06 Meeting Notes

Agenda:

Attendees:

JV, Enrico, Sijie, Yiming

Minutes:

JV: we have some a leak on on the reader

from Slack:

we ran into fullGC and zktimeouts over a long run and the GC shows like this359752 3: 1650087 79204176 java.util.HashMap
359753 4: 718190 74691760 org.apache.bookkeeper.client.PendingReadOp
359754 5: 823944 59323968 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
Pointing to PendingReadOp
protected void cancelSpeculativeTask(boolean mayInterruptIfRunning) {
if (speculativeTask != null) {
speculativeTask.cancel(mayInterruptIfRunning);
speculativeTask = null;
}
}
But we never assign anything to speculativeTask
so we never cancel them
That appears to be one issue for sure
do you guys see any other issues? or have run into similar problem?

Yiming reports that at Twitter they saw something similar, but it was due to the bookie which was throttling the clients, which in turn they were queuing up the requests

Sijie: The problem should be around the fact we are not cancelling the  PendingReadOp

JV has a quick fix

Sijie: about the 128bit ledger id. switching to 128bit should not have impact on performances. It will need some engineering work for the implementation.

JV:  Sijie's proposal is fine, but how can we start wokring on this, it is a bug change

Sijie: We can split the work in smaller tasks, it is important to stay always backward compatible.

Enrico: we have to decide a name for 'scope' soon

JV and Sijie: oky for "scope", it sounds like a "workaround", but it is fine, let's start this thing soon.