v0.9.4 Node Classes

  • FlumeNode
    • FlumeVMInfo
    • SystemInfo
    • LivenessManager (issues heartbeats)
    • MasterRPC (interface for all client to master rpcs)
    • LogicalNodeManager (manages logical nodes lifecycle)
    • ReportManager (gets node status/metrics reports)
    • MasterReportPusher (sends node status/metrics reports to master)
    • field walMans (mapping from logical node name to WAL Manager instance)
    • field dfoMans (mapping from logical node name to DFO manager instance)
    • ChokeManager (manages experimental throttling mechanism)
  • LivenessManager
    • MasterRPC (reference to FlumeNode's instance)
    • LogicalNodeManager (reference to FlumeNode's instance)
    • HeartbeatThread
    • CheckConfigThread
    • WALAckManager
    • WALCompletionNotifier
    • field fcdQ (flume config data queue)

How a heartbeat works

  1. LivenessManager instantiated
  2. LivenessManager starts, starting CheckConfigThread and HeartbeatThread
  3. HeartbeatThread periodically perform heartbeat checks asking the master:
    • Does this physical node have all logical nodes instantiated?
      • If not instantiated, get configuration and spawn it. (possible issue, same thread)
      • If instantiated, skip
      • If node exists but not on master, decommission it.
    • Does the current logical nodes have their configs up-to-date?
      • If so, skip.
      • If not, queue up heartbeat info so CheckConfigThread can attempt to update atomically.
    • Are there any new e2e ack groups the node should check for?
      • Ask master if the acks that the WALManager have registered are safe.
      • Remove ack from pending queue and signal WALManagers that ackgroup is safe
    • Are there any old e2e ack groups that the node should resend?
      • Pending ack groups have a timestamp. If retransmit time has elapsed, signal to WALManagers that ackgroup has expired

How a logical node gets spawned

How a logical node gets reconfigured

  • No labels