This document gathers the notes on the original design of the Agent Load Balancing across management servers feature. There might be changes to it that happened after 2.0; please feel free to update the doc if you notice any inconsistencies between it and the current code.
The primary goal for agent LB is to distribute the direct agents (like XenServer and Vmware) across a cluster of management servers. Currently there is a 1-1 mapping between an agent and a resource (like Xenserver or Vmware resource) and any operation to the hypervisor host goes through the corresponding agent. So distributing the agents will lead to hypervisor operations also getting distributed across management servers. Even though the agents are distributed evenly (to the best extent possible), the load will also depend on how the VMs are distributed on the hypervisors.
Note that agent LB has nothing to do with load balancing of API requests. These needs to be handled separately by means of an external load-balancer sitting in front of the MS cluster.
The load calculation for comparing with the load threshold (agent.load.threshold) is done using the following logic.
Load = (Managed routing hosts) / (All routing hosts)
The average host per MS is calculated using the following logic.
Average host count = (Managed routing hosts) / (# of MS in 'Up' state)
Routing host : Host type is set to 'Routing' in host table DB
Managed routing host : Host is owned by a MS (MS id is set in host table in DB)
MS state : Refer to mshost table in DB for the state of each MS
Agent load balancing is not triggered unless global config parameter "agent.lb.enabled" is set to true. It has false value by default.
1) Management server M1 is up and running, management server M2 joins the cluster.
1.1) gets information about all management servers and agents in the system
1.2) calculates average agent load
1.3) For each management server we use AgentLoadBalancerPlanners to find out how many hosts the MS can give away. Currently we have just one planner - ClusterBasedAgentLoadBalancerPlanner - it gives away hosts on per-cluster basis
1.4) after the map "ManagementServer->ListOfHostsToGiveAway" is formed, for each MS in the map M2:
* check if the host being processed by someone. If yes - skip this host from rebalancing.
* create an entry in op_host_transfer table (hostId, currentOwnerId, futureOwnerId, "TransferRequested" state)
* send RebalanceCommand with event "RequestAgentRebalance" to the currentOwner MS.
* if the RequestAgentRebalance fails, the corresponding entry is removed from op_host_entry table and we start processing next entry from "ManagementServer->ListOfHostsToGiveAway" map
* if the RequestAgentRebalance succeds, we start processing next entry from "ManagementServer->ListOfHostsToGiveAway" map
1.5) After the map is processed, StartRebalance is considered to be completed.
* receives RebalanceRequest with event=RequestAgentRebalance from M2.
* does nothing with Agent Attache at this point.
* creates an entry for the host in _agentToTransferIds HashSet.
2) Management server M1 scans agentsToTransfer queue.
There is a getTransferScanTask (runs only when "agent.lb.enabled"=true) which periodically scans the agentToTransferIds. If it finds an entry there it does:
2.1) check if the task is timeout or future owner of the host is no longer active. If one of the conditions is true, the entry is removed from agentToTransferIds and op_host_transfer map. Nothing needs to be done with the agent
as actual agent attache wasn't changed yet. If none of the conditions is true, proceed to the next step
2.2) check listener and request queue for the corresponding attache. If both queues are empty, proceed to the next step. If one of the queues is not empty, skip the agent.
2.3) Updates the state for op_host_transfer entry with the new state "TransferStarted" and start actual agent transfer.
3) Actual agent transfer
3.1) Puts agent attache to run in Transfer mode. Transfer mode means that we collect all the incoming requests into Requests queue without processing them.
3.2) Update host with the status "Rebalancing"; update op_host_transfer record with status "TransferStarted"
3.3) Send command to the M2
3.4) Connects agent and sends the result of this command to the M1
3.5) Get result from 3.4, if its:
* false - remove the entry from op_host_transfer map and disconnect the agent.
* true - go to the next step
3.6) Get all the requests from the transfer attache.
3.7) Create forward attache
3.8) RouteToPeer all the requests
3.9) Finish rebalance by deleting the entry in op_host_transfer map
agent.lb.enabled - Enable agent load balancing between management server nodes. Defaulted to false, i.e. agent LB is disabled
agent.load.threshold - Percentage of the agents that can be held by one management server before load balancing happens. Defaulted to 70%
A new table op_host_transfer is added to keep track of agent transfers
Applicable for all directly managed agents like XenServer, Vmware, Simulator