Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-6047

Branch

MASTER

Introduction

Currently VR has an scalability issue during the large deployment. Everytime when VR need to be re-create or reboot, CloudStack would send lots of programming commands to it. VR would treat them as individual commands then execute them. In large deployment, it would take tens of minutes or even hours to complete all the necessary updates, like setup DHCP and programming firewall.

For example, in the past, everytime we setup DHCP in VR, we need to restart dnsmasq service for every programming, which is very time consuming. Though we've introduced a way to reload without restart dnsmasq, but the same issue existed with apache2 and other services as well. And every SSH to VR would also time consuming.

The new approach of reprogramming VR, would help greatly on this issue, and hopefully great reduce the VR programming time. It would introduce a mechanism to "aggregate" the commands to be executed, and do it in one batch inside VR. And restart the related services(if necesary) only after the whole batch is completed. The configuration would be transfer to VR in one piece as well, eliminate any unnecessary ssh.

We would expect in such scenario, most configuration would only be text update and involve no more time consuming operations. We would leave every possible time consuming operation to the end of execution of aggregated commands.

 

Document History

Feature Specifications

  • The reboot and re-create of Virtual Router would be much faster than before after the feature completed, especially in large scale deployment
  • The related simulator and performance measure method would be developed as well.
  • User would see no UI change in experience.

Architecture and Design description

  • The first thing first, make all hypervisors use the same resource base for Virtual Router.
  • Introduce a new interface to network elements, named "AggregatedCommandExecutor"
    1. If the network element is an "AggregatedCommandExecutor", "prepareAggregatedExecution(Network)" would be called when developer want all the following commands to be aggregated executed.
    2. After the aggregated period past, "completeAggregatedExecution(Network)" would be called, then network element would start to execute the all commands in it's queue
      1. The queue would be formed into one command in VR's case, then give to VirtualRoutingResource for execution.
    3. prepareAggregatedExecution and completeAggregatedExecution() must be called in pair.
      1. Another function abortAggregatedExecution(Network) should be called if completeAggregatedExecution() cannot be called(e.g. some commands failed to generated during the process), to clean up the queue and fail the task.
      2. Call prepareAggregatedExecution() alone would result in all the following commands failed to executed and blocked forever.
    4. For the current VR, since network orchestrator would program the firewall and ip rules, VR itself would program dhcp etc rules.
  • Virtual Router implemenation:
    1. All the following executed commands would be put in a queue(per VR) in the mgmt server, waiting to be executed
      1. Currently we would still return true for each commands for backward compatibility
      2. But the result of "aggregated command" is the real result.
    2. The aggregated commands would form a new command
    3. VirtualRouting would execute the aggregated command in following sequence:
      1. Get all the preparation done.
        1. e.g. Plug in nics or find nic no, dependence on the hypervisors
      2. Generate a configuration file represent all the commands in the aggregated command
        1. Some commands are a little different, e.g. configure load balancer, which would require copy files to VR, which may need extra process since the file would be overrided in the VR by other configure load balancer command later.
      3. Send the configuration to VR to execute
      4. Get the clean up done.
    4. Two new network element command would be introduced to indicate the start and end of aggregated commands: AggregationControlCommand with Start or Finish as parameter.
      1. There are two path to program VR, one is in Network Orchestrator, another is in VR's start up. The former one would send commands one by one to VR, and the later one would send a commandset to VR. The solution must able to deal the both situation
      2. In VR, prepareAggregatedExecution() would send AggregationControlCommand with Start action to VR, and VR resource would prepare the queue and delay the following command's execution, till AggregationControlCommand with Finish action.
        1. All the commands result would be returned as result of AggregationControlCommand with Finish action.
      3. In VR booting up, these commands would be insert to the start and end of commands sent to VR. 
      4. Failure of AggregationControlCommand should always result in whole operation failure(e.g. restart network, reboot router).
    5. All kinds of VR state query commands(command return a state rather than true/false) won't be supported in aggregated mode
      1. GetDomRVersion, CheckRouter, CheckS2SVPN, NetUsage
      2. Would return failure immediately for the result, because caller would demand the result immediately rather than waiting to see if whole command set success or fail
      3. So, only support VR reboot/re-create which can be either success or fail.
      4. AggregationControlCommand with Start action would be insert after GetDomRVersion command executed when VR is booting up.
  • Mind the gap!
    • Order is extremely important!
      • Every commands called in original order should happen in the exactly same order in aggregated execution.
        • We don't want to program firewall before ipassoc.
      • No other command should be executed before VR complete the aggregated execution
        • It can be guaranteed if the aggregated command has been executed earlier in VR, because it would hold the VR lock inside.
    • Reboot of VR should clean the queue.
    • If the failure happened in the execution process, it should provide enough debug information
      • e.g. which command failed with what's parameter, and what's the error message.
    • No other concurrent operation should happen to the same VR
    • Only support two state of execution result: Success, or Fail.
      • The result and debug info can be get, but partial success won't be supported.

Web Services APIs

No API

UI flow

  • No UI

Appendix

Appendix A:

Appendix B: 

  • No labels