Ovm3 implementation for CloudStack:

Introduction

Initially the implementation began for a specific customer project and was hurried the first ovmsupport implementation, of ovm3, was undoubtedly a quite horrendous one. After the pressure for this specific use case dropped, getting away from Oracle licensing on hypervisors, the implementation became a side project from my daily and nightly work. This stretched over quite a period of time, and included a full refactor of the integration layer and some effort put in unit testing. The way the implementation is driven is purely by the use case and design of the cloud that I built with some of my close friends at SBP, meaning that some features may be lacking as the design and implementation follow the philosophy that a hypervisor should be as dumb as a brick, the orchestrator is omnipotent and that we use a specific set of features that are proven to work in that cloud infrastructure (tm).

Testing was done in a KVM environment with OVM3(xen), XenServer and sometimes KVM running side by side on top of KVM and doing full rollout integration testing of the features that were implemented. By doing so being able to compare if a feature was not broken all over the board or if it was just my implementation that was rotten after merging code.

Using it

A prerequisites is that there are  a bunch of OVM3 hypervisors in place, obviously, and making sure that some of the configuration is done at a bare level. The implementation takes care of the NONZEROCONF option for networking, but if not explicitly set on OVM3.3.x it will make the hypervisor, when it is virtualised, choke fall over and die slowly upon deployment of a SystemVM. Due to the patching that is done by the agent plugin triggered by Cloudstack this won't happen a second time, unless you've set the option yourself as it will just not happen. This is not a problem for OVM3.2.x but it's better to disable it there too nonetheless.

Base network configuration should be taken care of. The setup has been tested with two bridges on a bond, where one takes care of the isolated traffic and the other takes care of management, storage and public traffic in respective VLANs. An interface can be used as an Ovm3 traffic label and will mean that that specified traffic will end up on that specific Nic/Bond/Bridge/Vlan/etc that is used in the traffic label. The control0 interface is auto populated when the machine gets added to a cluster in Cloudstack, and will be checked every time a systemVM that requires it is deployed. The config file for control0 is discarded after it's deployed as upon reboot the control0 interface will not come up correctly and will break internal communication, hence it is repopulated when required. From a network perspective only VLANs are supported for now, with the linux bridge, and not with Openvswitch yet, as the Ovm3.2 and Ovm3.3 provided Openvswitch is rather old. The VLANs will be populated on the bridge with that resembles the Ovm3 traffic label that is used on the public interface in the zone.

Initially there was no SystemVM template for raw images, support is in the SystemVM generation scripts now, so a conversion can be done of an existing image and prepopulating that either via the copy template scripts or putting it somewhere on a web server, adapting the URL in the database and allowing Cloudstack to download it. (conversion is easy and requires qemu-img: qemu-img convert -f vpc -O raw systemvm64template-master-xen.vhd systemvm64template-master-ovm.raw".

Integration

The implementation is a native implementation which uses the OVM3.2.x/3.3.x agent. Most of the work has been done based on how Oracle Virtual Manager interacts with the agent. Initial setup is done by introducing a Cloudstack module into the agent that exposes functionality that's missing from the agent framework Oracle provides. A goal of the implementation was to allow for the original OVM(2) implementation to co-exist with the new integration. The communication with OVM3 itself from a Cloud Management perspective is drawn out in the diagram below:

In true effect the integration for the API has been split in four layers. Initially the base API itself which is a literal reflection of the agent API, and actually mirrors that in the Objects directory of the plugin, the only exception here being the Ovm3ResourceException handler, the Connection handler and the RpcTypeFactory, the last we need for converting some python RPC XML bits to the wonderful world of Java. The second layer is the main integration with Cloudstack extending on and implementing the base requirements for Cloudstack, things like the HypervisorGuru, Listener, VirtualRouterDeployer, StorageProcessor, FenceBuilder, ResourceStateAdapter, Discoverer, etc are actually implemented here where the main importance is to just override the methods of the classes it's extending, and to keep code that is not overridden to a a bare minimum. The third layer are the resource helpers classes, named after the component they functionality "help" to expose to be used by the second layer. This was done to make sure we don't clutter the second layer up too much by inserting a lot of code there in a feeble attempt to keep it understandable and debuggable. The fourth and final layer are two python plugins which are placed on the hypervisor, of which one is inserted into the agent to natively expose missing calls combined with some things that come in "handy". The second python script is used for self fencing of primary NFS storage, and is actually portable to XenServer and might be adapted to also support KVM at some point.

The actual code to figure out what the api exposes is a bit of python and was also used to figure out the difference between 3.2.x and 3.3.x. Besides this an installation of Oracle Virtual Machine Manager was used combined with socat to MITM the SSL connectivity and figure out how the functionality exposed by the agent was actually leveraged.

gadget.py
from agent.target.api import MODULES

from agent.api.base import Agent
from agent.lib.settings import get_api_version
from agent.utils.xmlrpclib import ServerProxy

class Gadget(Agent):
    def get_services(self, version=None):
        return {
             'get_api_services': get_api_services}

def get_api_services(version=None):
    import inspect
    services = {}
    print("[")
    for mod in MODULES:
        services = mod().get_services()
        for name, func in services.items():
            print("    /*")
            print("     * %s, %s" % (name, mod))
            info = inspect.getargspec(func)
            for idx,arg in enumerate(info[0]):
                dev = None
                try:
                   dev=info[4][idx]
                except:
                   pass

                print("        argument: %s - default: %s" % (arg, dev))
            print("     */\n")
    print("]")
    return services

if __name__ == "__main__":
    get_api_services()

 

Functional

Things that are known to work and have been tested to fulfil the base use case of running a zone with a cluster, in no specific order.

  • Basic networking.

  • Advanced networking.
  • primary and secondary storage via NFS.
  • RouterVM, SSVM and Console Proxy work.
  • PV guests (HVM not tested).
  • Attaching/detaching of disks and ISOs, live
  • Attaching/detaching of NICs, live.
  • Pool of 4 hypervisors.
  • VM and SystemVM Migration (tested with pool only).
  • Host maintenance.
  • Hypervisor failure.
  • Fencing.
  • VM Live Migration from OVM3.2.x to OVM3.3.x.
  • Console proxy

TODO or not todo and limitations

In no specific order things that are on the shortlist to be implemented at some point, require fixing, are not there, have not been tested or will never be done.

  • Snapshots (can't be done live with RAW filesystems, unless ocfs2, replace qemu-img or do it from underlying storage shrug).
  • Native OVM Clustering (seems rather "fragile" due to ocfs2 img used and locking of it over the cluster which can render a complete cluster unusable, having to three finger salute the whole bunch or forcibly remove the lock...)
  • OCFS2, clustering and the FS for hot snapshots.
  • iSCSI and local storage.
  • Openvswitch, which is bundled with OVM3.2.x./ OVM3.3.x (ridiculously old version though)
  • Meta data on the NICs with regard to NIC roles. (seems not required yet).
  • Cpu pinning (although making sure the service offering is correct without over provisioning, this has the same effect licensing wise conceptually. Nice side note is that the agent doesn't support pinning either).
  • Xentools version detection.
  • A sane master UUID, it's a fixed UUID now.
  • Dynamic scaling of VM (reconfigure can be used).
  • The SSL option for communication (is there but not tested).
  • VLAN/VM state keeping for cleaning VLANs (VLANBridges are auto created but not auto cleaned ATM).
  • The event/statistics subsystem from OVM3 is not used, but tricked rather.
  • Assemblies.
  • ISO boot.
  • IPv6.
  • Removing a host from Cloudstack does not clean it.
  • Loose hypervisors.
  • NVP/NSX and Openvswitch.

Bugs:

Some.

 

  • No labels