Apache CloudStack Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Agents+Framework
Dev ML discussion: http://markmail.org/message/mrmzywynecituxcz
The CloudStack SystemVM template (presently based on Debian Linux) is a lightweight VM appliance that is used to build various System VMs. For example, the Virtual Router (VR, responsible for networking and network services such as DHCP, DNS, VPN, SG, Firewall etc), Console Proxy VM (CPVM, responsible for console viewing) and Secondary Storage VM (SSVM, responsible for storage related tasks and operations) are all based on the same SystemVM appliance template and CloudStack patches this template to create one of the SSVM, CPVM or VR VMs. Various combinations of networking components and services such as User Data, DNS, DHCP, Firewall and VPN can be customized as a network offering and be used to create customized shared and isolated networks. Irrespective of the networking model used, VRs are always accessible by user VMs and vice-versa for a given network a user VM is on. CloudStack has few user VM related features such as reset user VM password and reset SSH public key that are served by the VR among other services such as DNS and DHCP. In recent times, several security issues have been discovered that affect all the System VMs including VRs such as POODLE, GHOST and several openssl related vulnerabilities. This has historically required CloudStack sysadmins to deploy and use updated systemvm templates while the general fix for most issues was to upgrade system packages such as openssl, apache etc.
CloudStack provides no general-purpose framework that can be used to manage, monitor and maintain VRs and other systemvms, improve user VM related features such as password server (for example to server passwords over SSL/TLS and support IPv6 clients), and implement new features such as integration of monitoring and backup solutions for user VMs. The agents’ framework is a proposal to design and implement a general-purpose framework that aims to solve the above-mentioned issues.
Design and implement a general-purpose agents framework that;
The above diagram is a bird’s eye view of the overall architecture of the “agents framework” that shows end-to-end integration, communication and orchestration of services/solutions/tools. The agents framework consists of four main components: (1) a bootstrapping agent that runs in a user VMs and securely communicates with a (2) provisioning server that runs in a VR, the provisioning server gets recipes and commands from a (3) agents orchestration manager in management server’s resource layer, each integration or feature such as reset password will have it’s own (4) API or backend plugin that gets triggered by a corresponding CloudStack UI component or plugin or via a CLI such as CloudMonkey.
To provide a better user experience via CloudStack’s UI, we can enhance the existing CloudStack UI plugin framework by making suitable changes in the CloudStack UI code so that custom UI plugins can be written that can add buttons on a resource (for example, reset password button on a VM page) or add tabs with custom content in resource wizards (for example, selection options of monitoring or backup solution when deploying a VM using the deploy VM wizard). The UI components can also trigger notifications from CloudStack regarding VR health and status of the operation/solution being deployed to the user VM (such as success/failure of resetting user VM password, or setting up monitoring tool in user VM), and such events can also be published on CloudStack event bus (for example, on RabbitMQ). A backend plugin can provide high level provisioning APIs (to both CloudStack UI and CLIs via APIs) and this way we can have feature specific implementation decoupled from core CloudStack codebase. The plugin would provide an asynchronous push mechanism to provision software tools/services/solutions inside user VMs. The provisioning server itself could be based on something lightweight such as Ansible, Chef, Puppet or something that we customize or write from scratch. We propose to write a lightweight-bootstrapping agent that is cross-platform (works on Linux, Windows etc) and standalone (has no external library dependency) binary (both 32 and 64 bit) that is pre-seeded on user VM templates. This approach is similar how reset password feature works as it too needs a reset password script to be installed on a user VM template. A general bootstrapping agent will ensures that we have a single agent that works with any service whether it’s reset password feature or something else; and when a user VM starts with such a bootstrapping agent we’ll have end-to-end communication - from the CloudStack UI to the CloudStack backend plugin (using its asynchronous/non-blocking APIs) to CloudStack’s resource layer to the agent-provisioning server running in the Virtual Router to finally the bootstrapping agent that is running in the user VM. The bootstrapping-agent has the role to get commands, recipes or configuration rules from the agent-provisioning server and execute them.
In case a user uploads a custom template and starts VM, it will have no bootstrapping agent. In this case, the backend plugin retries provisioning couple of times (retries can be configured by a globally configuration) and when it gives up it can alert or email the user to take suitable steps to setup a bootstrapping-agent on the VM. An alternative approach to failure handling could be to attach an ISO to the user VM containing the bootstrapping agent that allows users to easily install the agent.
To provision software into user VMs, the bootstrapping-agent needs to run in a privileged mode. For example, if a tool/service/solution needs to be installed or a password needs to be reset, the bootstrapping-agent needs to run as root/admin but if root/admin level access is not required for provisioning operations we would not want to run it in privileged mode. This would be configurable when pre-seeding the agent on a VM template, or when doing manual installs or patching using an agent installer ISO. We aim to implement a bootstrapping agent library that can be used to implement custom bootstrapping agents (for example, a bootstrapping agent which restricts what kinds of receipes/operations can run on a user VM, or another bootstrapping agent that allows for any recipes/commands like a Puppet/Chef client). Implementation will aim to use SSL/TLS whenever communicating with non-local services, for example between VR and user VMs.