REST API Server Architecture

Problem Statement

Current CloudStack API layer design suffers from various issues:

• not REST compliant, and mixed use standard GET and POST method.
• Resource response object definition is driven by UI rather from end-user perspective, causing performance issues for large scale system.
• Lack of a generic mechanism to develop new APIs, leading to inconsistent implementation in different commands.
• No standard way to perform flexible search against CloudStack entities.
• Tightly coupling ACL checking in our query logic.
  To address those issues, we are proposing to deprecate current api and api server, and redesign a REST compliant api server using JAX-RS(i.e. Apache CXF) or better technologies. This API server should be a separate component from cloud-engine and can be hosted on distributed machines to serve end-user request. Note that this new API server will only handle end-user API. Admin can request information through cloud-engine API.

REST API Server Architecture

The position of new API server in CloudStack can be depicted in the following diagram, and it has such major features:

1. API server will have its own war file to deploy as a separate webapp from cloudstack engine, and can even be deployed to distributed machines to load balance user requests.
2. API server will have its own cache DB to store entity information that we want to present to end-user. Note that this cache DB is totally decoupled from CloudStack DB (which is shared by multiple management servers). Each api server can has its own cacheDB. This cache DB can have totally different schema design to represent a correct end-user targeted response view. The cache DB schema will closely map to our new cloudstack resource model plus row-level permission information that can acquired from pluggable ACL component.
3. With independent cache DB on each API server, API server becomes stateless, shutting down one api server will not impact other api server peers.
4. When API server starts, it will build up the cache DB by fetching fresh data from cloud-engine through REST api, and invoking ACL apis to fill in row-level permission information.
5. API server will subscribe to our event framework to get notified for each resource update to update its cacheDB.
6. During cacheDB buildup, we can also build inverted index using Lucene to support google-like search against it.
7. API server will provide a REST compliant api with the following REST pattern:

Where
<rooturl>: API server web app root url.
<port> : API server web app port.
<entity> : CloudStack entity class, which will be explained below in Object Model section.
<identifier> : Unique identifier for an entity instance, can be uuid, or any other attribute that can uniquely identify a resource.
<opName> : Supported operation name for an entity.

API Server REST vs Cloud-Engine REST

Before discussing CloudStack object model, we would like to make a distinction of two types of REST apis in our system:

• NorthBound API: REST api provided by api servers to end user.
• SouthBound API: REST api provided by cloud-engine for CloudStack internal components communication to avoid tight coupling.

Note that although these two types of REST apis will consume the similar REST URL pattern as documented above, supported <entity> and exposed entity information will be different. For NorthBound APIs, we need to pay attention to how much information we want to expose to end users.

CloudStack Object Model

A pre-requisite to come up with a clean REST style API is to build CloudStack Resource Object Model. This object model should have the following information for each CloudStack resource, including those resources directly managed by CloudStack (like VM, volume, network, etc) and those indirectly related resources (like Domain, Account, etc). Note that NorthBound API and SouthBound API can share the same object model, controlled by different visibility flags.

1. Accessibilty of this resource entity, is it public to the end user or internal to CloudStack internal? For example, PhysicalNetwork should be a resource not public to end user, so API server should not expose a REST url with PhysicalNetwork as the <entity> in above URL pattern.
2. Each resource should be characterized by:
   • Attributes (including its visibility)
   • Relationships (characterizing various different relationships, like association, aggregation, composition, inheritance etc)
   • Supported Methods (including construction, delete, and operations, etc)

With this object model in place, we can have a generic and standard mechanism to

• Construct each Response class, instead of current ad-hoc way based on UI.
• Create cache DB schema or view based on resource response.
• Clearly identify supported methods for each resource.
• Provide a natural map of entity supported methods to REST url pattern. For example, POST create will easily map to the Constructor method of an entity, REST DELETE will map easily to the Delete method of an entity, etc.

Here is a simple Object Model snippet to briefly illustrate the idea: (In the diagram below, entities colored with light blue should serve as our first-level entities that can be exposed to end-user in our REST url pattern, while those entities colored with light brown should be internal to cloud-engine southbound API).

Project Planning

Design and implementing this brand-new API server is not an easy task, and need coordinated work from other component, like cloud-engine, event notification, ACL, UI, internal service refactoring, etc. We need to take a phased approach to reach our final goal. Here are some subtasks I can identified so far and specified them in order here:

1. Define CloudStack Object Model for both API server and cloud-engine API.
2. Implement a new set of Response classes based on object model defined. We may need two set of Response classes, one for API server, one for Cloud-Engine API due to different granularity of information.
3. Create API server Cache DB view schema based on responses to be presented to the user.
4. Cloud-engine should provide REST api for api server to build the cache.
5. ACL component should provide APIs for api server to populate row-level permission information for each resource cached.
6. Cloud-engine should hook up with event framework to publish any resource update change.
7. API server needs to subscribe to event framework to be notified by each resource update change.
8. Implement API server cache build and update algorithm.
9. Building inverted index for API server cache db view using Lucene to support real keyword search.
10. Analyze all current Cmd classes and map to our standarized REST pattern, and implement them using JAX-RS implementation Apache CXF.