This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.
Here is a patch against the CS Master branch as of 2014/04/12...
This implementation allows OpenStack Swift to be integrated directly into the CloudStack UI as an Object Storage Service to be used by the CloudStack end users.
Everything in this document reflects functionality which has already been developed unless it is marked with 'TODO'. If you have comments or questions please let me know and I will update this document.
This document lays out the integration of the OpenStack Swift Object Store as an object store service made available through the CloudStack UI. This integration allows CloudStack users to authenticate against Swift using their CloudStack credentials as well as exposing a UI in CloudStack for users to manage their documents in Swift.
A key enabling component for this integration is a Swift Auth middleware that I wrote called 'mauth'. This middleware supports CloudStack by default, but it is also extensible and open source. The middlware implementation uses the CloudStack username and api key as the Swift user credentials. After authentication, Swift will return a token to be used for the subsequent API calls.
To get thing started I will show a curl example using mauth to authenticate a CloudStack user.
In this example, http://127.0.0.1:8080/v1.0 is pointing at the Swift cluster.
The $username and $apikey reflect the CloudStack user and his api key respectively.
Request for authentication curl -v -H "X-Auth-User: $username" -H "X-Auth-Key: $apikey" http://127.0.0.1:8080/v1.0 returns: $auth_token and $swift_storage_url Request container list curl -v -X GET -H "X-Auth-Token: $auth_token" $swift_storage_url
In this example, the user requests authentication and they are given back a token and their storage url for where their objects are stored. They can then use that token and url for subsequent requests.
A UI for Object Store is added to CloudStack in the left hand services menu. When a user clicks on the Object Store menu option, they are authenticated with the Swift backend (assuming they have api keys generated, if they don't, it will notify them that they need to generate an api key to use the service). Once authenticated, the user's containers and objects are listed in the main area using the treeView mechanism. Selecting a container or object gives details about it as well as shows actions that can be taken on it.
There are a few challenges with this integration. I will give a basic overview of the components in play and touch on some of the reasons for the decisions, but more detail will be added later in the document.
Because the object storage data is often large and very network intensive, we do not want the data to actually flow through the CloudStack box as it would introduce a huge bottleneck. Because of this, there is no Java component of this implementation in CloudStack. Instead, the traffic is routed to Swift or CloudStack, based on the url structure, by a load balancer which sits in front of the two services. The technology used for load balancing (and ssl termination) is not important, but to illustrate the concept, I used HAproxy and Pound for these tasks.
CS - CloudStack
Swift - OpenStack Swift Object Storage
For now, refer to the Feature Specifications or the UI Flow sections.
Most of this has already been covered. I will add to this section as questions arise or if there is a request for more detailed descriptions of the different pieces.
No changes to the CloudStack APIs
I will continue to add screenshots to this section as I have time...
(Old Screenshot: The Object Store menu item is actually added as the last item in the services menu, it is not the only item in the menu)
(Note: URL is empty because MyContainer is private in this case)
(options are: `delete`, `add folder`, `upload object` and `make container (and all its contents) public`)
(options are: `delete`, `add sub-folder`, `upload object`)
NOTE: Undocumented Step - Before this last screenshot the `make container public` action was applied to show the public URL.
Note: If you have followed this implementation at all (from previous conferences), you may have heard of 'cs_auth' being used as the Swift auth middleware. 'cs_auth' was the first Swift auth middleware I wrote to authenticate CS users in Swift and was the predecessor of 'mauth'. 'mauth' will continue to be supported, but 'cs_auth' will not and should not be used for future projects.
Note: Regarding Swift Usage Reporting
The Swift implementation is made up of two parts, the auth middleware in Swift and the AJAX CloudStack UI in the users browser. Since we are not integrating with the Java backend of CS at all, I have not implemented any Swift usage reporting in CS.
However, I have developed a middleware for Swift called 'swift_usage' (git repo) that exposes the Swift usage for accounts as a REST API. This was developed to be used with my auth middlewares, so that has been tested. I have not touched this project in quite a while and it was only used by a small group of us, so I need to do a once over on the project to make it 'open source' ready. I need to do a bit of a code cleanup and provide much better documentation, but the core is a working product already. We will be using this project internally, so expect some updates soon.
'swift_usage' depends on the 'slogging' Swift middleware. 'swift_logging' just exposes the usage data collected by 'slogging' as a REST API. 'slogging' is not an entirely obvious piece of software to work with and it creates many different logs. Here is a summary of some of my notes on 'slogging'.