Purpose

In 4.2 release, we have added support for object storage like S3/Swift as secondary storage, but missed a systematic way for current customers who are using NFS secondary storage to migrate them to S3 without introducing extra downtime. In this proposal, we are implementing a way for users to achieve that goal.

References

CLOUDSTACK-4024

Feature Specifications

Requirements

Enable CloudStack to allow the addition of an object store with an existing NFS secondary storage
a.      All new resources (snapshots, templates, volumes) created will go directly into object store.
b.      Only reads and deletes will be allowed on old NFS.
c.      ExtractTemplate, CopyTemplate, CreateVolumeFromSnapshot and ExtractVolume will result in a copy of the resource to the object store.  This is to ensure that we maintain the requirement that the object store is region-wide and not zone-wide as the NFS limitation.

Architecture and Design description

New API Provided

We will introduce a new API to add a S3/Swift object store after automatically converting existing NFS secondary storage  to NFS staging store for each zone. This API is not exposed in CloudStack UI, and should be invoked outside by admin when he/she wants to do such migration.

updateCloudToUseObjectStore: This api will convert existing NFS secondary storage to NFS staging store for each zone. Internally, it will update DB entries in template_store_ref, volume_store_ref and snapshot_store_ref to indicate that those objects are now stored in Staging store instead of secondary storage. After that, it will add S3/Swift object store as the new secondary storage. This API takes almost the same input parameters as current addImageStore.

1. Parameter: name – New object store name.
2. Parameter: url – New object store URL.
3. Parameter: providerName – New object store provider name
4. Parameter: details – New object store details
5. Response: ImageStoreResponse.

Migration Steps

Here are steps we recommended to migrate NFS to object store:

1. Invoke updateCloudToUseObjectStore api to convert existing NFS secondary storages to NFS staging stores for each zone and add your S3 store as a new secondary storage. From UI, you will see that all previous NFS stores are shown in the list of Secondary Staging store, only the new S3 object store is shown in list of Secondary storage. During this step, internally, we will fake entries in template_store_ref, volume_store_ref and snapshot_store_ref to indicate that templates, volumes and snapshots previously presented in NFS secondary storage (now NFS staging store) are also presented in new S3 store (even though physically they are not there). This way, from UI, all previous templates, volumes and snapshots are still available to the users from UI so that they still can read, delete them on the old NFS.  We also automatically increment reference count of these old templates, volumes and snapshots on old NFS so that they cannot be recycled during our scheduled cache store cleanup thread. Their reference count will be decremented when we really migrate the content to S3.
2. For each new snapshot taken after migration, we will always take a full snapshot to newly added S3, so there will no issues to coalesce delta snapshots cross NFS and S3 stores. For old snapshots taken before migration, they will be pushed to S3 store when user is trying to use that snapshot by invoking createVolumeCmd by passing snapshot id.
3. For template, users should still be able to deploy VM from them since they are already in NFS staging store. We will trigger copy of the template to the new S3 object store when user invokes ExtractTemplate or CopyTemplate.
4. For volume, we will trigger copy of the volume to the new S3 object store when user invokes ExtractVolume.

Potential Script to Obsolete NFS Secondary Storage

In the future, customer may want to get rid of existing NFS secondary storage completely. At that point, we need to provide a script to migrate all those remaining items on old NFS to S3, and then completely shut down the old NFS. This script is out of scope for 4.3.