Storage OverProvisioning as Per Primary Basis

Introduction

Most Storage solutions now come with the capability to over provision the available storage. This allows over-committing the storage and using the space more efficiently.

CloudStack currently is able to leverage this functionality by specifying global parameter storage.overprovisioning.factor which is the admin defined over provision factor for all the primary storages.

Purpose

This functional specification describes the requirements for support of per-primary storage overprovision setting.

Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-6092

Feature Specification

Feature Description

• The feature allows CloudStack admins to define over-provisioning for individual primary data storages. This will eliminate the tight dependency over the global parameter to leverage underlying overprovisioning.
  A typical Cloud deployment will have different combinations of primary storages. They could either be from different vendors or from the same vendor but different products.
• Depending on the storage capability, CloudStack admins will have the flexibility to determine what the over-provisioning factor should be for each primary storage.
• Available storage exposed for the primary data store will be (actualStorageSize) * (storage.overprovisioning.factor for the primary)
• The total summation of capacity of all the primary storages will be displayed on the dashboard with appropriate alerts.
• All operations like attach Volume, copy Template, resize Volume, Migration etc. will be handled to adhere to the overprovisioning of the primary data store involved.
  For Instance : In case of volume migration the current behavior is that CloudStack calculates the totalOverProvCapacity which is (total storage capacity) * (storage.overprovisioning.factor) for the destination Primary Storage.
   As part of validation, the destination Primary Storage must have a higher totalOverProvCapacity than the expected total allocated space.
• Threshold parameters like notificationthreshold and disablethreshold will be used to alert and disable alarming usage of storage.
• Primary stores at Local, Cluster level and Zone Wide Primary Storages will be supported.
• Data Stores of Type NFS and VMFS are only supported to use overprovisioning. Support of other types will be addressed in CLOUDSTACK-5806.
  

  General Workflow:

• admin can update an existing primary store by setting overprovisioning in the per primary setting.
• This value will override the value at the global level. This leverages the granularity of global parameters introduced in 4.2
• To fall back to the global value, null value can be passed.
• To disable overprovision a value of 1 will be passed.

DB Schema changes

Not required.

All existing records of zone level storage.overprovisioning.factor settings will be migrated from data_center_details to storage_pool_details

Web Services APIs

listStoragePool will have overprovision factor also listed.

#list storagepools

Logging and debugging

.All operations like setting and resetting storage overprovision factor, attach Volume, copy Template, resize Volume etc that involve Primary Storages capacity calculations would be logged at different log levels (INFO, ERROR, DEBUG) in management server log file.

Upgrade Scenarios

Upgraded setup will have an primary store setting for each primary store with the value same as the global setting.

All storage pools in  a zone which has modified the value from the original value, will have the some modified value in the new setup for each primary data store belonging to that zone.

UI Changes

Primary Storage Tab will have the settings to edit storage,overprovisioning.factor also.

Future Scope (Post 4.4)

Ability to intuitively decide overprovision factor

• Storage stats will now include actual used size (requires writing/modifying resource calls). This value will be used by the admin to decrease/increase the overprovision factor depending upon the actual capacity used.

• For example, a higher value of used size will be indicative of lowering overprovisioning for further vm deployments.

Allow overprovisioning for types other than NFS and VMFS:

• Depending upon the fix for CLOUDSTACK-5806, which as suggested by community requires implementing isOverProvisionSupported(), which will be used to calculate allocated storage as per-primary type basis, thus not restricting this functionality to NFS and VMFS only.

• Incorporating the fix will allow admin to overprovision primary based on the overprovisioning capability.