Granular Resource Limit Management

Project Introduction

Granular Resource Limit Management brings the concept of tagged resource limits and improves CloudStack APIs and orchestration framework to allow listing, filtering and deploying instances and volumes with suitable offerings on the specific compute and storage resources. Currently the tagged resources primarily in CloudStack are hosts, storage pool, which are matched against offering related tags during placement, allocation or deployment. Resource limits on tags would allow admins and service providers to limit accounts/domains to usage of tagged resources. For example, place limits on accounts/domain to have a max. allow number VMs on hosts with GPU, x86 and ARM tags to limits how they access specialised hardware.

CloudStack already supports managing account and domain resource limits, ie, the amount of compute, storage and network resources used by the account or domain. But in some cases, the operator may wish even enforce certain limits for specialized hardware. Therefore,  this feature allows adding granular limit on the following host and storage resource types with host and storage tag functionality:

• Host limit resource types : 

  •  user_vm

  • cpu

  • memory

• Storage limit resource type:
  • volume
  • primary_storage

Specialized compute and storages can then be tagged and deployment on them can be controlled with the following:

• Template - For instance deployment using template tag
• Compute offering - For instance and its ROOT disk deployment using host and storage tag respectively
• Disk offering - For DATA volumes using storage tag

The use-case can be understood with an example:

• The operator has an account A in their environment which has different hypervisor hosts, lets say some Intel and some AMD based KVM hosts

• Now operator has set a limit of 10 instances or VMs for account A.
• But the operator wants account A can deploy only 4 instances on the AMD hosts
• With this feature, the operator can use template tag or compute offering host tag. Then use the same tag on AMD hosts.
• This feature will then allow the operator to specify tagged limit for resources, instances in this case.

Currently, there is no way to achieve this scenario in CloudStack. Also, there are the following issues while dealing with deployment on such specialized hardware:

• Template tags are not universally used for deployment.
• UI allows choosing a compute offering that may conflict with the template.
• UI allows choosing a disk offering for a DATA volume creation for an instance that may not be suitable

Functional Description

1. Define new resource limit tags global settings

Add new global settings that can be used for specifying resource limit host and storage tags. This will allow the operators to specify their compute and storage resources as the specialized resource. The operator will be able to set resource limits using these tags for the resource types mentioned above.

1. resource.limit.host.tags

   For compute or host tags. The global setting will accept a comma-separated list of tag strings. These tags can be used to enforce limits on the resource types - user_vm, cpu, memory

2. resource.limit.storage.tags

   For storage tags. The global setting will accept a comma-separated list of tag strings. These tags can be used to enforce limits on the resource types - volumes, primary_storage

2. Allow listing tagged capacities

The operator should be able to list capacities for specialized resources. To enable this, refactor listCapacities API to allow listing for a specific tag. When no tag is passed then return capacities for tags specified in resource.limit.*.tags global settings along with the overall capacities.

3. Allow listing tagged resource count for domains and accounts

The following APIs should return a new response parameter - taggedresources to return a list of the count of resources for the tags specified in the resource.limit.*.tags global settings and resource type - user_vm, cpu, memory, volumes, primary_storage

• listAccounts
• listDomains

4. Allow updating tagged resource counts for domains and accounts

The resource counts for domains and accounts can be recalculated using the updateResourceLimit API. API should allow update for a sub-limit for the tags specified in the resource.limit.*.tags global settings for the respective resource types - user_vm, cpu, memory, volumes, primary_storage


A new request parameter - tag  should be added for the API. This tag must be present in the resource.limit.*.tags global settings.

A new response parameter - tag  should be added for the API.

5. Allow updating tagged resource limits for domains and accounts

The updateResourceLimit API should allow specifying a sub-limit for the tags specified in the resource.limit.*.tags global settings for the respective resource types - user_vm, cpu, memory, volumes, primary_storage


A new request parameter - tag  should be added for the API. This tag must be present in the resource.limit.*.tags global settings.

A new response parameter - tag  should be added for the API.

The specified limit in such a case should be lower than the overall limit for the resource type.

6. Allow listing tagged resource limits for domains and accounts

The listingResourceLimits API should allow listing sub-limits for a given tag which is specified in the resource.limit.*.tags global settings for the respective resource types - user_vm, cpu, memory, volumes, primary_storage


A new request parameter - tag  should be added for the API. This tag must be present in the resource.limit.*.tags global settings.

A new response parameter - tag  should be added for the API.

7. Template tags must be used by different host allocators

Template tag is similar to host tag specified for a compute offering. It should be 

8. UI should not allow using a compute offering that is not suitable for the selected template

To facilitate this listServiceOfferings API should refactored to allowing list offerings for a given template.

A new parameter - templateid should be added.

When this parameter is passed API should return only those service offerings that would be suitable for instance deployment using the given template.

9. UI should not allow using a disk offering for volume creation that is not suitable for a given instance

To facilitate this listDiskOfferings API should refactored to return a flag with each of the disk offerings in the response list

A new parameter - virtualmachineid should be added.

A new response parameter - suitableforvirtualmachine should be added. This would return true or false depending on the offering's suitability for the VM or instance.

UI should handle the filtering. As the suitability is not just a factor of DB stored values and could depend on the hypervisor, filtering from the server side will break pagination for the API.

10. Ensure during deployments (specifically instance and volume) tagged limits are enforced

During deployments, CloudStack should adhere to the overall limits for resource types. It should also adhere to tagged limits for accounts and domains. Resource reservation mechanism should be refactored to prevent any over-utilization.

Database Changes

1. New tag column in cloud.resource_limit table

2. New tag column in cloud.resource_count table

3. Update indexes for cloud.resource_count table

4. New tag column in cloud.resource_reservation table

5. Recreate cloud.account_view to prevent duplicates for tagged resource types

6. Recreate cloud.domain_view to prevent duplicates for tagged resource types