Feature Design for: Webhooks feature

Project Introduction

Admins and users want to consume events without much setup, allowing them to call some script/url or something that triggers when some event happens. (somewhat like a webhooks or workflow feature). A larger workflow feature can be time-consuming, but webhooks is basically what we can target for.

The biggest benefits of this feature are (a) no need to configure rabbitmq/kafka, (b) available for all user types.

Functional Description

  • Webhooks allow external services to be notified when certain events happen.

  • When the CloudStack events happen, it should send a POST request to each of the URLs provided by the admin or user account. See Github webhook docs for inspiration: https://docs.github.com/en/webhooks

  • Content type: the event data is serialized as JSON and sent as the body of the HTTP POST request

  • The webhook secret must be encrypted and stored in CloudStack’s DB

  • Define a global or domain-level webhooks timeout (default 10s); this is the amount of time the server should respond/handle the webhooks request

  • Define thread pool size which are workers to handle webhooks

  • Thoughts on how webhooks are handled:

    • A global dispatcher thread per mgmt server is responsible for dispatching webhook events

    • For each CloudStack event → It should loop through all “active” webhook rules that aren’t removed for the scope (local | domain | global) and create and dispatch runnable payloads to a separate webhook thread pool.

    • For each webhook payload received via the dispatcher, it should attempt to send an HTTP POST request and be bound by a timeout (global setting TBD - says ~10s); for each delivery the history/audit is saved in the DB purging older entries (determined by a global setting) - all events are tried only “X” times (default 1, determined by a dynamic global setting)

    • The feature is implemented as a custom events plugin, that feeds into the webhooks dispatcher.

  • Send webhook headers as part of POST request: refer https://docs.github.com/en/webhooks/webhook-events-and-payloads#delivery-headers

    • X-CS-Event-ID: the event UUID

    • X-CS-Event: the event type

    • X-CS-Signature: This header is sent if the webhook is configured with a secret. This is the HMAC hex digest of the request body, and is generated using the SHA-256 hash function and the secret as the HMAC key.

    • User-Agent: This header will always have the prefix CS-Hookshot/<account UUID>.

1.  New APIs

  • For webhooks entity:
    • createWebhook : create a webhook
    • listWebhooks : list webhooks by project, account/domain, and all (for admin)
    • deleteWebhook : delete a webhook
    • updateWebhook : update a webhook (to enable/disable and change other attributes of the hook)

  • For webhooks history:
    • listWebhookDeliveries : List recent history (keep last X number of deliveries history for troubleshooting and history purposes; controlled by a global setting) (see Github’s for inspiration)
    • deleteWebhookDelivery : Delete the history of deliveries/dispatches for a webhook or a management server
    • executeWebhookDelivery : Ability to trigger a test payload for troubleshooting/testing

2. New Global configurations

  • webhook.delivery.timeout  - Domain-level configuration. Wait timeout (in seconds) for a webhook delivery dispatch. Default 10s.
  • webhook.delivery.tries  - Domain-level configuration. Number of tries to be made for a webhook dispatch. Default value: 3.
  • webhook.dispatch.thread.pool.size  - Global configuration. Size of the thread pool for webhook dispatchers. Default value: 5.
  • webhook.dispatch.history.limit  - Global configuration. Limit for the number of webhook dispatches to keep in history. Default value: 100.`
  • webhook.dispatch.history.cleanup.interval  - Global configuration. Interval (in seconds) for cleaning up webhook dispatch history. Default value: 3600 seconds.

Database Changes

1. New tables

  • cloud.webhook


id of the webhook
uuidvarchar(255)uuid of the webhook
namevarchar(255)name of the webhook
descriptionvarchar(4096)description for the webhook
statechar(32)state of the webhook - Enabled or Disabled
domain_idbigintid of the owner domain of the webhook
account_idbigintid of the owner account of the webhook
payload_urlvarchar(255)payload URL for the webhook
secret_keyvarchar(255)secret key for the webhook
for https payload url
scope for the webhook - Local,Domain,Global
date the webhook was created
removeddatetimedate removed if not null

  • cloud.webhook_dispatch


id of the webhook dispatch
uuidvarchar(255)uuid of the webhook dispatch
event_idbigintid of the event
webhook_idbigintid of the webhook
mshost_msidbigintmsid of the management server
payloadTEXTpayload for the webhook dispatch
successbooleanwebhook dispatch succeeded or not
responseTEXTresponse of webhook dispatch
start_timedatetimestart timestamp of the webhook dispatch
secret_keydatetimeend timestamp of the webhook dispatch

2. New views

  • cloud.webhook_view - Based on cloud.webhook, cloud.account, cloud.domain, cloud.projects
  • cloud.webhook_dispatch_view - Based on cloud.webhook_dispatch, cloud.event, cloud.webhook, cloud.mshost