1. Introduction
As the new scheduler depends on ETCD, it is necessary to manage data insertion/deletion/restoration to/from ETCD.
For example, when a new queue is created, the queue endpoint should be included in ETCD.
When a new container is created for a given action, the container information should be stored in ETCD to properly calculate the required number of containers.
When a scheduler is up and running, the endpoint of the scheduler should also be stored in ETCD so that a new queue can be created in the queue.
If such data are removed for some reason even though they are actually running well, data should be restored.
Since many components like MemoryQueue, ContainerProxy, Scheduler, etc, require the same logic, we introduced a dedicated service to manage this kind of workload.
So the DataManagementService is in charge of communicating with ETCD.
It stores data to ETCD, if it couldn't store it for some reason, it retries until succeeds. So it guarantees the data is eventually stored in ETCD.
2. Detail Design
This service receives three types of events,
- Event to store to ETCD.
- Watch-related event.
- Transaction(leader election) event.
- ETC.
2.1 DataStorage event
This type of event is mostly sent from MemoryQueue and ContainerProxy.
This event is to let DataManagementService store data to ETCD.
And there are two events of this type.
- RegisterData: Store data to overwrite the given key.
- RegisterInitialData: Store data if and only if there is no data in ETCD.
- UnregisterData: Remove data with the given key.
For example, when we store data that is used by only one component, we can make sure no other component is updating the data.
The status of a certain MemoryQueue or ContainerProxy would never be updated by the others.
In this case we can use RegisterData to update the data.
But for some data such as namespace throttling status which is shared by multiple queues in a namespace, if there is already data, it should be respected.
In this case, we can use RegisterInitialData. So if one queue already stores the namespace throttling data and updated it to "throttled" status, when another queue is created, it should not overwrite it with other data.
When storing data, a caller can select whether to enable fail-over or not.
When fail-over is enabled, DataManagementService watches the given key and restores data when it is unexpectedly removed.
When removing the data, it tries to unwatch the given key first, and unwatch operation works, then remove the data in ETCD.
So no restoration happens.
2.2 Watch-related event.
DataManagementService is supposed to restore data if it is removed by something other than itself.
For this feature, DataManagementService is watching some ETCD keys.
When the data is removed for some reason The WatchEndpointRemoved event will be sent and it should try to restore the data.
2.3 Transaction event.
This is used when a scheduler tries to create a new queue.
In the current version, there is only one queue running for a given action.
So we need to guarantee a queue is created only in one scheduler.
So when a scheduler receives a queue creation request, it first tries to do the ETCD transaction for a MemoryQueue key.
If the transaction succeeds, other schedulers' transactions would fail and only one scheduler creates a new queue.
The transaction is generally an expensive operation but queue creation happens only once for each action if activations keep coming.
2.4 ETC
Sometimes, a component such as InvokerHealthManager repeatedly stores data to ETCD.
InvokerHealthManager keeps reporting the invoker status(free memory, busyMemory, etc).
If the data is same, we don't need to store the same data to ETCD multiple times.
SO we have UpdateDataOnChange event to store the data only if there is any change.
3. EtcdWorker
Actual ETCD-related operations take place in EtcdWorker only.
So that we can isolate the ETCD dependency to EtcdWorker only.
This worker is only used by DataManagementService and it stores/removes data to/from ETCD according to requests from DataManagementService.