...
- Migrate one or more healthy Cassandra instances of a cluster fast.
- Minimize Minimise the down time of a Cassandra instance while migrating.
- Have cleaner abstraction/interfaces for the source.
- The Cassandra instance which is getting replaced is treated as source for the data in this case. Data can be copied from different sources by supporting pluggable source providers/implementations(through cleaner abstraction/interfaces). It allows Sidecar to support other use cases like backup and restore.
Non-Goals
- Making/applying changes to configuration of a Cassandra cluster and instances.
- Managing the Cassandra processes like bringing them up or down while migrating the instances. CEP-1 has it in scope already.
- Migrating non-healthy instances like instance having disk issues. In such cases, they should fall back to normal host replacement.
...
- First identify the destination instance. One may make changes to their configuration like topology to bring-in the destination instance into the cluster. How these configuration changes are made and applied are not in the scope of this document.
- Keep the Cassandra process running as usual at source. Do not start the Cassandra process on the destination but start the Sidecar. Submit data copy request to destination Sidecar to pull data from source using Sidecar.
- During data copy, the destination Sidecar first pulls the list of files present at the source at that point of time.
- Then, the destination sidecar checks the list of files present locally against the list of files at the source.
- If a file is present at the source but not at the destination, then that file will be added to the download list.
- If a file is present at the destination but does not match by size with the source file, then the local file is deleted and added to the list of files to download. It also avoids filling up the disk with unnecessary files.
- For all files existing at source and destination, file digest is calculated and compared. Files for which digest does not match will be deleted and added to the list of files to download. During implementation a flag will be provided to enable it on demand so that users can enable it selectively (user may opt for disabling it during initial data copy and enable it during final data copy).
- Downloads the shortlisted files.
- While copying the data, source continues to write/update the data leading to add more SSTables or delete existing SSTables due to compaction etc... It is fine to be in this state at this stage.
- After downloading the list of files, the destination may not have some files or may have some files that are no longer present at the source. It may not be possible to ensure that 100% of the data at the destination matches the source as the source continues to run and makes changes. We can relax 100% matching of the data to some lower threshold to consider the data copy a successful one. This success threshold can be specified as part of the request payload. This success threshold will be tested before downloading shortlisted files. If the success threshold is met, then the data copy task is considered as success.
- It could be possible that, after downloading the files once, the threshold was not met. This process can be repeated multiple times to meet the threshold. The number of iterations can be specified as part of the request payload.
- The first iteration is considered a zeroth iteration as the destination may not have anything at that time and the success threshold will not be met.
- If the threshold is met in an iteration, then the data copy operation ends with a success status.
- If the threshold is not met even after multiple iterations, then the data copy task fails. The operators can still continue the migration if they want, because the data will be copied again after bringing down the source.
- Now, bring down the Cassandra process at source, keeping the Sidecar up and running. Once the source Cassandra process goes down, we can expect that there will be no changes made to the files. How the instances are brought down is not in scope of this document.
- Initiate the final data copy at destination with 100% success threshold. Destination pulls a list of files, deletes unnecessary files and the files whose digests are not matching, and downloads the required files. Afterwards, it compares the files with the source and checks if there is a 100% match.
- Now, bring up the Cassandra instance at the destination. At this moment, the destination will have the same data as its source and will be equivalent to an instance coming up with a new IP address. The Sidecar at the source can now be brought down.
- If the destination has more number of data_file_directories compared to source, then executing nodetool relocatesstables is required.
- How to bring up/down Cassandra/Sidecar instances or making/applying config changes are outside the scope of this document. CEP-1 has it in scope already and it can be leveraged when implemented.
Below diagram depicts the above procedure.
...
Below new APIs need to be added to Sidecar to facilitate data transfer.
List-files endpoint
Endpoint to list all instance files as URLs by a source, so that a sidecar can download the files using them. Along with a list of files to download, sending total bytes to download will help to check if the success threshold met to proceed further or not. Cassandra handles multiple types of directories for data (data_file_directories), commit log(commitlog_directory), hints(hints_directory), cdc (cdc_raw_directory) etc...
Each file in those directories listed by sidecar roughly looks like this: "<api_version>/migration/files/<type_of_directory>/<optinal_index>/ + relative"
path from the directory.
For example:
If the cassandra.yaml has the config like:
data_file_directories:/var/lib/cassandra/data commitlog_directory:/var/lib/cassandra_commitlog |
---|
And if source has the following files
/var/lib/cassandra/data/ks1/table1/1_Data.db /var/lib/cassandra_commitlog/1.log |
---|
Then the files list exposed by source side car will have the following urls
/api/v1/migration/files/data_file_directories/0/ks1/table1/1_Data.db /api/v1/migration/files/commitlog_directory/1.log |
---|
Notes:
- data_file_directories can have a list of directories. Hence using index to identify and separate each directory.
- This endpoint can also accept a list of files/directories to exclude so that only desired files are listed.
Data copy task endpoints
New endpoints will be introduced to create, get and cancel (if required) data copy tasks from the Sidecar. Each data copy task will make use of a file downloader. File downloader takes the responsibility to get the list of files from the source, remove unnecessary local files, validates the integrity of existing files and downloads the files from source. File downloader doesn't make assumptions about how the source is going to list the files, for it, it is just an URL to download. It also makes use of a location mapper (which can be plugged-in) to map source file urls to local file paths. Since files need to be copied to relatively the same location at the destination too, local paths remain unchanged after the directories path configured in cassandra.yaml. This endpoint can also accept a list of files/directories to exclude so that only desired files are listed.
File download/transfer endpoint
...
Downloads a single file.
File digest endpoint
Returns digest of a single file specified in the path.
Security
The proposed endpoints are required only at the time performing migrations otherwise they are not enabled by default. These endpoints should be enabled on demand. There is no point in making them available when a migration is not in progress and enabling for instances for which no migration is planned. To facilitate it, a request filter will be introduced for these endpoints which will allow/deny requests. How the filter is configured goes to the implementation details.
...