Table of Contents |
---|
Status
Current state: [One of "Under Discussion", " Accepted", "Rejected"]
Discussion thread: here
JIRA: hereCASSANDRASC-138
Released: <Cassandra Sidecar Version>
...
- Migrate one or more healthy Cassandra instances of a cluster fast.
- Minimize Minimise the down time of a Cassandra instance while migrating.
- Have cleaner abstraction/interfaces for the source.
- The Cassandra instance which is getting replaced is treated as source for the data in this case. Data can be copied from different sources by supporting pluggable source providers/implementations(through cleaner abstraction/interfaces). It allows Sidecar to support other use cases like backup and restore.
Non-Goals
- Making/applying changes to configuration of a Cassandra cluster and instances.
- Managing the Cassandra processes like bringing them up or down while migrating the instances. CEP-1 has it in scope already.
- Migrating non-healthy instances like instance having disk issues. In such cases, they should fall back to normal host replacement.
...
- All files may not be required to copy from source to destination (such as file related to heapdumps, stack traces etc... or any other files maintained for operational purposes but not realted related to Cassandra). User may want to exclude some files/directories from migration. Using simple filters to exclude files/directories for now. Using complex expressions for matching files/directories for exclusion could be added later.To verify whether the desired file set matches with source, only file path and size is considered at the moment. Strict binary level verification is deferred for later.
Proposed Changes
This approach proposes changes only to Cassandra Sidecar. No changes are expected in Cassandra. Below procedure at high level can be followed for migrating a Cassandra instance (source) to another instance (destination) with the help of Sidecars and without bootstrap.
...
- First identify the destination instance. One may make changes to their configuration like topology to bring-in the destination instance into the cluster. How these configuration changes are made and applied are not in the scope of this document.
- Keep the Cassandra process running as usual at source. Do not start the Cassandra process on the destination but start the Sidecaron destination host/instance. Submit data copy request to destination Sidecar to pull data from source using Sidecar.
- During data copy, the destination Sidecar first pulls the list of files present at the source at that point of time.
- Then, the destination sidecar checks the list of files present locally against the list of files at the source.
- If a file is present at the source but not at the destination, then that file will be added to the download list.
- If a file is present at the destination but did does not match ( by size or timestamp) with the source file, then the local file is deleted and added to the list of files to download. It also avoids filling up the disk with un-necessary files. If the file has matched, then it will be excluded from the download listunnecessary files.
- For all files existing at source and destination, file digest is calculated and compared. Files for which digest does not match will be deleted and added to the list of files to download. During implementation a flag will be provided to enable it on demand so that users can enable it selectively (user may opt for disabling it during initial data copy and enable it during final data copy).
- Downloads the shortlisted files.
- While copying the data, source continues to write/update the data leading to add more SSTables or delete existing SSTables due to compaction etc... It is fine to be in this state at this stage.
- After downloading the list of files, the destination may not have some files or may have some files that are no longer present at the source. It may not be possible to ensure that 100% of the data at the destination matches the source as the source continues to run and makes changes. We can relax 100% matching of the data to some lower threshold to consider the data copy a successful one. This success threshold can be specified as part of the request payload. This success threshold will be tested before downloading shortlisted files. If the success threshold is met, then the data copy task is considered as success.
- It could be possible that, after downloading the files once, the threshold was not met. This process can be repeated multiple times to meet the threshold. The number of iterations can be specified as part of the request payload.
- The first iteration is considered a zeroth iteration as the destination may not have anything at that time and the success threshold will not be met.
- If the threshold is met in an iteration, then the data copy operation ends with a success status.
- If the threshold is not met even after multiple iterations, then the data copy task fails. The operators can still continue the migration if they want, because the data will be copied again after bringing down the source.
- Now, bring down the Cassandra process at source, keeping the Sidecar up and running. Once the source Cassandra process goes down, we can expect that there will be no changes made to the files. How the instances are brought down is not in scope of this document.
- Initiate the final data copy at destination with 100% success threshold. Destination pulls a list of files, deleting deletes unnecessary files, skipping matching filesfiles and the files whose digests are not matching, and downloading downloads the required files. Afterwards, it compares the files with the source and checks if there is a 100% match.If the time to perform the final copy is less, then the time difference between the source instance going down and the destination coming up is also less.
- The operator can ensure that files in the destination matches with the source. In the first iteration of this feature, an API is introduced to calculate digest for the list of file names and their lengths to identify any mismatches. It does not validate the file contents at the binary level, but, such feature can be added at a later point of time.
- Now, bring up the Cassandra instance at the destination. At this moment, the destination will have the same data as its source and will be equivalent to an instance coming up with a new IP address. The Sidecar at the source can now be brought down.
- If the destination has more number of data_file_directories compared to source, then executing nodetool relocatesstables is required.
- How to bring up/down Cassandra/Sidecar instances or making/applying config changes are outside the scope of this document. CEP-1 has it in scope already and it can be leveraged when implemented.
Below diagram depicts the above procedure.
This procedure is roughly equivalent to doing a rsync multiple times. Sidecar’s file transmission handler FileStreamHandler is pretty fast and this proposal leverages it.
...
Below new APIs need to be added to Sidecar to facilitate data transfer.
List-files endpoint
Endpoint to list all instance files as URLs by a source, so that a sidecar can download the files using them. Along with a list of files to download, sending total bytes to download will help to check if the success threshold met to proceed further or not. Cassandra handles multiple types of directories for data (data_file_directories), commit log(commitlog_directory), hints(hints_directory), cdc (cdc_raw_directory) etc...
Each file in those directories listed by sidecar roughly looks like this: "<api_version>/migration/files/<type_of_directory>/<optinal_index>/ + relative"
path from the directory.
For example:
If the cassandra.yaml has the config like:
data_file_directories:/var/lib/cassandra/data commitlog_directory:/var/lib/cassandra_commitlog |
---|
And if source has the following files
/var/lib/cassandra/data/ks1/table1/1_Data.db /var/lib/cassandra_commitlog/1.log |
---|
Then the files list exposed by source side car will have the following urls
/api/v1/migration/files/data_file_directories/0/ks1/table1/1_Data.db /api/v1/migration/files/commitlog_directory/1.log |
---|
Notes:
- data_file_directories can have a list of directories. Hence using index to identify and separate each directory.
- This endpoint can also accept a list of files/directories to exclude so that only desired files are listed.
Data copy task
...
endpoints
New endpoints will be introduced to create,
...
get and cancel (if required) data copy tasks
...
from the Sidecar. Each data copy task will make use of a file downloader. File downloader takes the responsibility to get the list of files from the source, remove unnecessary local files, validates the integrity of existing files and downloads the files from source. File downloader doesn't make assumptions about how the source is going to list the files, for it, it is just an URL to download. It also makes use of a location mapper (which can be plugged-in) to map source file urls to local file paths. Since files need to be copied to relatively the same location at the destination too, local paths remain unchanged after the directories path configured in cassandra.yaml. This endpoint can also accept a list of files/directories to exclude so that only desired files are listed.
File download/transfer endpoint
...
Downloads a single file.
...
File digest endpoint
...
Returns digest of a single file specified in the path.
Security
The proposed endpoints are required only at the time performing migrations otherwise they are not enabled by default. These endpoints should be enabled on demand. There is no point in making them available when a migration is not in progress and enabling for instances for which no migration is planned. To facilitate it, a request filter will be introduced for these endpoints which will allow/deny requests. How the filter is configured goes to the implementation details.
...
- Cassandra Sidecar has a way to transmit SSTables exposed via route /api/v1/keyspaces/:keyspace/tables/:table/snapshots/:snapshot/components/:component . It allows to transmit SSTables of a keyspace. For migrating one instance to another, we need to copy all the files and not just SSTables.
- One may utilize utilise the "rsync" Linux tool for data transfer. However, it might not be permitted by the administrator or available in various environments such as Kubernetes or virtual instances like EC2. Enabling data transfer through a sidecar facilitates smooth instance migration.
...