...
This page is meant as a template for writing a CEP. To create a CIP choose Tools->Copy on this page and modify with your content and replace the heading with the next CIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: [One of "Under Discussion", "Accepted", "Rejected"]
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
For those managing Cassandra clusters over extended periods, it's common to encounter the following scenarios:
...
Instead of bootstrapping, data could be transferred from the instance getting replaced to the new instance, and then new instance could just start with same data; this would save a lot of time. This approach is equivalent to an instance coming up with a different IP and can take full advantage of the file streaming Cassandra Sidecar already has. File streaming using the sidecar is much faster than bootstrapping. Multiple instances can be migrated parallelly without any downtime by migrating instances such that quorum of any replica set is live at any point of time (assuming *QUORUM consistency levels are used).
Audience
Cassandra admins, operators & developers who build solutions around Cassandra.
Goals
- Migrate one or more healthy Cassandra instances of a cluster fast.
- Minimize the down time of a Cassandra instance while migrating.
Non-Goals
- Making/applying changes to configuration of a Cassandra cluster and instances.
- Managing the Cassandra processes like bringing them up or down while migrating the instances.
- Migrating non-healthy instances like instance having disk issues. In such cases, they should fall back to normal host replacement.
Out of initial scope
- All files may not be required to copy from source to destination (such as file related to heapdumps, stack traces etc... or any other files maintained for operational purposes but not realted to Cassandra). User may want to exclude some files/directories from migration. Using simple filters to exclude files/directories for now. Using complex expressions for matching files/directories for exclusion could be added later.
- To verify whether the desired file set matches with source, only file path and size is considered at the moment. Strict binary level verification is deferred for later.
Proposed Changes
This approach proposes changes only to Cassandra Sidecar. No changes are expected in Cassandra. Below procedure at high level can be followed for migrating a Cassandra instance (source) to another instance (destination) with the help of Sidecars and without bootstrap.
...
Here, the migration process starts when the source instance is live (hence it is live migration) and the instance is switched in the ring at the end. Let’s look into the details of how a single Cassandra instance will be migrated to a new instance.
MIGRATING ONE INSTANCE
- First identify the destination instance. One may make changes to their configuration like topology to bring-in the destination instance into the cluster. How these configuration changes are made and applied are not in the scope of this document.
- Keep the Cassandra process running as usual at source. Do not start Cassandra process on destination but start Sidecar on destination host/instance. Submit data copy request to destination Sidecar to pull data from source using Sidecar.
- During data copy, the destination Sidecar first pulls the list of files present at the source at that point of time.
- Then, the destination sidecar checks the list of files present locally against the list of files at the source.
- If a file is present at the source but not at the destination, then that file will be added to the download list.
- If a file is present at the destination but did not match (by size or timestamp) with the source file, then local file is deleted and added to list of files to download. It also avoids filling up the disk with un-necessary files. If the file has matched, then it will be excluded from the download list.
- Downloads the shortlisted files.
- While copying the data, source continues to write/update the data leading to add more SSTables or delete existing SSTables due to compaction etc... It is fine to be in this state at this stage.
- After downloading the list of files, the destination may not have some files or may have some files that are no longer present at the source. It may not be possible to ensure that 100% of the data at the destination matches the source as the source continues to run and makes changes. We can relax 100% matching of the data to some lower threshold to consider the data copy a successful one. This success threshold can be specified as part of the request payload.
- It could be possible that, after downloading the files once, the threshold was not met. This process can be repeated multiple times to meet the threshold. The number of iterations can be specified as part of the request payload. If the threshold is met in an iteration, then the data copy operation ends with a success status.
- If the threshold is not met even after multiple iterations, then the data copy task fails. The operators can still continue the migration if they want, because the data will be copied again after bringing down the source.
- Now, bring down the Cassandra process at source, keeping the Sidecar up and running. Once the source Cassandra process goes down, we can expect that there will be no changes made to the files. How the instances are brought down is not in scope of this document.
- Initiate the final data copy at destination with 100% success threshold.
- Destination pulls a list of files, deleting unnecessary files, skipping matching files, and downloading the required files. Afterwards, it compares the files with the source and checks if there is a 100% match.
- If the time to perform the final copy is less, then the time difference between the source instance going down and the destination coming up is less.
- The operator can ensure that files in the destination matches with the source. In the first iteration of this feature, an api is introduced to calculate digest for the list of file names and their lengths to identify any mismatches. It does not validate the file contents at the binary level, but, such feature can be added at a later point of time.
- Now, bring up the Cassandra instance at the destination. At this moment, the destination will have the same data as its source and will be equivalent to an instance coming up with a new IP address. The Sidecar at the source can now be brought down.
- How to bring up/down Cassandra/Sidecar instances or making/applying config changes are outside the scope of this document.
...
This procedure is roughly equivalent to doing a rsync multiple times. Sidecar’s file transmission handler FileStreamHandler is pretty fast and this proposal leverages it.
MIGRATING MULTIPLE INSTANCES
If multiple instances need to be migrated, then above procedure can be executed sequentially. To improve speed of migrations, one can pick multiple instances in such a manner that the number of instances that are brought down at any point of time does not impact the quorum of any replica set at any point of time, assuming *_QUORUM consistency level is used. If much stronger consistency levels are used like ALL, then client would observe interruptions in query processing. The degree of parallelism and execution of migrations are at the discretion of the operator.
Operator can optionally initiate a round of data copy tasks across the board even before starting this procedure on multiple instances to reduce the time taken to migrate an instance or set of instances.
New or Changed Public Interfaces
Below new APIs need to be added to Sidecar to facilitate data transfer.
- Data copy task endpoints: to create, manage file downloads, monitor and cancel (if required) data copy tasks
- List-files endpoint: to list all instance files as Sidecar URLs
- File download/transfer endpoint: to download a single file
- Files digest endpoint: to calculate a digest based on file names and their sizes to identify any mismatch between source and destination
Security
The proposed endpoints are required only at the time performing migrations otherwise they are not enabled by default. These endpoints should be enabled on demand. There is no point in making them available when a migration is not in progress and enabling for instances for which no migration is planned. To facilitate it, a request filter will be introduced for these endpoints which will allow/deny requests. How the filter is configured goes to the implementation details.
When migration is in progress, the new endpoints can expose files present on the disk which are downloadable. So, access to these endpoints should be strictly restricted to admin/operator/tooling only. Hence, protecting these endpoints is mandatory.
Endpoint | Enabled at source? | Enabled at destination? |
---|---|---|
List files endpoint | Yes | No (can be enabled for debugging) |
Data copy task endpoints | No | Yes |
File digest endpoint | Yes | Yes |
File downloader | Yes | No |
Compatibility, Deprecation, and Migration Plan
This is a new feature. Compatibility, deprecation and migration plan are not applicable.
Test Plan
Tests will be added in Cassandra Sidecar convering end-to-end flow.
Rejected Alternatives
- Cassandra Sidecar has a way to transmit SSTables exposed via route /api/v1/keyspaces/:keyspace/tables/:table/snapshots/:snapshot/components/:component . It allows to transmit SSTables of a keyspace. For migrating one instance to another, we need to copy all the files and not just SSTables.
- One may utilize the "rsync" Linux tool for data transfer. However, it might not be permitted by the administrator or available in various environments such as Kubernetes or virtual instances like EC2. Enabling data transfer through a sidecar facilitates smooth instance migration.
...