This page is meant as a template for writing a CEP. To create a CIP choose Tools->Copy on this page and modify with your content and replace the heading with the next CIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: https://lists.apache.org/thread/r0nhyyn6mbpy55fl90xqcj17v6w3wxg3

JIRA: here

Released: <Cassandra Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Scope

The scope of this change is to modify the low level readers/writers to use a modified ChannelProxy instance for all reads and writes.

Goals

  1. To redirect the data storage of specified tables to different storage systems.

Approach

The assumption is that current Cassandra architecture for moving in-memory tables to sstables on disk will be unchanged.  The writing of the files to a different location will be handled by a new ChannelProxyFactory that can determine where to locate the files.

Design Concept

  • Provide a mechanism (ChannelProxyFactory) to configure the FileChannel in SSTable formats and other table directory reads/writes via a ChannelProxy instance.  The ChannelProxyFactory receives a request for a read or write channel for a File.  The factory implementation determines where the File should be read/written and constructs the proper ChannelProxy.  The FileChannelProxy implementation is specified in the cassandra.yaml file as a ParameterizedClass.
  • Modify system so that ChanelProxy can be passed into SSTable readers and writers.  The ChannelProxy contains the FileChannel for the final read/write.  This will allow any system that can be represented as a FileChannel to be used for SSTable I/O.  Additional minor changes are required for some offline functionality, specifically where File objects are used to detect the presence of the file.

SSTable formats will operate as normal, Compaction will operate as normal.  All low level reads and writes will be executed through the FileChannel provided by the ChannelProxy.

No changes to read/write paths until the lowest level.

No changes to format classes, except to allow ChannelProxy to be specified during construction.

Changes

  • Extend the existing ChannelProxy class to provide access to the channel.
  • Create a ChannelProxyFactory abstract class to provide a basis for constructing the ChannelProxys from File objects.
  • Add channel_proxy_factory as a ParameterizedClass in the cassandra.yaml configuration.
  • Implement static methods in ChannelProxyFactory to retrieve ChannelProxys for reading or writing.
  • Implement static ChannelProxyFactory.instance method to construct a ChannelProxy if it has not already been created.  Factory will be created from:
    1. The channel_proxy_factory parameter if it is specified; or
    2. A default factory that constructs ChannelProxies that are equivalent to  the current non-proxied File based usage.
  • Modify associated low level classes to utilize the ChannelProxyFactory.
  • Use ChannelProxyFactory to retrieve FileSystem implementation for tests like File.exists() that normally resolve to the local file system.

Motivation

The initial motivation behind this proposal was the ability to implement a “cold” storage for data that is infrequently accessed and stored on a cheaper storage network.  Subsequently additional use cases were discovered, specifically the ability to move a keyspace and/or table to a different disk to free up space for compaction when the disk has become over full.

We also want to implement this change in a minimally invasive way.

Audience

Devops:

  • Implement “cold” storage.
  • Ability to move some data to pluggable storage without major modifications so that local disk can be freed up for disk intensive operations like compaction where there is an emergent issue.

Goals

  • Allow system operators to temporary move keyspaces or tables to out-of-tree storage to free up space for compaction that would otherwise run out of space while allowing the node to continue  to operate on the moved data.
  • Allow users to specify key ranges that are stored on slower "cold" storage systems.

Non-Goals

This proposal does not implement a "cold" storage system, it only puts in place the framework to allow such a system to be built.

Proposed Changes

  • Add channel_proxy_factory property as a ParameterizedClass implementing ChannelProxyFactory.
  • Implement ChannelProxyFactory as a pluggable component.  It has a static instance() method to retrieve the current factory.  The current factory is set by:
    • Calling the static setInstance(ChannelProxyFactory) method.  Usually used for testing.
    • Reading the channel_proxy_factory property in the Config if available; or
    • Creating a default ChannelProxyFactory that produces ChannelProxies that match the current read/write FileChannels
  • Modify ChannelProxy to allow access to the internal FileChannel
  • Modify SequentialWriter to 
    • Accept ChannelProxy argument in place of File arguments in the constructor.
    • Use ChannelProxy instances internally.
    • Deprecate File based methods in favor of ChannelProxy based methods.
  • Modify FileOutputStreamPlus to accept ChannelProxy as a constructor argument.
  • Modify ChecksummedSequentialWriter to 
    • accept ChannelProxy in place of File arguments in the constructor. 
    • Change the digestFile to an Optional<ChannelProxy>
  • Modify ChecksumWiter to include writeFullChecksum(ChannelProxy) method.
  • Modify DataComponent to create  ChannelProxies for CompressedSequentialWriter and ChecksummedSequentialWriter constructor calls.
  • Modify CompressionMetadata to use ChannelProxy instead of File.
  • Modify CompressedSequentialWriter to use ChannelProxy.
  • Modify FileHandle to use the ChannelProxyFactory.instance() to create the ChannelProxy.  This should handle the SSTableReader implementations.
  • Locate all ChannelProxy constructor calls and switch to ChannelProxyFactory calls.
  • It may be necessary to modify the cassandra.io.util.File class to use a different FileSystem implementation when the File is proxied.  In this case the ChannelProxyFactory will also have to be able to create a FileSystem instance.

New or Changed Public Interfaces

A ChannelProxyFactory class is introduced. The SequentialWriter and FileHandle will be modified to ChannelProxies rather than Files.  I do not believe that either of these changes rise to the level of Changed Public Interfaces.

There is a change to configuration to allow the definition of the ChannelProxyFactory implementation to use.  Not setting the configuration option causes the system to work as it does now.

It may be necessary to change the current ChannelProxy constructor to private to ensure that we find all instances where the ChannelProxy is created directly in the current code.  We will also have to check each occurrence of File.exists() to verify that it does not have to check a remote location.  However, there does not appear to be any change to public interfaces.

Compatibility, Deprecation, and Migration Plan

  • There is no impact on existing users as the change must be triggered via a configuration change or by explicitly calling a ChannelProxyFactory method.
  • Once triggered any data written to alternative locations will not be included in the snapshotting, incremental backups or other processes that use hard file links.
  • The methods that currently use Files but are being replaced with ChannelProxies will be deprecated and will use the ChannelProxyFactory to create ChannelProxies and then delegate to the ChannelProxy version of the methods.
  • There is no need for special migration tools.
  • Any deprecated methods may be removed at the next major release if desired.
  • The compaction behavior should work as it does now.  However if the implementation writes to a slow file system compaction will take much longer to execute.

Test Plan

Initial test plan is to create a configuration that utilizes the ChannelProxyFactory to place the files in a directory outside of the normal space.  Execute the complete test suite and verify passing.  Verify that no files are written to the normal space.

Rejected Alternatives

Implementing redirection of read/writes above the current File/ChannelProxy layers including:

  • A new format - However, the goal here is to use the standard formats but redirect the I/O to a different (non-standard) location.
  • Reworking the read/write paths above the SSTables layer.  This involves significant reworking of a number of classes and major disruption to current processing.


  • No labels