Status

Current state: "Accepted"

Discussion thread: here

JIRA: CASSANDRA-20975

Released: Not Released

Motivation

Cassandra currently supports compressors LZ4, Zstd, Deflate & Snappy.  During flush & compaction, the CPU spends many of its cycles in compress/decompress functions.  In addition, compression may also be enabled for commitlog and data transfer over network.

Some of the latest processors have integrated hardware accelerators which can speed up these operations. QuickAssist Technology (QAT) is one such accelerator, found on Intel Xeon processors, which can provide hardware acceleration to LZ4, Zstd & Deflate algorithms. Here are some links to details:

1) https://www.intel.com/content/www/us/en/content-details/784036/accelerate-cryptographic-operations-and-data-compression-workloads-with-intel-qat.html?DocID=784036

2) https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html

The idea behind this proposal is to enable offloading compress/decompress to hardware, where applicable. Offloading these operations will free up CPU cores to do other work and will improve compress/decompress performance.  

Audience

Cassandra users and developers

Goals

We propose a framework which can perform the offload when hardware is available and will default to software otherwise. The framework also allows for additional accelerators in the future.  

We expect the hardware to improve throughput over the software implementation anywhere between 6-38% (this result is based on accelerating with QAT under the experiment setup listed below) while decreasing the CPU consumption. Gains will depend on the workload and will be proportional to the time spent in compress/decompress operations.

We will also provide an implementation which accelerates LZ4, ZSTD & Deflate compression using QAT.

Non-Goals

Setting up and enabling accelerators on a node is not a goal for this proposal, we expect the system to be setup correctly with the hardware accelerators according to their documentation, before they are used. If they don’t function correctly, operations will default to software path.

Proposed Changes

Following changes are being proposed,

Each built-in compressor may support a list of accelerators. These will be made available to Cassandra as service providers, when the associated plugin jars are added to the classpath. 
When Cassandra creates a compressor instance, a plugin jar is loaded by Java ServiceLoader, if there are providers available for the compressor. Both the default compressor instance and a plugin compressor instance (obtained from the provider), will be maintained by Cassandra. For subsequent read/write operations, the plugin compressor will be used. However, if the plugin version encounters an error, default compressor will handle the operation.

Source code for QAT pluggable jar will be hosted on Intel's GitHub repository (https://github.com/intel) and jar file will be available via maven central.

New or Changed Public Interfaces

  • ICompressorFactory: Service which creates and provides an instance of an accelerated compressor.
  • AbstractCompressorDecorator, CompressorDecorator: These classes will help maintain a version of the default compressor created by CompressionParams.createCompressor(...), in addition to the plugin. For any reason, if the plugin compressor fails doing an operation, operation will be handled by the default compressor.

Compatibility, Deprecation, and Migration Plan

For QAT, some limitations exist based on history buffer and content size. Software fallback will be used in such scenarios.

More details on feature availability on different platforms and limitations can be found at

https://github.com/intel/QATzip?tab=readme-ov-file#limitations

https://github.com/intel/QAT-ZSTD-Plugin?tab=readme-ov-file#limitations

Test Plan

We plan to have unit tests and data compatible tests between hardware and software.

Setup used to generate results above

System configuration

  • Hardware - Intel(R) Xeon(R) 6979P, 120 cores, 500W TDP, HT On, Turbo On
  • Memory - 1536GB (24x64GB DDR5 6400 MT/s [6400 MT/s])
  • BIOS - BHSDCRB1.IPC.3544.P15.2410232346, microcode 0x81000341
  • Storage - 8x 3.5T SAMSUNG MZQL23T8HCLS-00A07
  • OS, Kernel - Ubuntu 24.04.2 LTS, 6.8.0-59-generic

Cluster configuration

Experiment was run with 8 individual Cassandra server/client instances (did not create a cluster). All the servers were run on one socket and client on the other socket.

Cassandra configuration

  • Version : 5.0.4
  • yaml : cassandra_latest.yaml with
    • flush_compression: table
    • concurrent_reads: 90
    • memtable_flush_writers: 8
    • compaction_throughput: 128MB/s
  • Java : OpenJDK 17.0.15
  • Dataset size : 500M entries per instance


Client configuration

  • Workload driver: NoSQLBench (nb4)
  • Schema: cql-timeseries2.yaml
  • Workloads tested: Read, Mix(50:50), Write
create keyspace if not exists baselines
 	WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}
        AND durable_writes = true;
create table if not exists baselines.iot (
            machine_id UUID,     // source machine
            sensor_name text,    // sensor name
            time timestamp,      // timestamp of collection
            sensor_value double, //
            station_id UUID,     // source location
            data text,
            PRIMARY KEY ((machine_id, sensor_name), time)
            ) WITH CLUSTERING ORDER BY (time DESC)
             AND compression = { 'class' : 'DeflateCompressor' }
             AND compaction = {
             'class': 'TimeWindowCompactionStrategy',
             'compaction_window_size': '60',
             'compaction_window_unit': 'MINUTES'
            };

Results

Throughput and latency were measured, holding load on the server constant in both cases.   


 

Rejected Alternatives

To accommodate new plugins, add a new interface or abstract class for each compressor. But this will require each table to be configured with an appropriate concrete class for compressor, otherwise offloading will not work.



  • No labels