Status

Current state: Accepted

Discussion thread: Discussion

JIRA: TBD

Released: TBD

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).


Scope

This proposal introduces configuration management capabilities to Cassandra Sidecar, enabling programmatic control over cassandra.yaml and JVM options through a REST API. The system uses a two-tier configuration model (base templates + overlays), a pluggable storage abstraction for integration with centralized configuration systems, and version-aware validation to prevent startup failures. All changes are implemented in Sidecar with no modifications to Cassandra itself.

Motivation

Many Cassandra settings (e.g., memtable configuration, SSTable settings, storage_compatibility_mode) must be configured via cassandra.yaml and cannot be modified at runtime. Similarly, startup parameters such as heap sizing and GC behavior are specified through JVM options files (e.g., jvm-server.options). Unlike runtime settings that can be changed through JMX, Cassandra does not offer any API to modify these persisted configuration artifacts. Managing them manually or through custom tooling is cumbersome, lacks a stable API surface, and complicates integration with centralized configuration providers.

The introduction of lifecycle management in CASSSIDECAR-266 gave Sidecar the ability to start and stop Cassandra instances through a pluggable LifecycleProvider interface. However, Sidecar does not currently provide a way to programmatically modify the configuration artifacts consumed during startup. Since Sidecar already controls the process lifecycle, it can also manage the Cassandra configuration artifacts, ensuring that changes are validated, persisted, and applied consistently each time an instance is started.

This CEP introduces configuration management capabilities to Sidecar, providing a deterministic, version-aware, and optionally remotely synchronized mechanism for managing cassandra.yaml and JVM options.

Goals

  • Programmatic Management of cassandra.yaml and JVM Options: Provide a stable API in Sidecar to read and update the effective cassandra.yaml configuration and extra JVM options of a Cassandra instance.
  • Persist Configuration Changes Across Restart: Enable configuration changes to be durably stored and automatically applied when Cassandra instances are restarted.
  • Support Parameters That Cannot Be Modified at Runtime: Provide a mechanism to change settings in cassandra.yaml and JVM options that cannot be modified via JMX or CQL, like storage_compatibility_mode.
  • Support All Cassandra Versions Managed by Sidecar: Provide configuration management capabilities for all Cassandra versions currently supported by Sidecar.
  • Pluggable Configuration Provider: Support integration with custom configuration providers that can be local or remote/centralized.
  • Version-Aware Safety: Validate configuration updates against the Cassandra version detected for each instance, preventing unsupported configuration keys from being written.
  • Extensibility: Allow the solution to be extended to future configuration types.

Non-Goals

  • Runtime Configuration Mutation: This CEP does not introduce changes to runtime configuration via JMX or CQL. All changes apply to cassandra.yaml and JVM options and take effect on restart.
  • Cassandra Internal Configuration System Refactoring: This proposal does not refactor Cassandra's internal configuration architecture as described in CASSANDRA-15254. This CEP manages the persisted configuration files externally through Sidecar, while CASSANDRA-15254 addresses how Cassandra internally represents, validates, and processes configuration after loading.
  • Cluster-wide Orchestration: This proposal does not define rolling restart coordination or cluster-level change orchestration.
  • Configuration History or Audit Tracking: No change history, diff API, or configuration audit log is provided in this version.
  • Cross-Version Configuration Migration: This CEP does not handle automatically migrating or evolving cassandra.yaml settings across Cassandra major versions. This will be addressed in a follow-up CEP covering Cassandra upgrades via Sidecar.

Proposed Changes

A new configuration management module will be introduced to sidecar with the following components:

Configuration API: an HTTP interface for retrieving and updating instance configuration, including cassandra.yaml and JVM Options.

Configuration Manager: centralizes all aspects of configuration management, including serving requests from the API, persisting the configuration and fetching configuration from the configuration provider.

Configuration Store: a directory on disk where configuration artifacts are stored and managed by Sidecar.

Configuration Provider: A pluggable abstraction for storing and retrieving configuration overlays. The default implementation is DefaultConfigurationProvider, which stores overlays as JSON files locally. Custom providers can integrate with centralized configuration stores such as etcd, Consul, or custom REST APIs.

Lifecycle Manager: this existing component handles Cassandra startup requests, it will interact with the configuration manager to refresh the configuration when starting up a Cassandra process.

Lifecycle Provider: this existing component implements the lifecycle start action and must ensure the configuration artifacts from the configuration store are used by the runtime Cassandra process.


Fig 1: New architecture proposed by this CEP

Configuration Model

The instance configuration is represented as a multi-section JSON, where each section represents a configurable entity. Additional metadata fields like "hash" and "lastModified" are provided for integrity and change tracking.

This CEP supports two configuration types:

  • cassandraYaml — A sparse representation of the settings defined in cassandra.yaml. It contains only settings that have been explicitly set and does not include default values.
  • extraJvmOpts — A list of JVM option strings that are appended to the Cassandra JVM startup command.

Additional configuration types like cassandraRackDc or environmentVariables can be incorporated in the future without changing the configuration format. An example configuration is provided below.

{
  "hash": "sha256:5f4dcc3b5aa765d",
  "lastModified": "2026-02-20T14:32:18Z",
  "configuration": {
    "cassandraYaml": {
      "cluster_name": "my_cluster",
      "memtable_flush_writers": 8,
      "storage_compatibility_mode": "CASSANDRA_5",
      "commitlog_sync": "periodic",
      "num_tokens": 16,
      "memtable": {
        "configurations": {
          "skiplist": { "class_name": "SkipListMemtable" },
          "trie": { "class_name": "TrieMemtable" },
          "default": { "inherits": "skiplist" }
        }
      }
    },
    "extraJvmOpts": [
      "-Dcassandra.available_processors=8",
      "-Dcassandra.ring_delay_ms=60000"
    ]
  }
}

Configuration Overlay

The base template provides the base configuration for cassandra.yaml, defined in the instance configuration section of sidecar.yaml. It contains the default or standard configuration values that apply to each instance. The base template is required for cassandraYaml, while extraJvmOpts has no base template and defaults to an empty list.

An overlay is a sparse set of configuration values that have been explicitly set via HTTP PATCH operations. Overlay values can overwrite base template values or add new configuration attributes not present in the template. The overlay is stored and retrieved by the ConfigurationProvider, a pluggable abstraction that allows overlays to be persisted locally or in remote centralized systems.

The effective configuration is computed by merging the base template with the overlay, where overlay values take precedence over template values. This produces the final cassandra.yaml and extra-jvm.options files that are used by the Cassandra process. The effective configuration is what the HTTP GET API returns and what gets materialized to disk for the Lifecycle Provider to use.

Example:

Base Template (from sidecar.yaml):

cluster_name: production_cluster
memtable_flush_writers: 4
commitlog_sync: periodic
commitlog_sync_period: 10000

Overlay:

memtable_flush_writers: 8
concurrent_reads: 64

Effective Configuration (materialized):

cluster_name: production_cluster      # from template (not in overlay)
memtable_flush_writers: 8             # from overlay  (overwrites template)
commitlog_sync: periodic              # from template (not in overlay)
commitlog_sync_period: 10000          # from template (not in overlay)
concurrent_reads: 64                  # from overlay  (not in template)

This two-tier approach ensures that common settings are defined once in the base template while instance-specific customizations are captured in the overlay. During PATCH operations, updates are applied to the overlay, not the base template. Operators can update the base template for all instances by modifying sidecar.yaml and calling the reload configuration endpoint, while individual instance customizations remain preserved in their respective overlays.

Configuration Store

The configuration store is a directory on disk where the Configuration Manager writes configuration artifacts for each managed Cassandra instance. The directory location is specified by the ${configuration_management.configuration_store} attribute in sidecar.yaml.

The store contains both configuration overlays cached locally and materialized effective configuration artifacts. The Configuration Manager creates the materialized artifacts by merging base templates with overlays, producing the final cassandra.yaml and extra-jvm.options files that the LifecycleProvider uses when starting Cassandra processes.

Directory Structure

The store is partitioned by instance, with each subdirectory containing the configuration artifacts for a specific instance:

${configuration_management.configuration_store}/
├── instance-1/
│   ├── overlay.json
│   ├── cassandra.yaml
│   └── extra-jvm.options
├── instance-2/
│   ├── overlay.json
│   ├── cassandra.yaml
│   └── extra-jvm.options
└── instance-N/
    ├── overlay.json
    ├── cassandra.yaml
    └── extra-jvm.options

Artifacts

Each instance directory contains the following artifacts:

overlay.json — A sparse JSON file containing configuration values that have been explicitly set, along with its SHA-256 hash and last modification timestamp. DefaultConfigurationProvider treats this as the authoritative overlay. This is a cached copy of the overlay for custom configuration providers used to support offline operation.

cassandra.yaml — The materialized effective configuration file, computed by merging the base template with the overlay. This file is a cache of the merged result and is read by the Lifecycle Provider when starting the Cassandra process.

extra-jvm.options — The materialized list of extra JVM options, one option per line. This file is computed from the overlay and is used by the Lifecycle Provider to append options to the Cassandra JVM startup command.

Overlay Format

The overlay.json file follows the same structure as the configuration model, but is sparse and contains only explicitly set values:

{
  "hash": "sha256:3a5b7c9d1e2f4a6b8c0d2e4f6a8b0c2d4e6f8a0b2c4d6e8f0a2b4c6d8e0f2a4b",
  "lastModified": "2026-02-20T15:04:22Z",
  "cassandraYaml": {
    "memtable_flush_writers": 8,
    "concurrent_reads": 64
  },
  "extraJvmOpts": [
    "-Dcassandra.available_processors=8",
    "-Dcassandra.ring_delay_ms=60000"
  ]
}


Configuration API

The configuration management module exposes three REST endpoints under the Cassandra namespace:

  • GET /api/v1/cassandra/configuration — Retrieve the effective configuration for an instance.
  • PATCH /api/v1/cassandra/configuration — Update the configuration overlay using JSON Patch operations.
  • POST /api/v1/cassandra/configuration/reloadConfig — Reload the effective configuration from base templates and the ConfigurationProvider

Full API specifications are detailed in the New or Changed Public Interfaces section.

Authorization

Sidecar uses per-endpoint authorization where each handler declares the permissions it requires. Permissions follow a DOMAIN:ACTION format (e.g., SNAPSHOT:CREATE, LIFECYCLE:MODIFY) and are scoped to a resource level (cluster, keyspace, or table). Permissions are resolved from Sidecar's sidecar_internal.role_permissions_v1 tables.

When Cassandra is not running in the local node,  Sidecar can resolve permissions from other nodes in the cluster. Operators should configure multiple contact points in sidecar.yaml so that Sidecar can reach the authorization tables even when the local managed instance is down. Alternatively, operators can configure the admin_identities property to define identities that bypass authorization entirely, ensuring that administrative operations remain available even when no Cassandra node is reachable for permission resolution.

The configuration management endpoints introduce two new permissions at cluster scope:

  • CONFIGURATION:READ — Required for GET /api/v1/cassandra/configuration and POST /api/v1/cassandra/configuration/reloadConfig
  • CONFIGURATION:MODIFY — Required for PATCH /api/v1/cassandra/configuration

Configuration Provider

The Configuration Provider is responsible for storing and retrieving configuration overlays. It is a pluggable abstraction that decouples Sidecar's configuration management from any specific storage backend, allowing operators to integrate with their existing infrastructure.

DefaultConfigurationProvider

The default implementation, DefaultConfigurationProvider, persists configuration overlays as local JSON files within the configuration store. When retrieving an overlay, it reads the overlay.json file from the instance directory. When persisting an overlay update, it atomically writes the updated overlay to overlay.json.

Custom Configuration Providers

Operators can implement custom providers to integrate with centralized configuration systems such as etcd, Consul, or custom REST APIs. A custom provider might fetch overlays from a remote key-value store, allowing configuration to be managed centrally across multiple Sidecar instances.

Configuration Manager

The Configuration Manager coordinates all configuration operations between the REST API, the ConfigurationProvider, and the Configuration Store. It handles the following responsibilities: merging base templates with overlays to compute effective configurations, materializing configuration artifacts to disk, and validating updates against a version-aware configuration schema.

This component integrates with the Configuration API to serve GET and PATCH requests, with the ConfigurationProvider to fetch and persist overlays, with the Configuration Store to write configuration artifacts, and with the LifecycleManager to determine instance Cassandra versions for validation.

Initializing the Configuration

During Sidecar startup, the Configuration Manager initializes the configuration store for each managed instance. It fetches the overlay from the ConfigurationProvider and merges it with the base template from sidecar.yaml to produce the effective configuration.

Before materializing the artifacts, the manager validates the effective configuration against the version-aware schema for the instance's Cassandra version. This validation ensures that the base template does not contain invalid keys for the detected Cassandra version. If validation fails, initialization is aborted and an error is logged.

Once validated, the manager writes the effective configuration to the Configuration Store as cassandra.yaml and extra-jvm.options files. The manager also writes overlay.json to cache the overlay locally when it differs from the currently stored version.

Retrieving the Configuration

When handling GET requests, the Configuration Manager fetches the current overlay from the ConfigurationProvider and merges it with the base template to compute the effective configuration. It calculates the SHA-256 hash of the effective configuration and returns the complete snapshot with hash and lastModified timestamp to the API client. The DefaultConfigurationProvider reads the local overlay.json file, while custom providers may fetch from remote systems like etcd or Consul.

As a side effect, retrieval refreshes the Configuration Store when the fetched overlay differs from the cached version. The manager updates overlay.json with the new overlay, then rematerializes the effective configuration artifacts. This keeps the local cache synchronized even when the overlay has been modified externally through the provider.

When the ConfigurationProvider is unavailable, the configured failure policy determines whether update falls back to the cached overlay or fails the request. Failure policy behavior is detailed in the Failure Policy section. If no cached overlay exists (e.g., first-time initialization), the base template alone is used to materialize the effective configuration.

Updating the Configuration

When handling PATCH requests, the Configuration Manager acquires a per-instance exclusive lock to serialize concurrent updates and prevent race conditions. It validates the expectedHash from the request against the current effective configuration hash to detect stale updates from conflicting modifications.

The manager applies the JSON Patch operations to compute the updated overlay. It then validates this result against a version-aware configuration schema maintained per Cassandra major version to ensure that all cassandraYaml keys being updated are recognized for that version. Updates of unknown keys are rejected to prevent Cassandra startup failures unless skipValidation is set to true. The extraJvmOpts values bypass schema validation as they are opaque JVM arguments.

Once validated, the manager persists the updated overlay to the ConfigurationProvider. The DefaultConfigurationProvider atomically writes to overlay.json, while custom providers may propagate changes to remote systems. After the update is propagated to the provider, the cached overlay as well as materialized artifacts are updated in the configuration store.

When the ConfigurationProvider is unavailable, the configured failure policy determines whether update falls back to the cached overlay or fails the request. Failure policy behavior is detailed in the Failure Policy section.

Lifecycle Provider Integration

Lifecycle Management in sidecar offers APIs to start and stop Cassandra processes via a pluggable LifecycleProvider interface. A default ProcessLifecycleProvider is shipped to start and stop Cassandra as a local process. Currently there’s no way to programmatically update the configuration of a Cassandra process started via the LifecycleProvider.

While the Configuration Manager will manage the configuration lifecycle, the component responsible for loading the configuration into the Cassandra process is the LifecycleProvider.

When a Cassandra instance is started through the lifecycle API (PUT /api/v1/cassandra/lifecycle with payload {"state": "start"}), Sidecar ensures configuration is up-to-date by fetching the latest overlay from the Configuration Manager. The configured LifecycleProvider then must use the materialized configuration artifacts (cassandra.yaml and extra-jvm.options) from the configuration store to start the Cassandra process.

ProcessLifecycleManager

The default ProcessLifecycleProvider configures a $CASSANDRA_CONF directory where all runtime configuration artifacts are expected by Cassandra. This directory is distinct from the local configuration store managed by Sidecar since it contains additional configuration artifacts not currently managed.

Prior to starting the Cassandra process, the ProcessLifecycleProvider copies the materialized cassandra.yaml file from the local configuration store to the $CASSANDRA_CONF location where it is expected by the Cassandra process, and appends the extraJvmOpts entries to the JVM startup command. Because the effective configuration is always recomputed and written to the configuration store before starting the process, the materialized artifacts are guaranteed to reflect the latest state from the base template and the ConfigurationProvider overlay. Any prior corruption or manual modification of the materialized files is overwritten during this step.

Custom ConfigurationLoader Considerations

The DefaultConfigurationProvider and ProcessLifecycleProvider are designed to work together: overlays are persisted locally, and the materialized cassandra.yaml is copied to $CASSANDRA_CONF where Cassandra's default YamlConfigurationLoader reads it. When operators use a custom Cassandra ConfigurationLoader (e.g., one that fetches configuration from a remote system instead of reading cassandra.yaml from disk), the materialized cassandra.yaml file may not be consumed by the Cassandra process. In this scenario, operators should implement a custom ConfigurationProvider that shares the same backing store as their ConfigurationLoader, ensuring that configuration updates made through Sidecar's API are propagated to the system that the ConfigurationLoader reads from.

Failure Policy

The behavior when sidecar is not able to communicate with the ConfigurationProvider is governed by the failure_policy setting. The policy affects the following operations:

  • Sidecar Initialization — Initializing the configuration store during Sidecar startup.
  • Read operations (Retrieve, Reload Config, Lifecycle Start) — Operations that fetch the overlay from the provider to compute or refresh the effective configuration.
  • Update (PATCH /api/v1/cassandra/configuration) — Modifying the configuration overlay through the provider.

Three policies are available:

  • CACHED_READ_WRITE — All operations fall back to the cached configuration. Read operations use cached configuration, and updates are applied to the cache. This policy maximizes availability at the cost of potential divergence between cached and remote state.
  • CACHED_READ_ONLY (default) — Read operations fall back to the cached configuration, but updates are rejected when the provider is unreachable. This favors availability for reads while preventing divergence from writes.
  • FAIL — All operations fail when the provider is unreachable. This policy favors consistency, ensuring no operation proceeds with potentially stale or divergent configuration.

The behavior for each combination of policy and operation is summarized in the following table:

PolicyInitializationRead operationsUpdate
CACHED_READ_WRITEUse cached configurationUse cached configurationUpdate cache
CACHED_READ_ONLY (default)Use cached configurationUse cached configurationFail
FAILFailFailFail

Version-Aware Configuration Schema

Sidecar maintains a version-aware configuration schema used to validate configuration updates. This is required because Cassandra rejects unknown or invalid configuration keys during startup, and Sidecar must ensure that configuration changes applied via its API cannot render a node unstartable.

The schema is represented using JSON Schema (Draft 2020-12). During the Sidecar build process, a JSON Schema specification is generated for each supported Cassandra major version, capturing the valid configuration keys, their types, and structure for that version. These schemas are bundled with Sidecar and loaded at runtime based on the detected Cassandra version of each managed instance.

The schema is keyed by Cassandra major version (for example 5.0, derived from an installed version such as 5.0.2). Sidecar uses this schema in two places:

  1. PATCH validation (strict): Sidecar validates configuration updates against the JSON Schema for the instance’s Cassandra version, rejecting updates that introduce unknown keys or type mismatches.

  2. Remote overlay consumption (tolerant): When fetching configuration from the provider, Sidecar validates against the same JSON Schema but skips unknown keys and logs a warning, ensuring provider integration cannot prevent startup due to extraneous remote settings.

Example JSON Schema (abbreviated, for Cassandra 5.0):


{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "cassandra-5.0-config-schema.json",
  "title": "Cassandra 5.0 Configuration Schema",
  "type": "object",
  "properties": {
    "cluster_name": {
      "type": "string"
    },
    "num_tokens": {
      "type": "integer"
    },
    "memtable_flush_writers": {
      "type": "integer"
    },
    "commitlog_sync": {
      "type": "string",
      "enum": ["periodic", "batch", "group"]
    },
    "commitlog_sync_period": {
      "type": "integer"
    },
    "storage_compatibility_mode": {
      "type": "string",
      "enum": ["NONE", "UPGRADING", "CASSANDRA_4"]
    },
    "concurrent_reads": {
      "type": "integer"
    },
  },
  "additionalProperties": false
}


The additionalProperties: false constraint is what enables Sidecar to reject unknown keys during strict PATCH validation. During tolerant provider consumption, unknown properties are stripped rather than causing a rejection.

Version Detection

Currently sidecar does not track Cassandra versions of the instances it manages. Since the Cassandra version is an attribute managed by the LifecycleProvider and is implementation-dependent, a new method getCassandraVersion will be added to the LifecycleProvider interface to fetch that information.

When version cannot be determined: If getCassandraVersion returns null or the version cannot be parsed, PATCH operations that modify cassandraYaml are rejected with a 503 Service Unavailable error indicating that version-aware validation cannot be performed.

Note that extraJvmOpts values are opaque strings and are not subject to version-aware schema validation. They are passed through directly to the JVM command line without any version-specific validation by Sidecar.

Immutable Settings Blocklist

Certain cassandraYaml settings are destructive or unsafe to modify after initial setup. Sidecar maintains a global blocklist (not per-version) of immutable settings that PATCH operations cannot overwrite once set. The initial blocklist includes:

  • cluster_name
  • partitioner
  • num_tokens
  • initial_token

A PATCH targeting a blocklisted key is allowed only if the key is absent from both the base template and the current overlay (i.e., it has never been explicitly configured). This permits initial cluster setup via the API while preventing accidental mutation of values that would corrupt the cluster. If the key already exists in either the template or the overlay, the PATCH is rejected with 422 Unprocessable Entity. The skipValidation flag does not bypass the immutable settings check, as these protections guard against data-loss scenarios rather than schema unknowns.

Configuration Hash

The hash field of the configuration model is a SHA-256 checksum of the effective configuration (base template merged with overlay), computed on-the-fly and prefixed with sha256:. It serves as a version identifier for optimistic concurrency control, a mechanism that detects conflicting updates without requiring locks during reads.

Clients must include the hash from their most recent GET when submitting a PATCH. If another client has modified the configuration in the meantime, the hash will no longer match and the update is rejected with 409 Conflict, prompting the client to re-fetch and retry. This check can be bypassed with force: true, though this risks overwriting concurrent changes.

To prevent race conditions when multiple PATCH requests arrive simultaneously with the same hash, Sidecar serializes all PATCH operations per instance using exclusive locks. The first request to acquire the lock succeeds if the hash matches; subsequent requests wait for the lock, then fail with 409 Conflict because the hash will have changed. These mechanisms are complementary: locks prevent race conditions when requests arrive simultaneously, while hash checks prevent lost updates when requests are based on stale data.

Sidecar maintains two hash values: the effective configuration hash (base template + overlay) is computed on-the-fly when serving API requests and is used for API client concurrency control. The overlay hash is cached from the ConfigurationProvider and stored in overlay.json. It is used for provider-level concurrency control when sending configuration updates to the ConfigurationProvider to prevent concurrent modifications to the overlay.

New or Changed Public Interfaces

LifecycleProvider Interface

The existing LifecycleProvider interface manages the lifecycle of Cassandra instances (start, stop, status checks). A new getCassandraVersion method will be added to support version-aware configuration validation.

Since the Cassandra version is implementation-dependent, each LifecycleProvider implementation must report the version of the instance it manages. The Configuration Manager uses this to determine which version-aware schema to apply when validating configuration updates.

For example, ProcessLifecycleProvider parses the version from CHANGES.txt at $CASSANDRA_HOME, while a Docker-based provider would use the configured image version.

LifecycleProvider.java
/**
 * Manages the lifecycle of Cassandra instances through Sidecar
 */
public interface LifecycleProvider
{
    void start(InstanceMetadata instance);
    void stop(InstanceMetadata instance);
    boolean isRunning(InstanceMetadata instance);

    /**
     * Returns the Cassandra version of the given instance.
     * Used by the configuration manager for version-aware schema validation.
     *
     * @param instance Cassandra instance metadata
     * @return the Cassandra version of the instance, or null if the version cannot be determined
     */
    Version getCassandraVersion(InstanceMetadata instance);
}

Sidecar Configuration

A new configuration_management section is added to sidecar.yaml with the structure below.

configuration_management:
  enabled: true
  configuration_store: "/etc/cassandra-sidecar/configuration"
  templates:
    cassandra_yaml: "/etc/cassandra/base-template.yaml"
  failure_policy: CACHED_READ_ONLY  # CACHED_READ_WRITE | FAIL
  provider:
    class_name: org.apache.cassandra.sidecar.configuration.DefaultConfigurationProvider
    # provider-specific settings...

It should be possible to override configuration templates per instance, as in the example below:

cassandra_instances:
- id: 1
  host: localhost1
  port: 9042
  configuration_management:
    templates:
      cassandra_yaml: "/etc/cassandra/custom-node1.yaml"

- id: 2
  host: localhost2
  port: 9042
  # Uses global template

- id: 3
  host: localhost3
  port: 9042
  configuration_management:
    templates:
      cassandra_yaml: "/etc/cassandra/custom-node3.yaml"

Configuration REST API

Following existing Sidecar API conventions, we propose a small set of endpoints under the Cassandra namespace to read and update the effective instance configuration (cassandra.yaml and extraJvmOpts). The API is designed around:

  • A single configuration object that can incorporate additional kinds later.

  • Optimistic concurrency using a hash of the effective configuration (see Configuration Hash section).

  • RFC 6902-inspired JSON Patch for updates (with restrictions on path depth).

  • Pluggable ConfigurationProvider integration

Relationship with Existing Settings Endpoints

Sidecar already exposes two endpoints for querying Cassandra configuration:

  • GET /api/v1/cassandra/settings — Returns immutable gossip-level node metadata such as release version, partitioner, datacenter, RPC address/port, and tokens. These values are fixed for the lifetime of the process and are not affected by JMX changes or by configuration changes made through the API proposed here.
  • GET /api/v2/cassandra/settings — Returns runtime configuration values read from the system_views.settings CQL virtual table. These values reflect the live state of the running process and can be modified at runtime via JMX.

In contrast, the endpoints introduced by this proposal manage the startup (persisted) configuration — the cassandra.yaml and JVM options that are loaded when Cassandra starts. The persisted configuration and the v2 runtime settings can diverge; for example, a setting modified at runtime via JMX will be reflected by /api/v2/cassandra/settings but will not appear in the persisted configuration managed here. Conversely, a change applied through the configuration API proposed here will only take effect after the next instance restart.

GET /api/v1/cassandra/configuration

Get the effective configuration for the instance by fetching the overlay from the ConfigurationProvider and merging it with the base template.

Response Body

{
  "hash": "sha256:...",
  "lastModified": "2026-02-20T14:32:18Z",
  "configuration": {
    "cassandraYaml": { },
    "extraJvmOpts": [ ]
  }
}


Response Codes

  • 200 OK — configuration returned

  • 500 Internal Server Error — unable to read/parse local configuration

  • 503 Service Unavailable — ConfigurationProvider unavailable and failure_policy=FAIL

Sample Response

The response returns the effective configuration snapshot, including the hash for optimistic concurrency, the last modification timestamp, and the sparse cassandraYaml and extraJvmOpts sections.

{
  "hash": "sha256:5f4dcc3b5aa765d61d8327deb882cf99e3f0f0e6a0d8b3f4a1c2e5f6b7a8c9d0",
  "lastModified": "2026-02-20T14:32:18Z",
  "configuration": {
    "cassandraYaml": {
      "cluster_name": "my_cluster",
      "memtable_flush_writers": 8,
      "storage_compatibility_mode": "CASSANDRA_5"
    },
    "extraJvmOpts": [
      "-Dcassandra.available_processors=8",
      "-Dcassandra.ring_delay_ms=60000"
    ]
  }
}

PATCH /api/v1/cassandra/configuration

Update the effective configuration for the instance by modifying the configuration overlay.

PATCH applies JSON Patch operations (inspired by RFC 6902, with restrictions) to modify the configuration overlay. The request must include an expectedHash for optimistic concurrency. An optional force flag bypasses failure_policy enforcement when the ConfigurationProvider is unavailable, allowing local persistence regardless of provider state.

Remove operations delete keys from the overlay, causing the effective configuration to fall back to the base template value (if the key exists in the template) or to the Cassandra default (if the key does not exist in the template).

This CEP uses JSON Patch syntax inspired by RFC 6902, but with restrictions that prevent strict RFC 6902 compliance. Each operation includes a path field that uses JSON Pointer syntax (RFC 6901) to navigate the document structure: paths start with / and use / as a delimiter between keys (e.g., /configuration/cassandraYaml/concurrent_writes refers to the concurrent_writes field within cassandraYaml within configuration). For arrays, use numeric indices or - to append.

Only two operations are supported: add (insert or update a value, acting as an upsert) and remove (delete a value). Unsupported RFC 6902 operations (replace, move, copy, test) will be rejected with a 400 Bad Request error. For example, {"op": "add", "path": "/configuration/cassandraYaml/concurrent_writes", "value": 128} will set concurrent_writes to 128, whether or not it already exists.

Request Body

FieldTypeOptionalDefaultDescription
expectedHashstringfalseN/AHash returned by the most recent GET of the effective configuration. Used to prevent lost updates.
forcebooleantruefalseBypasses expectedHash validation and failure_policy enforcement. Does not bypass schema validation.
skipValidationbooleantruefalseUNSAFE: Bypasses version-aware schema validation for cassandraYaml keys. Intended for break-glass scenarios when adding configuration keys not yet supported by the schema. May render the node unstartable if invalid settings are used. If the node fails to start, the invalid setting must be removed via another PATCH operation.
patcharrayfalseN/AJSON Patch operations (RFC 6902-inspired) applied to the effective configuration.

Patch Operation Requirements

  • Uses RFC 6902 JSON Patch syntax with the following restrictions:

    • Only add and remove operations are supported (not replace, move, copy, test)
    • Patch operations must target top-level cassandraYaml keys only (e.g., /configuration/cassandraYaml/memtable). Partial updates to nested paths within a complex setting (e.g., /configuration/cassandraYaml/memtable/configurations/more_shards/shards) are not supported and will be rejected. Support for partial updates of complex settings may be added in a future enhancement.
    • Multiple operations targeting the same field path are not allowed in a single PATCH request and will be rejected with 400 Bad Request.

  • PATCH is strict (unless skipValidation: true):

    • Unknown configuration kinds are rejected.

    • Unknown cassandraYaml keys (per version-aware schema) are rejected.

    • Immutable blocklisted keys (cluster_name, partitioner, num_tokens, initial_token) are rejected with 422 if the key already exists in the template or overlay. skipValidation does not bypass this check (see Immutable Settings Blocklist).

  • If patch results in invalid keys for the Cassandra version line, Sidecar returns 422 (unless skipValidation: true).

Response Codes

  • 200 OK — update succeeded; returns updated configuration

  • 400 Bad Request — malformed JSON, invalid patch structure, or multiple operations targeting the same field

  • 409 Conflict — expectedHash mismatch (bypassed when force: true)

  • 422 Unprocessable Entity — patch introduces invalid keys/unsupported kinds for this Cassandra version

  • 500 Internal Server Error — unable to persist configuration locally

  • 503 Service Unavailable — ConfigurationProvider unavailable and failure_policy=FAIL (bypassed when force: true); or Cassandra version cannot be determined and patch modifies cassandraYaml

Sample Input

This request updates memtable_flush_writers from 8 to 16 and appends a new -Xmx4G entry to extraJvmOpts. The expectedHash must match the hash from the most recent GET to prevent lost updates.

{
  "expectedHash": "sha256:5f4dcc3b5aa765d61d8327deb882cf99e3f0f0e6a0d8b3f4a1c2e5f6b7a8c9d0",
  "force": false,
  "patch": [
    {
      "op": "add",
      "path": "/configuration/cassandraYaml/memtable_flush_writers",
      "value": 16
    },
    {
      "op": "add",
      "path": "/configuration/extraJvmOpts/-",
      "value": "-Xmx4G"
    }
  ]
}


Sample Response

The response reflects the updated configuration after the patch has been applied. Note that memtable_flush_writers is now 16 and -Xmx4G has been appended to extraJvmOpts. A new hash is returned for subsequent updates.

{
  "hash": "sha256:9b74c9897bac770ffc029102a200c5de4f3d1e8d3a4c8b7f9e6a1b2c3d4e5f6a",
  "lastModified": "2026-02-20T15:04:22Z",
  "configuration": {
    "cassandraYaml": {
      "cluster_name": "my_cluster",
      "memtable_flush_writers": 16,
      "storage_compatibility_mode": "CASSANDRA_5"
    },
    "extraJvmOpts": [
      "-Dcassandra.available_processors=8",
      "-Dcassandra.ring_delay_ms=60000",
      "-Xmx4G"
    ]
  }
}


Remove Operation Example

This example shows removing a configuration key that exists in both the overlay and the base template. After removal, the effective configuration falls back to the template value.

Assume the base template defines memtable_flush_writers: 4 and the overlay currently sets it to 16. The current effective configuration shows memtable_flush_writers: 16.

{
  "expectedHash": "sha256:9b74c9897bac770ffc029102a200c5de4f3d1e8d3a4c8b7f9e6a1b2c3d4e5f6a",
  "patch": [
    {
      "op": "remove",
      "path": "/configuration/cassandraYaml/memtable_flush_writers"
    }
  ]
}

 

Response:

{
  "hash": "sha256:2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d",
  "lastModified": "2026-02-20T15:10:45Z",
  "configuration": {
    "cassandraYaml": {
      "cluster_name": "my_cluster",
      "memtable_flush_writers": 4,
      "storage_compatibility_mode": "CASSANDRA_5"
    },
    "extraJvmOpts": [
      "-Dcassandra.available_processors=8",
      "-Dcassandra.ring_delay_ms=60000",
      "-Xmx4G"
    ]
  }
}

Note that memtable_flush_writers is now 4 (the template value), not 16 (the removed overlay value).

POST /api/v1/cassandra/configuration/reloadConfig

Refresh the effective configuration from both base templates and the ConfigurationProvider without restarting Sidecar. For each instance, re-reads the template file from sidecar.yaml, fetches the latest overlay from the ConfigurationProvider, merges to compute updated effective configuration, validates against version-aware schema, and materializes artifacts. If validation fails for an instance, its effective configuration remains unchanged. Running Cassandra instances are not restarted and continue with their current configuration until the next lifecycle restart.

This endpoint is useful when templates have been modified in sidecar.yaml or when overlays have been changed externally through the ConfigurationProvider (e.g., via a centralized configuration system).

Response Codes

  • 200 OK — all instances reloaded successfully
  • 207 Multi-Status — partial success
  • 500 Internal Server Error — unable to read templates or write to store
  • 503 Service Unavailable — provider unavailable and failure_policy=FAIL

Sample Response

{
  "status": "partial",
  "instances": [
    {
      "instanceId": 1,
      "status": "reloaded",
      "hash": "sha256:abc123...",
      "lastModified": "2026-03-18T10:15:30Z"
    },
    {
      "instanceId": 2,
      "status": "failed",
      "error": "Validation failed: unknown key 'invalid_setting' for Cassandra 5.0"
    }
  ]
}

Configuration Provider

A new pluggable abstraction ConfigurationProvider will be added to Sidecar. The default implementation, DefaultConfigurationProvider, stores configuration overlays as JSON files in the configuration store directory. Operators can implement custom providers to integrate with centralized configuration infrastructure (etcd, Consul, custom REST APIs, database-backed services).

One example of a custom ConfigurationProvider implementation would be an HTTPConfigurationProvider that fetches configuration from an HTTP server. However, this implementation will not be provided as part of this CEP. Operators can implement custom providers based on their centralized configuration infrastructure.

The following new public interfaces will be added.

CassandraConfigurationOverlay.java
/**
 * Represents a configuration overlay - a sparse set of configuration values
 * that overwrite template values or add new configuration attributes.
 */
public class CassandraConfigurationOverlay
{
    /**
     * The cassandra.yaml overlay represented as sparse JSON.
     */
    JsonObject cassandraYaml();

    /**
     * Extra JVM options overlay to be appended to the Cassandra JVM startup command.
     */
    List<String> extraJvmOpts();
}


ConfigurationOverlaySnapshot.java
/**
 * Represents a snapshot of a configuration overlay stored in the ConfigurationProvider.
 */
public class ConfigurationOverlaySnapshot
{
    /**
     * A sha256 checksum of the overlay contents (not the effective configuration).
     * This is the overlay hash used for optimistic concurrency control.
     * (e.g. sha256:9b74c9897bac770ffc029102a200c5de4f3d1e8d3a4c8b7f9e6a1b2c3d4e5f6a)
     */
    String getHash();

    /**
     * The last time the overlay was modified (e.g. 2026-02-20T15:04:22Z)
     */
    DateTime getLastModified();

    /**
     * The configuration overlay (sparse)
     */
    CassandraConfigurationOverlay getConfiguration();
}
ConfigurationProvider.java
/**
 * Provides storage and retrieval of configuration overlays for Cassandra instances.
 * The provider stores sparse overlays, not complete effective configurations.
 */
public interface ConfigurationProvider
{
    /**
     * Retrieve the most recent configuration overlay snapshot for the given instance.
     * The returned overlay is cached locally by Sidecar (along with its hash).
     *
     * @param instance Cassandra instance metadata
     * @return the configuration overlay snapshot
     */
    ConfigurationOverlaySnapshot getConfiguration(InstanceMetadata instance);

    /**
     * Apply a set of updates to the configuration overlay for the given instance.
     * This modifies the overlay stored in the provider.
     *
     * @param instance Cassandra instance metadata
     * @param originalHash the overlay hash from the most recent getConfiguration call;
     * @param updates the updates to apply to the overlay
     * @throws ConcurrentModificationException if it does not match the current overlay hash
     * @return the updated configuration overlay snapshot
     */
    ConfigurationOverlaySnapshot patchConfiguration(InstanceMetadata instance, String originalHash, Map<String, JsonNode> updates);
}

Operational Guide

This section describes the expected workflow for using this feature in practice.

Bootstrapping

Before Sidecar can manage configuration, a base template for cassandra.yaml must be provided. There are two bootstrapping models depending on how the cluster is managed:

Per-instance templates: An external configuration manager provisions a dedicated base template for each instance with node-specific settings already defined (e.g., listen_address, data_file_directories, commitlog_directory). Each instance is configured in sidecar.yaml with its own template path. This model works well when an existing configuration management system already handles per-node customization.

Single shared template: A single stock cassandra.yaml template is used for all instances, containing only the common cluster settings. Node-specific settings are then applied per instance via the PATCH API, either manually or through an external orchestrator. Since immutable blocklisted settings (e.g., cluster_name, num_tokens) can only be set when absent from the template and overlay, the initial PATCH can establish these values during first-time setup.

In both models, bootstrapping the base template is performed outside of Sidecar — either by an orchestrator, configuration management tool, or manual setup.

Updating Configuration

To update a setting, the operator retrieves the current effective configuration via GET (which includes the hash), then submits a PATCH with the expectedHash and the desired changes. The updated configuration is persisted but does not take effect until the instance is restarted via the lifecycle API. This gives the operator a window to verify or correct the change before it impacts the running process.

If a bad configuration is pushed, the operator can submit another PATCH to fix or remove the invalid setting before restarting. If the node was restarted and fails to start due to the bad configuration, the operator can still PATCH through Sidecar (Sidecar does not depend on Cassandra being running) and retry the start.

Backup and Recovery

When using the DefaultConfigurationProvider, the overlay.json files in the configuration store are the authoritative source of overlay data. It is recommended to take periodic backups of the configuration store directory to allow recovery in case of filesystem corruption or accidental deletion. The base templates are managed externally and should be backed up as part of the operator's existing configuration management practices.

For custom providers backed by remote systems (etcd, Consul, etc.), the overlay is stored remotely and the local overlay.json is a cache. Backup and recovery is handled by the remote system's own durability guarantees.

Change Tracking

This CEP does not provide built-in configuration history or audit logging. Operators who need revision control can implement a custom ConfigurationProvider that integrates with a version control or audit system, or manage their base templates and overlay changes through an external revision control system.

Compatibility, Deprecation, and Migration Plan

Backward Compatibility: All changes are additive and introduce no breaking changes to existing Sidecar APIs. The configuration management feature will be disabled by default (via configuration_management.enabled: false in sidecar.yaml), ensuring existing deployments continue to function without modification.

Cassandra Impact: These changes are implemented entirely in Cassandra Sidecar and have no effect on Cassandra itself. Cassandra users who do not use Sidecar are unaffected.

Migration Path: Operators who wish to adopt configuration management can enable it by setting configuration_management.enabled: true and providing a base template for cassandra.yaml. Sidecar will initialize the configuration store from the template on startup. Existing Cassandra instances can continue running while the configuration store is initialized; configuration changes will only take effect on the next restart via the lifecycle API.

Version-Aware Schema: Configuration schemas for new Cassandra versions will be added to Sidecar as those versions are released. Older Sidecar versions will not have schemas for newer Cassandra versions and will reject configuration updates for unsupported versions with a 422 error.

Rollback: If configuration management is disabled after being enabled, Sidecar will stop managing configuration artifacts but will not delete the local configuration store. Operators can manually revert to their previous configuration management approach.

Test Plan

Unit tests will cover individual components such as schema validation, configuration merging, and JSON Patch application.

Integration tests will verify end-to-end behavior with a running Cassandra instance managed by Sidecar. The following high-level integration test scenarios are planned:

  • Initialization — Verify that the configuration store is correctly bootstrapped from templates on Sidecar startup, fetching the overlay from the ConfigurationProvider.
  • GET retrieval — Verify that the GET API returns the effective configuration, including both cassandraYaml and extraJvmOpts, by fetching the overlay from the ConfigurationProvider.
  • PATCH update with version-aware validation — Verify that valid patches are accepted and persisted, that patches introducing unknown cassandraYaml keys for the instance's Cassandra version are rejected with 422, that skipValidation: true allows unknown keys to be persisted, and that extraJvmOpts values bypass schema validation.
  • PATCH update with complex settings validation — Verify that complex nested cassandraYaml settings (e.g., memtable) are validated and replaced as a whole — partial updates to fields within a complex setting are rejected with 422. Patches that replace an entire complex setting with a structurally valid value are accepted, and type mismatches (e.g., scalar where a map is expected) are rejected with 422.
  • PATCH with duplicate field updates — Verify that a PATCH containing multiple operations targeting the same field path is rejected with 400 Bad Request.

  • Immutable settings enforcement — Verify that PATCH operations targeting blocklisted keys (cluster_name, partitioner, num_tokens, initial_token) are rejected with 422 when the key exists in the template or overlay, and accepted when the key is absent from both (initial setup). Verify that skipValidation: true does not bypass the immutable check.

  • Optimistic concurrency — Verify that a PATCH with a stale expectedHash is rejected with 409.
  • Failure policy enforcement — Verify that each failure policy (CACHED_READ_WRITE, CACHED_READ_ONLY, FAIL) produces the correct behavior for retrieval, update, and lifecycle start operations when the ConfigurationProvider is unreachable.
  • Lifecycle start applies configuration — Verify that starting a Cassandra instance through the lifecycle API (PUT /api/v1/cassandra/lifecycle with payload {"state": "start"}) causes the ProcessLifecycleProvider to copy the managed cassandra.yaml from the configuration store to the $CASSANDRA_CONF directory and append extraJvmOpts to the JVM startup command, and that the instance starts successfully with the applied configuration.
  • Configuration update followed by restart — Verify the full round-trip: PATCH a configuration change, restart the instance via the lifecycle API, and confirm the running instance reflects the updated settings.

Rejected Alternatives

We discarded storing the configuration in Cassandra itself since the configuration must be available for modification before the Cassandra process is started so this would present a bootstrapping problem.

Future Work

Automatic Configuration Reload via Polling: This CEP requires operators to explicitly call the reloadConfig endpoint to refresh configuration from external providers. When using custom ConfigurationProvider implementations that integrate with centralized configuration systems (etcd, Consul, etc.), the local cached configuration can become stale if the external system is updated without calling reloadConfig. A future enhancement could add configurable polling of the ConfigurationProvider to automatically detect and reload configuration changes from external systems, eliminating the need for manual intervention and ensuring Sidecar stays synchronized with centralized configuration stores.

Configuration Drift Detection: This CEP does not provide a way to determine whether a running Cassandra instance is operating with the latest persisted configuration or if a restart is needed to pick up pending changes. A future enhancement could introduce a GET /api/v1/cassandra/configuration/status endpoint that compares the effective persisted configuration against the configuration loaded by the running process, reporting whether the instance is up-to-date or has pending changes awaiting restart. This would give operators visibility into configuration drift across their fleet without having to build custom monitoring on top of the existing APIs.

Configuration Change Tracking: This CEP does not maintain a history of configuration changes. A future enhancement could introduce change tracking in the ConfigurationProvider, storing previous overlay versions to enable rollback to a known-good configuration or roll forward after a reverted change. This would allow operators to recover from bad configuration changes without relying on external revision control systems.