...

• CDC utility will be started and automatically restarted in the case of failure by the OS or some external tools to provide stable change event processing.
• CDC feature may be used for the deployment that has WAL only.
• At the start of the CDC first consumed event will be the first event available in the WAL archive.
• The lag between the record change and CDC consumer notification will depend on segment archiving timeout and requires additional configuration from the user.
• CDC failover depends on the WAL archive segment count. If the CDC application will be down a relatively long time it possible that Ignite deletes certain archive segments,
  therefore consumer can't continue to receive changed records and must restart from the existing segments.

Online (real-time) CDC

In case of using separate process for capturing data changes from WAL archives makes the lag between CDC event happens and consumer notified about it is relatively big. It's proposed to provide opportunity to capture data and notify consumers directly from Ignite process. It helps minimize the lag by cost of additional memory usage.

User paths

Enable OnlineCDC on cluster:

1. Configure Ignite to enable OnlineCDC.
2. Configure ignite-cdc process
3. Start Ignite cluster in ACTIVE_READ_ONLY mode.
4. Start background process ignite-cdc in BACKUP mode.
1. ./ignite-cdc.sh cdc-config.xml
5. Move Ignite cluster to ACTIVE state.
   

Note, that ignite-cdc.sh can be run in 2 modes - BACKUP, ACTIVE:

1. BACKUP is used as backup process for OnlineCDC, and then such process may fetch CDC configuration from IgniteConfiguration. Case is async replication between master and stand-by clusters.
2. ACTIVE is used as independent process that doesn’t rely on OnlineCDC, has its own configuration. Case is filling a cold data lake.
   

Ignite node restart after failure:

...

Stop OnlineCDC and use ignite-cdc instead:

...

Stop both CDC - Online and ignite-cdc:

...

Restart OnlineCDC after buffer overflow:

1. Try to switch Ignite from BACKGROUND to ONLINE mode:
   1. ./control.sh --cdc online --try-switch-to-online
2. Command will return immediately, but doesn't guarantee success of the switch. User should check logs and metrics here.

User interface

Ignite

1. IgniteConfiguration#OnlineCdcConfiguration - CdcConsumer, keepBinary.
2. DataStorageConfiguration#onlineCdcBufSize - by default (walSegments * walSegmentSize). it’s now 640 MB by default.
   1. All non-archived segments are fitted in memory. If OnlineCDC requires more space than it, it looks like ordinary CDC process should be used instead.
3. DataRegionConfiguration#cdcMode - BACKGROUND, ONLINE (default is BACKGROUND)
   1. BACKGROUND - make hard links of archived segments into cdc directory, that is watched by the background ignite-cdc process.
   2. ONLINE - OnlineCDC enabled + still do BACKGROUND mode job.
4. Logs: 
   1. initialization (amount of records read during the restore)
   2. failure 
   3. buffer is full
   4. switch from BACKGROUND to ONLINE mode.
5. Metrics: 
   1. ordinary cdc metrics (count of wal segments, wal entries)
   2. current buffer size
   3. status of CDC (on/off)
   4. last committed WALPointer
   5. lag between buffer and WAL archive (segments)
   6. lag between WAL and CDC consumer (milliseconds).

ignite-cdc

1. CdcConfiguration#mode - ACTIVE, BACKUP (default ACTIVE if OnlineCDC is not configured, and BACKUP otherwise)
2. Logs: 
   1. clearing cdc dir, switch state.
3. Metrics:
   1. current state.

control.sh

1. CdcOnline subcommand
   1. ./control.sh --cdc online [ --enable ] [ --disable ] [ --try-switch-to-online ] 

Segments

Note, there is a confusion of using “segment” word:

1. WAL segments are represented as numerated files. Size of WAL segments is configured with DataStorageConfiguration#walSegmentSize.
2. ReadSegment is a slice of the mmap WAL segment. It contains WAL records to sync with the actual file. Size of the segment differs from time to time and its maximum can be configured with DataStorageConfiguration#walBuffSize.

Initialization

On Ignite start during memory restore (in the main thread):

1. If DataRegionConfiguration#cdcMode == ONLINE, then create CdcProcessor.
2. CdcProcessor read from the Metastorage the last persisted CdcConsumerState.
1. CdcState#enabled is false then skip initialization.
2. If CdcState == null then initialize.
3. Initialization - collect logical updates from the CdcState#committedPtr until the end of WAL. See GridCacheDatabaseSharedManager#performBinaryMemoryRestore.

Online capturing of WALRecords

Entrypoint for WALRecords to be captured by CDC. Options are:

1. During read of SegmentedRingByteBuffer after fsync is invoked. It is a multi-producer/single-consumer data structure, then the only place to built-in is read operations (invoked at the moment of fsync).
   
   1. + Relying on the consumer workflow we can guarantee order of events.
   2. + Consumer is a background thread, capturing records doesn't affect performance of transactional threads
   3. - No opportunity to filter physical records at the entrypoint (might waste the buffer space). Will filter them before actual sending.
   4. - The consumer is triggered by a schedule - every 500ms by default.
   5. - Logic has some differences depending on the WAL settings (mmap true/false, FULL_SYNC) 
2. Capturing in FileWriteAheadLogManager#log(WALRecord).
   1. + Capture logical records only
   2. + Common logic for all WAL settings  
   3. - Captures record in buffer in transactional threads - might affect performance
   4. - CDC process must sort events by WALPointer by self - maintain concurrent ordering data structure, and implementing waiting for closing WAL gaps before sending.
   5. - Send events before they actually flushed in local Ignite node - lead to inconsistency between main and stand-by clusters.

First option is proposed to use.

CdcWorker

CdcWorker is a thread responsible for collecting WAL records, transforming them to CdcEvents and submitting them to a CdcConsumer. The worker collects records in the queue.

Capturing from the buffer (wal-sync-thread):

...

Otherwise, stop online CDC:

...

Also, it's possible to stop Online CDC using command in control.sh. In this case it also writes StopOnlineCdcRecord.

Body loop (cdc-worker-thread):

1. Checks metadata (mappings, binary_meta, caches - can check inside Ignite, not reading files), prepare updates if any.
2. Polls the Queue, transforms ReadSegment data to Iterator<CdcEvent>, pushes them to CdcConsumer.
3. If CdcConsumer#onEvents returns true:
   1. Persists CdcConsumerState.
   2. Write OnlineCdcRecord record to WAL with the WALPointer.
4. Optimization: transform segment buffers to CdcEvents in background (to reduce the buffer usage). CdcConsumer should be async then?

Try switch BACKGROUND to ONLINE mode:

...

OnlineCdcRecord extends WALRecord {
	private WALPointer last;
}

StopOnlineCdcRecord extends WALRecord {
	private WALPointer last;
}

TryStartOnlineCdcRecord extends WALRecord {
	
}

ignite-cdc in BACKUP mode

1. Parses WAL records, looking for OnlineCdcRecord and StopOnlineCdcRecord
2. For OnlineCdcRecord - clears obsolete links from CDC directory
3. For StopOnlineCdcRecord - switch to ACTIVE mode, start capturing from the last WALPointer (from previous OnlineCdcRecord).

ignite-cdc in ACTIVE mode

1. Capturing WAL records
2. Looking for TryStartOnlineCdcRecord - after reaching it, persist CdcConsumerState locally, switch to BACKUP mode.

Meta Storage

1. Online CDC status - ON / OFF
2. Committed pointer (confirmed by CdcConsumer).

class CdcWorker {
	private final CdcConsumer consumer;
	
	private final long checkFreq;
	
	// Invoked in wal-sync-thread.
	public void offer(ReadSegment seg) {
		// Check capacity, adding segment to the queue.
	} 

	// online-cdc-thread
	public void body() {
		// Polling queue, push to CdcConsumer, writing CdcState to MetaStorage.
	}
}

Discussion Links

http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-IEP-59-CDC-Capture-Data-Change-tc49677.html

...