Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This FLIP proposes upgrading the version of FRocksDB in the Flink Project from 6.20.3 to 8.10.0.

RocksDBStateBackend is widely used by Flink users in large state scenarios.The last upgrade of FRocksDB was in version Flink-1.14, which mainly supported features such as support arm platform, deleteRange API, period compaction, etc. It has been a long time since then, and RocksDB has now been released to version 8.x. The main motivation for this upgrade is to leverage the features of higher versions of Rocksdb to make Flink RocksDBStateBackend more powerful. While RocksDB is also continuously optimizing and bug fixing, we hope to keep FRocksDB more or less in sync with RocksDB and upgrade it periodically.

（Although this upgrade almost doesn't change anything in flink code, the main reason for creating this FLIP is because the feature and performance changes it brings may be visible to users）

Main benefits

This section will introduce the main benefits of upgrading RocksDB, including optimizing the performance of Rocksdb rescaling using IngestDB API and other potential optimization points that have not yet been implemented, such as async io and tiered storage etc.

IngestDB

The problem of slow RocksDBStateBackend rescaling has been troubling users for a long time. Such as when the parallelism of a job changes from 2 to 1. When RocksDB performs rescaling recovery, it is necessary to merge the data of two RocksDB into one DB instance. 

In the current implementation, merging data from two RocksDB can only traverse all records of one DB and then insert them into to the other one. To address this issue, we extended RocksDB API to directly merge data from multiple RocksDB instances and support IngestDB Recovery Mode FLINK-31238 - Getting issue details... STATUS in RocksDBStateBackend. In this way, the time for merging DB data has been increased from minutes or even half an hour to a few seconds.

Now the APIs required by IngestDB have been merged into the main branch of RockDB FLINK-33337 - Getting issue details... STATUS , and the implementation of IngestDB has also already been completed https://github.com/apache/flink/pull/24031.(special thanks to Stefan Richter )

However, the feature of IngestDB still cannot be used because FRocksDB has not been upgraded to the corresponding version yet.

We tested the performance of ingestDB with a snapshot size of 1g. The results are as follows:

• In scenarios where parallelism increases(1->2, 2->8), the performance of using IngestDB Mode is the same as before, because during the recovery phase, only deleteRange will be called, which will return very quickly.
• In other scenarios, IngestDB has shown significant improvements compared to previous recovery methods.

Other Potential Optimization Points

In addition to IngestDB, upgrading Rocksdb can also bring some other features that will be very useful for Flink. These features have not yet been used and implemented in Flink, so they are listed here as potential optimization points

AsyncIO & MultiGet

RocksDB supports MultiGet & Async IO in higher versions to improve the performance by over 100% according to different data pipelines which is also required by one of the FLIP-426 changes.Therefore, using Async IO can significantly improve the performance of state batch access and traversal, which are also widely used in Flink. Although further discussion may be needed on how to adapt async io in Flink RocksdbStateBackend, all of these require us to first upgrade FRocksDB as a prerequisite.

Tiered Storage 

In FLIP-423, some limitations of current State management were mentioned, including Local Disk Constraints, Light and Fast Checkpoints, Spiky Resource Usage, Elasticity and Fast Rescaling.

RocksDB now supports Tiered Storage which can assign data temperature when creating the new SST which hints the file system to put the data on the corresponding storage media, so the data in a single DB instance can be placed on different storage media

This feature can also address some of the aforementioned issues. At the same time, RocksDB has also optimized the performance of Tiered Storage, including such as Per Key Placement Compaction and Secondary Cache.

Although FLIP-423 has designed a complete solution ForSt for these scenarios, perhaps RocksDBStateBackend can also directly reuse the existing features of RocksDB for some enhancements at a relatively low cost, and there is no conflict between these two optimization

BlobDB

In the usage scenarios of Flink, the state value may be very large in many scenarios, such as machine learning features or long-term user-defined states. In LSM, this large-value use case can cause a lot of read and write amplification, affecting the performance of RocksDB. Fortunately, RocksDB supports BlobDB to address this issue. The basic idea, which was proposed in the WiscKey paper, is a key value separation: by storing large values in dedicated blob files and storing only small pointers to them in the LSM tree, avoid copying the values over and over again during compaction. This reduces write amplification, which has several potential benefits like improved SSD lifetime, and better write and read performance. Perhaps BlobDB can also be supported in RocksDBStateBackend in future FLIP.

Public Interfaces

No changes 

Proposed Changes

• Flink
  • Update FRocksDB version to 8.10.0 after create official artifacts
• FRockDB
  • In FrocksDB 6 thread-local perf-context is disabled by reverting a specific commit (FLINK-19710). However, this creates conflicts and makes upgrading more difficult. We found that disabling PERF_CONTEXT can improve the performance of statebenchmark by about 5% and it doesn't create any conflicts. So we plan to drop the support of PERF_CONTEXT in the new FRocksDB release.

Compatibility, Deprecation, and Migration Plan

this change doesn't break any existing formats or jobs

Test Plan

For version upgrades, performance testing is very important, so we upgraded RocksDB to version 8.10.0 on the Flink Master branch and compared it with the current version 6.20.3 on StateBenchmark and Nexmark, respectively. The test results are as follows:

Nexmark

We ran the Flink Nexmark benchmark on 4 independent machines. Each machine consists of an AMD EPYC 9Y24 CPU and NVME ssd disk. Flink's cluster consists of 1 JobManager and 8 TaskManagers (each with 1 slot and  with 8 GB of memory). And we selected all queries that use state in Nexmark . From the test results, it appears that the performance of most Queries is similar to the previous throughput.

StateBenchmark

We conducted State Benchmark testing on RocksDB in Flink Speed Center (special thanks to Zakelly Lan and Roman Khachatryan ), and ran three rounds of testing to reduce jitter. From the test results, it appears that the read performance of RocksDB V8 is not much different from V6, but there is a slight regression in write performance

Overall, We still think that compared to the huge benefits brought by the upgraded version, the slight regression in write performance is still acceptable since the write overhead is a very small part of the overall.

Rejected Alternatives

• Upgrade FRocksDB to 8.11.x or 9.X version
  It has been some time since we started upgrading FRocksDB. At that time, the latest release version of RocksDB was 8.10.0, but now RocksDB has released several new versions. However, we still plan to upgrade to 8.10.0 in this upgrade. The main reason is that
  When we upgrade, we need to do a lot of adaptation, testing, and bug fixing work, which requires a lot of time and effort. In later released versions of RocksDB, we did not see such attractive features that are worth repeating these job. In fact, we have also conducted quick benchmark validation based on the latest version 9.X, and its performance is similar to 8.10.0. So we believe there is no need to upgrade to other versions.