Apache Pegasus 2.4.0 Release Notes

Apache Pegasus 2.4.0 Release Note

Apache Pegasus 2.4.0 is a feature list, the change-list is summarized here: https://github.com/apache/incubator-pegasus/issues/1032

New module

Apache Pegasus contains many ecological projects. In the past, they were maintained in different repository. Starting from this version, they will be gradually donated to the official Apache Pegasus repository. Currently, the following projects are donated to Apache Pegasus:

• RDSN:  in the past, rdsn was linked to Apache Pegasus as a sub module of GIT. Today, it has officially become the core module of Apache Pegasus.
• PegasusClient: Pegasus supports multiple clients. Now, the following client projects will be donated to Apache Pegasus: Pegasus-Java-Client, Pegasus-Scala-Client, Pegasus-Golang-Client, Pegasus-Python-Client, Pegasus-NodeJS-Client
• PegasusDocker: Pegasus supports building with docker. In the current version, the official provides dockfile samples of various build environments, and uses githubaction to build corresponding docker images and upload them to DockerHub
• PegasusShell: Pegasus has used C + + to build shell tools. In the latest version, we have built new shell tools using golang, including  AdminCli and for admin and Pegic  for user.

New architecture

In the test, we found that the shared-log engine with a single queue will cause a throughput bottleneck. Thanks to the optimization of random writes by concurrent writes of SSDs, we removed the shared-log written in sequence and only kept the private-log as the WAL. After the test, this will bring about a 15-20% improvement in the performance.

New feature

Support replica count update dynamically

In the past, once a table was created, its replica count could not be changed. The new version supports the function of dynamic change of table replica count. User can increase or decrease the count of a serving table, which is transparent to the foreground.

New batchGet interface

The old `batchGet` interface is only a simple encapsulation of the `get` interface. It does not have the `batch capability`. The new interface optimizes the batch operation. It will aggregate multiple requests according to the partiition-hash rules, and then send the aggregated requests in the unified partition to the corresponding nodes of the server atomically. This will improve the throughput of online writing

Client request limiter

Burst requests from the client will be piled up for the task queue. To avoid this situation, we added queue-controller to limit the task traffic in extreme scenarios.

In the past, Pegasus only controlled the write traffic. In the new version, we also supported the read traffic, which will enhance the stability of the cluster in emergencies.

Jemalloc memory management

Jemalloc is an excellent memory management library. In the past, we only used tcmalloc for memory management. In the new version, we also support jemalloc, the detail bench result see Jemalloc Performance

Multi architecture support

We have added support for MacOS and aarch64 systems, which will improve Pegasus' cross platform capability

Client Feature

The Java client adds a table creation and deletion interface, and supports `batchGetbypartition` to adapt the batchGet interface of the server

Go client adapts to RPC interfaces such as bulkload, compact and disk-add on the server side

AdminCli supports node-migrator, node-capacity-balance, table-migrator, table-partition-split command  and other functions.

Feature enhancement

Bulkload

The bulkload has added a lot of optimizations for performance, including using direct-io to perform data download, repair duplicate-check, optimize ingest-task strategy and other features to avoid the impact of IO-load on request latency during bulkload.

Bukload supports concurrent tasks of multiple tables meanwhile. However, it should be noted that due to the low-level speed limit, concurrency only allows multiple tables to queue to execute tasks, and does not improve the overall task efficiency unser same rate

Duplication

Duplication removes the dependence on remote file systems and supports checkpoint file transmission between clusters

Duplication supports batch-sending of log files to improve the synchronization efficiency of incremental data

The new duplication, when the user creates the task, the server will first copy the checkpoint files across the cluster, and then automatically synchronize the incremental logs, greatly simplifying the previous process

Other Important

PerfCounter: In the monitoring system, we optimized the CPU cache performance problems caused by false-share issue, and rebuilt the monitoring point system

ManualCompaction: We have added a control interface for ManualCompaction to the latest version so that users can easily trigger a ManualCompaction task and query the current progress in real time

NFS in Learn: NFS is a module for checkpoint transfer between nodes. In the past, the system has been affected by checkpoint transfer. In the new version, we have provided disk-level fine-grained rate control to reduce the impact of checkpoint transfer.

Link-tracking: The new link tracker supports data upload for monitoring systems to obtain link delay statistics

Environment Variables: We changed the `deny_write` environment, now it can also turn on read-deny at the same time and provide different response information to the client

Cold backup: backup speed will affect request latency, new version we provide dynamic configuration for HDFS upload speed during backup

RocksdB log size limit: sometimes rocksdb logs take up more space, which is limited by the new version

MetaServer: Supports Host domain name configuration

Bug fix

In the latest version, we focused on fix the following problems:

Server

• Node crash caused by ASIO's thread safety problem
• IO amplification caused by improper handling of RPC body
• Data overflow caused by unreasonable type declaration in AIO module
• Unexpected error when replica is closed

client

• The batchMultiGet interface of the Java client cannot obtain data completely
• The go client cannot access when the server enable the request-drop configuration
• The go client cannot recovery when encounter the `ERR_INVALID_STATE` and so on

Performance

In this benchmark, we use the new machine, for the result is more reasonable, we re-run the Pegasus Server 2.3：

• Machine parameters: DDR4 16G * 8 | Intel Silver4210*2 2.20Ghz/3.20Ghz | SSD 480G * 8 SATA
• Cluster Server: 3 * MetaServerNode 5 * ReplicaServerNode
• YCSB Client: 3 * ClientNode
• Request Length: 1KB(set/get)

Pegasus Server 2.3

Pegasus Server 2.4

Known issues

We have upgraded the ZK client version to 3.7. When the ZK version of the server is smaller than this version, the connection may be timeout.

When configuring periodic manual-compaction tasks with environment variables, there may be a calculation error and cause immediate start.