Apache Pegasus 2.1.0 is a feature release. The change-list is summarized here: https://github.com/apache/incubator-pegasus/issues/577.

Upgrading Notes

  • 2.1.0 is not compatible in storage format with previous versions. ReplicaServers upgraded to this version can't rollback to any previous releases.

  • 2.1.0 can only be upgraded from 2.0.0. ReplicaServers with version 1.x should upgrade to 2.0.0 firstly before upgrading to 2.1.0.

New Features

Bulk Load

Bulk Load provides a new way to write massive data into a Pegasus table. Using bulk load ingestion eliminates the RocksDB compaction and improves the stability of the online Pegasus cluster. See this document for more details: https://pegasus.apache.org/2020/02/18/bulk-load-design.html

Optimizations and improvements

Write latency optimization

  • Added a new thread pool to execute shared-log write-callback. This optimization significantly reduces the write-latency spike caused by RocksDB, which usually resides in isolated disks from shared-log. It was because they shared the same thread pools that shared-log latency was negatively impacted by RocksDB compaction. Now while the latency spike still exists, it's reduced by about 5-times in our experiments. See XiaoMi/rdsn#568

  • Pegasus ReplicaServer now supports the rate-limiting of RocksDB compaction. It prevents huge disk I/O spike and optimizes read/write latencies in write-intensive workload. See #543

Clear modifications to RocksDB

Pegasus used a modified version of RocksDB, which contained some disk-incompatible changes. In this version, we reduced most of the changes we made, moving towards the goal where finally Pegasus could use the official RocksDB. See #535 #556

Fixed Issues

  • Fixed a memory leak in the network UDP module. Since UDP is by default used in failure detection between MetaServer and ReplicaServer, this issue caused the memory usage of MetaServer to increase over time. See XiaoMi/rdsn#477

Known Issues

  • We plan to add a 2.0.0-compatible version only used for an emergency downgrade when upgrading to 2.1.0 meets serious availability problems.
  • This release may fail to compile with gcc-9. Please use a compiler <=gcc-7.  #642

  • No labels