...

• By default, use the local memory and disk to ensure high shuffle performance if the local storage space is sufficient.

• Improve shuffle stability for large-scale Batch jobs, by using remote storage as a supplement to avoid job failure when the local disk space is insufficient.

Public Interfaces

1. The config option is to specify the home dir of the remote storageThe config option is to specify the home directory of the remote storage.

   1. Key: taskmanager.network.hybrid-shuffle.remote.path

   2. Default: no default value

   3. Description: The base home path of remote storage to store shuffle data. If the option is configured, Hybrid Shuffle will use the remote storage path as a supplement to the local disks. If not configured, the remote storage will not be used.

2. The config option is to specify the minimum reserved space ratio per local disk.

   1. Key: taskmanager.network.hybrid-shuffle.local-disk.min-reserve-space-fraction

   2. Default: 5%

   3. Description: The minimum reserved space fraction per local disk when using Hybrid Shuffle. When using a local disk to store shuffle data, the local disk space may be exhausted if it is used without any limit, leading to job failures. This option controls the minimum reserved disk space fraction which cannot be used to store the shuffle data. When the left available disk space fraction reaches this limit, the new arriving data will be written to the remote storage, if the remote storage is not configured, an exception will be thrown.

...