Abstract

Uniffle is a high performance, general purpose Remote Shuffle Service for distributed compute engines like Apache Spark, Apache Hadoop MapReduce, Apache Flink and so on. It has been open-sourced by Tencent since last year. We are aiming to make Uniffle a universal shuffle service for distributed compute engines.

Proposal

Uniffle provides a high performance, push-based shuffle service for different distributed compute engines. With its client being plugged into compute engines, it provides the ability to push shuffle data to centralized shuffle service, without needing local storage. 

Changing the shuffle style from “local file pull-like style”  to “remote block push-like style” will bring in several advantages like supporting disaggregated compute storage deployment, super large shuffle jobs, and high elasticity.

Background

Shuffle is the key part for a distributed compute engine to exchange the data between distributed tasks, the performance and stability of shuffle will directly affect the whole job. 

Taking Spark’s built-in shuffle as an example, the current “local file pull-like shuffle style” has several limitations:

  1. Current shuffle is hard to support super large workloads, especially in a high load environment, the major problem is IO problem (random disk IO issue, network congestion and timeout).
  2. Current shuffle is hard to deploy on the disaggregated compute storage environment, as disk capacity is quite limited on compute nodes.
  3. The constraint of storing shuffle data locally makes it hard to scale elastically.

Taking Spark on k8s deployment as a case, current Spark on k8s deployment requires local disks to store shuffle data, which makes it hard to deploy in a hybrid environment, not to say serverless.

Uniffle - a Remote Shuffle Service - delivers a centralized shuffle service to solve the above problems.

Rationale

Uniffle’s key idea is brought from Salfish shuffle, it has several key design goals:

  1. High performance. Uniffle’s performance is close enough to local file based shuffle style for small workloads. For large workloads, it is far better than the current shuffle style.
  2. Fault tolerance. Uniffle provides high availability for Coordinated nodes, and failover for Shuffle nodes.
  3. Pluggable. Uniffle is highly pluggable, which could be suited to different compute engines, different backend storages, and different wire-protocols.

Remote Shuffle Service is the key technology for enterprises to build big data platforms, to expand big data applications to disaggregated, online-offline hybrid environments.

The implementation of Remote Shuffle Service -  “Uniffle”  - is heavily adopted in Tencent, and shows its advantages in production. Other enterprises also adopted or prepared to adopt Uniffle in their environments. It is obvious that such technology will be vastly adopted.

Initial Goals

The initial goals are mainly focused on building the project and community:

  1. Move existing codebase, website, documentation, and mailing lists to Apache-hosted infrastructure.
  2. Integrate with the Apache development process and infrastructure and move our code review, build, and testing workflows in the context of the ASF.
  3. Grow and diversify the community.

In the meanwhile, we will continue to iterate the project:

  1. High performance: continue to optimize the internal RPC framework and I/O stack.
  2. High availability: quorum write and read, fast failover and so on.
  3. Pluggable: support different compute engines, different storage backends and others.

Current Status

Meritocracy:

Uniffle was started at Tencent in 2020 with project name "Firestorm", we open sourced the project under Github Tencent in 2021. After the open source, we received lots of issue reports. We think by running with the Apache Way, we will better grow the community and project.

We value meritocracy and we understand that it is the basis for an open community that encourages multiple companies and individuals to contribute and be invested in the project’s future. We will encourage and monitor participation and make sure to extend privileges and responsibilities to all contributors.

Community:

Uniffle is currently being used by developers at Tencent and a growing number of users are actively using it in production environments. It has received contributions from developers out of Tencent since it was open sourced. By bringing Uniffle to Apache, we aim to assure current and future contributors that the community is neutral, meritocratic, and open, in order to broaden and diversify the user and developer community.

Core Developers:

  • Jerry Shao. He is the founder of this project, Apache Member and Spark PMC member, from Tencent.
  • Junjie Ma. He is the developer of this project, Apache Spark/Hive/Hbase contributor, from Tencent.
  • Meng duan. He is the developer of this project, from Tencent.
  • He qi. He is the developer of this project, Apache Spark/Iceberg contributor, from Tencent.
  • Zhifang Li. He is the developer of this project, from Tencent.

Alignment:

Uniffle is built upon Apache Spark and many other Apache projects such as Apache Hadoop, Zookeeper, etc. The codebase of Uniffle is already under Apache License Version 2.0. Meanwhile, our current core developers all have the experience of contributing to various Apache projects. These community connections help us focus on development practices that emphasize community engagement to align us with the ASF path to meritocratic recognition naturally.

Known Risks

Orphaned products

Since Uniffle is vastly adopted in Tencent, developers from Tencent will commit to future development of Uniffle. And we’re continuing to grow our community to attract more users and contributors. So currently the risk of orphaned products is limited.

Inexperience with Open Source:

Many of the Uniffle contributors have experience working on open source projects. They are also active committers and contributors to other Apache projects.

Homogenous Developers:

The majority of the committers work at Tencent, though we are committed to recruiting and developing additional committers from a wide spectrum of industries and backgrounds. Since being open sourced, many contributors that are outside of Tencent have engaged and begun contributing to the project.

Reliance on Salaried Developers:

Salaried engineers have made contributions to the Uniffle project to date, both on their salaried time and on volunteer time. They are all passionate about the project, and we are committed to recruiting additional committees, including non-salaried developers, and aim to diversify the user and contributor base further.

Relationships with Other Apache Products:

Uniffle is closely tied with Apache Spark, Apache Hadoop, Apache Flink, and other commons currently in numerous ways.

Uniffle pluggablely integrates into different distributed compute engines, and replaces their own shuffle mechanisms. It is strongly dependent on the current big data ecosystem, and is a significant complement to the stacks.

A Excessive Fascination with the Apache Brand:

The primary motivation for submitting Uniffle to the ASF is to build a diverse and strong community and to gain stability for long-term development. 

And we also wish to encourage organizations to adopt and contribute without any concerns about ownership or licensing.

Documentation

The documentation is here https://github.com/Tencent/Firestorm/tree/master/docs, with the basic architecture and build instructions.

Initial Source

The Uniffle project is hosted on Github with current project name Firestorm: https://github.com/Tencent/Firestorm

Source and Intellectual Property Submission Plan

The Uniffle source code in Github is currently licensed under Apache License v2.0 and the copyright is assigned to Tencent. If Uniffle becomes an Incubator project at the ASF, Tencent will transfer the source code and trademark ownership to the Apache Software Foundation via a Software Grant Agreement.

External Dependencies:

Apache Licence 2.0

com.fasterxml.jackson.core:jackson-annotations

com.fasterxml.jackson.core:jackson-core

com.fasterxml.jackson.core:jackson-databind

com.google.errorprone:error_prone_annotations

com.google.errorprone:error_prone_annotations

com.google.guava:guava

commons-codec:commons-codec

commons-logging:commons-logging

info.picocli:picocli

io.dropwizard.metrics:metrics-core

io.grpc:grpc-netty-shaded

io.grpc:grpc-protobuf

io.grpc:grpc-stub

io.grpc:grpc-testing

io.netty:netty-all

io.prometheus:simpleclient

io.prometheus:simpleclient_hotspot

io.prometheus:simpleclient_httpserver

io.prometheus:simpleclient_jetty

io.prometheus:simpleclient_servlet

net.jpountz.lz4:lz4

org.apache.commons:commons-lang3

org.apache.hadoop:hadoop-client

org.apache.hadoop:hadoop-common

org.apache.hadoop:hadoop-mapreduce-client-app

org.apache.hadoop:hadoop-mapreduce-client-core

org.apache.hadoop:hadoop-minicluster

org.apache.logging.log4j:log4j-core

org.apache.spark:spark-core_2.11

org.apache.spark:spark-core_2.12

org.apache.spark:spark-sql_2.11

org.apache.spark:spark-sql_2.12

org.codehaus.jackson:jackson-jaxrs

org.codehaus.jackson:jackson-xc

org.roaringbitmap:RoaringBitmap

BSD 3-clause

com.google.protobuf:protobuf-java-util

CPL 1.0 License

com.github.stefanbirkner:system-rules

EPL-1.0 License

junit:junit

MIT License

org.mockito:mockito-core

org.mockito:mockito-inline

org.slf4j:slf4j-api

org.slf4j:slf4j-log4j12

org.slf4j:slf4j-log4j12

Cryptography:

NA

Required Resources

Mailing lists:

The podling may also create a user mailing list, if needed.

Git Repositories:

Issue Tracking:

The Uniffle podling would use Apache’s gitbox integration to sync between github and Apache infrastructure. The podling would use github issues and pull requests for community engagement.

Initial Committers

Tencent

Baidu

Bytedance

Microsoft

Vipshop

Intel

Sponsors

Champion:

Nominated Mentors:

Sponsoring Entity:

We are expecting the Apache Incubator could sponsor this project.

  • No labels