TubeMQ is a distributed messaging queue (MQ) system developed by Tencent Big Data since 2013. It focuses on high-performance storage and transmission of massive data in big data scenarios.After nearly seven years of massive data precipitation, TubeMQ has certain advantages in production practice (stability + performance) and low cost compared to many open source MQ projects.
TubeMQ is suitable for high concurrency, massive data and tolerates a small amount of data loss scenarios under abnormal conditions, such as massive log collection, indicator statistics and monitoring, etc.
TubeMQ does not support highly reliable data transmission services yet. It could be on a future project roadmap, as many other MQs. but not today.
Just like other message queue systems, TubeMQ is built on the publish-subscribe pattern, aka pub-sub.
In this pattern, producers publish messages to topics while consumers subscribe to those topics. After incoming messages get proceeded, consumers send an acknowledgement back to producer. Once a subscription has been created, all messages will be tracked and retained by TubeMQ, even if the consumer go offline for some reasons. Retained messages will be discarded only when a consumer acknowledges that they've been successfully processed.
The overall architecture of TubeMQ is just like following:
Portal is responsible for interact with user and admin system which include two parts: API and web portal.
Master is controller of the cluster, which include one or multiple master node(s) which is responsible for managing state of cluster, resource scheduling, authentication check and maintaining of metadata. As a reliable system, TubeMQ provides HA solution for master node.
Broker is responsible for data store which include a cluster of broker nodes. Every broker node is managing a set of topics, include: append, delete, update, query of topic information. In TubeMQ, these brokers can be horizontal scaled and can be very large size for massive data case.
Client is responsible for producing and consuming messages. When a pub-sub topic get setup, we can support two ways (push and pull) for delivering message from producers to consumers.
Zookeeper is for storing offset of messages which is used to recover topic during some components get failed.
The initial goal will be to move the current codebase in github’s repository under Tencent account to Apache and integrate with the Apache development process and infrastructure.
A primary goal of incubation will be to grow and diversify the TubeMQ community. We are well aware that the project community is largely comprised of individuals from a single company. We aim to change that during incubation.
As previously mentioned, TubeMQ is under active development at Tencent, and is being used in processing large volumes of data for most services and products.
We value meritocracy and we understand that it is the basis for an open community that encourages multiple companies and individuals to contribute and be invested in the project’s future. We will encourage and monitor participation and make sure to extend privileges and responsibilities to all contributors.
TubeMQ is currently being used by developers at Tencent and a growing number of users are actively using it in production environments. TubeMQ has received contributions from developers working outside of Tencent since it was open sourced on github in September 2019 By bringing TubeMQ to Apache we aim to assure current and future contributors that the TubeMQ community is neutral, meritocratic, and open, in order to broaden and diversity the user and developer community.
TubeMQ was initially developed at Tencent and is under active development. We believe Tencent will be of interest to a broad range of users and developers and that incubating the project at the ASF will help us build a diverse, sustainable community.
TubeMQ utilizes other Apache projects such as Hadoop, HBase and Zookeeper. We anticipate integration with additional Apache projects as the TubeMQ community and interest in the project grows.
Tencent is committed to the future development of TubeMQ and understands that graduation to a TLP, while preferable, is not the only positive outcome of incubation.
Should the TubeMQ project be accepted by the Incubator, the prospective PPMC would be willing to agree to a target incubation period of 2 years or less, knowing that every Incubator project incurs a certain cost in terms of ASF infrastructure and volunteer time.
Five of the initial mentors/committers are Apache Members and Incubator PMC Members. They will work with the other community members to teach them the Apache Way.
The majority of the committers work at Tencent, though we are committed to recruiting and developing additional committers from a wide spectrum of industries and backgrounds. Since being open sourced, many contributors that are outside of Tencent have engaged and begun contributing to the project.
It is expected that Tencent development will occur on both salaried time and on volunteer time, after hours. Most of the initial committers are paid by Tencent to contribute to this project. However, they are all passionate about the project, and we are both confident and hopeful that the project will continue even if no salaried developers contribute to the project.
As mentioned in the Rationale section, TubeMQ utilizes a number of existing Apache projects (Avro, Zookeeper, etc.), and we expect that list to expand as the community grows and diversifies. Any Apache project in the big data space that needs to process data in streaming way would be potentially relevant, such as Flink, Spark Streaming, etc. To provide convenient access to event streams served by TubeMQ, we plan to open source connectors to different stream computation engines. With these connectors, Spark and Flink can read and write data directly from/to topics served by TubeM.
We are applying to the Incubator process because we think it is the next logical step for the TubeMQ project after open-sourcing the code. This proposal is not for the purpose of generating publicity. Rather, we want to make sure to create a very inclusive and meritocratic community, outside the umbrella of a single company. Tencent has a long history of contributing to Apache projects and the TubeMQ developers and contributors understand the implication of making it an Apache project.
The podling may also create a user mailing list, if needed.
The TubeMQ podling would use Apache’s gitbox integration to sync between github and Apache infrastructure. The podling would use github issues and pull requests for community engagement.
The TubeMQ source code in Github is currently licensed under Apache License v2.0 and the copyright is assigned to Tencent. If TubeMQ becomes an Incubator project at the ASF, Tencent will transfer the source code and trademark ownership to the Apache Software Foundation via a Software Grant Agreement.
Not applicable.