2. Supports cross-DC joins by using Paxos to detect duplicates, etc.

Apache Apex (incubating)

Pipeline processing architecture, can be used for real-time and batch processing in unified architecture.
Architected for scalability, low-latency processing, high availability, operability.
Stateful fault tolerance (checkpoints operator state without user having to write code for it).
Runs natively on YARN and HDFS, local mode for development.
Rich library of pre-built operators (Malhar) with many adapters for message buses, databases, file systems etc.
Supports Kafka as source and sink (at any point in the topology), connector with offset management for exactly once semantics / idempotency.

Resource Manager Frameworks

...