Ecosystem
Skip to end of metadata
Go to start of metadata

Here is a list of tools we have been told about that integrate with Kafka outside the main distribution. We haven't tried them all, so they may not work!

Clients, of course, are listed separately here.

Distributions & Packaging

Stream Processing

  • Storm - A stream-processing framework.
  • Samza - A YARN-based stream processing framework.
  • Storm Spout - Consume messages from Kafka and emit as Storm tuples
  • Kafka-Storm - Kafka 0.8, Storm 0.9, Avro integration
  • SparkStreaming - Kafka reciever supports Kafka 0.8 and above 

Hadoop Integration

  • Camus - LinkedIn's Kafka=>HDFS pipeline. This one is used for all data at LinkedIn, and works great.
  • Kafka Hadoop Loader A different take on Hadoop loading functionality from what is included in the main distribution.
  • Flume - Contains Kafka Source (consumer) and Sink (producer)
  • KaBoom - A high-performance HDFS data loader

Search and Query

  • ElasticSearchThis project, Kafka Standalone Consumer will read the messages from Kafka, processes and index them in ElasticSearch.
  • Presto - The Presto Kafka connector allows you to query Kafka in SQL using Presto.
  • Hive - Hive SerDe that allows querying Kafka (Avro only for now) using Hive SQL

Management Consoles

  • Kafka ManagerA tool for managing Apache Kafka.
  • kafkat - Simplified command-line administration for Kafka brokers.
  • Kafka Web Console- Displays information about your Kafka cluster including which nodes are up and what topics they host data for.
  • Kafka Offset Monitor - Displays the state of all consumers and how far behind the head of the stream they are.
  • Capillary Displays the state and deltas of Kafka-based Apache Storm topologies. Supports Kafka >= 0.8. It also provides an API for fetching this information for monitoring purposes.

AWS Integration

Logging

Flume - Kafka plugins

Metrics

Packing and Deployment

Kafka Camel Integration

Misc.

Labels
  • No labels