Table of Contents |
---|
How The Kafka Project Handles Clients
Starting with the 0.8 release we are maintaining all but the jvm client external to the main code base. The reason for this is that it allows a small group of implementers who know the language of that client to quickly iterate on their code base on their own release cycle. Having these maintained centrally was becoming a bottleneck as the main committers can't hope to know every possible programming language to be able to perform meaningful code review and testing. This lead to a scenario where the committers were attempting to review and test code they didn't understand.
...
If you are aware of other clients not listed here (or are the author of such a client), please add it here. Please also feel free to help fill out information on the features the client supports, level of activity of the project, level of documentation, etc.
For 0.8.x
The 0.8 release changes the protocol fairly substantially. The new protocol is documented here.
Producer Daemon
Bruce is a producer daemon that supports clients in various programming languages. Clients write messages to Bruce's UNIX domain datagram socket in a simple binary format. Bruce then takes full responsibility for reliable message delivery. Monitoring infrastructure can query Bruce's web interface for JSON-based status and data quality reports.
Maintainer: Dave Peterson
License: Apache v.2.0
https://github.com/tagged/bruce
Python
Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.
Maintainer: David Arthur
License: Apache v.2.0
https://github.com/mumrah/kafka-python
Go (AKA golang)
Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.
Maintainer: Shopify
License: MIT
https://github.com/Shopify/sarama
Another Pure Go implementation built to work as like the "Consumer Group" High Level Consumer with more features including plug-able partition ownership and work fan out.
...
C/C++
-------------
Robust high performance C/C++ library with full protocol support
A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.
https://github.com/edenhill/librdkafka
-------------
Native C++ library with protocol support for Metadata, Produce, Fetch, and Offset.
Maintainer: David Tompkins
License: Apache 2.0
https://github.com/adobe-research/libkafka
-------------
C++ Header-only Kafka Client Library using Boost Asio
Maintainer: Daniel Joos
License: MIT
https://github.com/danieljoos/libkafka-asio
-------------
A C++11 asyncronous producer/consumer library for Apache Kafka based on boost asio
Maintainer:
License: Boost Software License - Version 1.0
https://github.com/bitbouncer/csi-kafka
-------------
libasynckafkaclient - C++ based single threaded asynchronous library
Maintainer: Vijay Jadhav
License: 2-clause BSD
https://github.com/GSLabDev/libasynckafkaclient
-------------
https://github.com/quipo/kafka-cpp
Kafka Version: 0.7.x
-------------
Python
-------------
High performance Python client based on Librdkafka with full protocol support.
https://github.com/confluentinc/confluent-kafka-python
Docs: http://docs.confluent.io/current/clients/index.html
-------------
Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP, LZ4, and Snappy compression supported.
http://github.com/dpkp/kafka-python
Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x, 1.0.x
Maintainer: Dana Powers
License: Apache 2.0
-------------
Python driver with full protocol support, Pythonic API, and managed balanced consumer implementation.
https://github.com/Parsely/pykafka
Kafka Version: 0.8.x,0.9.x,0.10.x
Maintainer: Parse.ly
License: Apache 2.0
-------------
Protocol support for Kafka 0.7 in Python. GZip and Snappy compression supported
Kafka Version: 0.7.x
Maintainer: David Arthur
License: Apache v.2.0
https://github.com/mumrah/kafka-python/tree/0.7
-------------
Also:
https://github.com/dsully/pykafka
Kafka Version: 0.7.x
Maintainer:: Dan Sulley, LinkedIn
License: Apache 2.0
-------------
Python client from Disqus:
https://github.com/getsamsa/samsa
Kafka Version: 0.7.x
Maintainer:: Keith Bourgoin, Parse.ly
License: Apache 2.0
-------------
Python client from Urban Airship: https://github.com/urbanairship/pykafkap
Kafka Version: 0.7.x
-------------
Python client from Datadog: https://github.com/datadog/brod
(Producer, Simple Consumer, ZK-Consumer)
Kafka Version: 0.7.x
-------------
Go (AKA golang)
-------------
Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.
Kafka Version: 0.8.x
Maintainer: Shopify
License: MIT
https://github.com/Shopify/sarama
-------------
Enhanced Go Kafka consumer and producer implementations.
Kafka Version: 0.8.x
Maintainer: Big Data Open Source Security
License: Apache v2.0
https://github.com/stealthly/go_kafka_client
-------------
A pure Go implementation of the low level Kafka API.
Kafka Version: 0.8.x
Maintainer: Big Data Open Source Security
License: Apache v2.0
https://github.com/stealthly/siesta
-------------
A fast pure Go Kafka implementation with a clean API.
Kafka Version: 0.8.x
Maintainer: OptioPay
License: MIT
https://github.com/optiopay/kafka
-------------
https://github.com/nuance/kafka
https://github.com/jdamick/kafka.go
Kafka Version: 0.7.x
-------------
confluent-kafka-go: Confluent's Kafka client for Golang wraps the librdkafka C library, providing full Kafka protocol support with great performance and reliability.
The Golang bindings provides a high-level Producer and Consumer with support for the balanced consumer groups of Apache Kafka 0.9 and above.
Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x
Maintainer: Confluent
License: Apache v2.0
https://github.com/confluentinc/confluent-kafka-go
Docs: http://docs.confluent.io/current/clients/index.html
-------------
Erlang
-------------
Kafka client library in Erlang. Full support for 0.9+ consumer protocol, very efficient producer implementation.
https://github.com/klarna/brod
Kafka Version: 0.9.x, 0.10.x
Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)
License: Apache 2.0
-------------
A minimal, high-performance Kafka client in Erlang
https://github.com/helpshift/ekaf
Kafka Version: 0.8.x
Maintainer: Helpshift
License: Apache v2
-------------
erlkafka is a kafka client written in erlang
https://github.com/milindparikh/erlkafka.git
Kafka Version: 0.7.x
Maintainer: Milind Parikh
*License: BSD, LGPL
Also:
https://github.com/wooga/kafka-erlang
.NET
-------------
A fully featured .NET client for Apache Kafka based on librdkafka (a fork of rdkafka-dotnet).
Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x
Maintainer: Confluent Inc. (original author Andreas Heider)
License: Apache 2.0
https://github.com/confluentinc/confluent-kafka-dotnet
-------------
Pure C# client with full protocol support. Includes consumer, producer,
lower level components and gzip support (no snappy)
Kafka Version: 0.8.x
Maintainer: James Roland
License: Apache 2.0
https://github.com/Jroland/kafka-net
-------------
This is a .NET implementation of a client for Kafka using C# for Kafka 0.8. It provides for an implementation that covers most basic functionalities to include a simple Producer and Consumer.
Kafka Version: 0.8.x
Maintainer: ExactTarget
License: Apache 2.0
https://github.com/ExactTargetDev/kafka-net
-------------
.Net implementation of the Apache Kafka Protocol that provides basic functionality through Producer/Consumer classes. The project also offers balanced consumer implementation. The project is a fork from ExactTarget's Kafka-net Client.
Kafka Version: 0.8.x, 0.9.x
Maintainer: Microsoft
License: Apache 2.0
https://github.com/Microsoft/Kafkanet
-------------
C# client, asynchronous, all 3 compressions supported (read and write), tracks leader partition changes transparently, long time in production.
Kafka Version: 0.8.x
Maintainer: Vadim Chekan
License: Apache-2.0
https://github.com/ntent-ad/kafka4net
-------------
kafka-sharp - "High Performance" .NET Kafka Driver
Kafka Version: 0.8.x
Maintainer: Criteo
License: Apache 2.0
https://github.com/criteo/kafka-sharp
-------------
kafunk - Fully-featured, native F# client which works on .NET Core
Kafka Version: 0.8.x, 0.9.x, 0.10.x
Maintainer: Jet.com
License: Apache 2.0
-------------
Clojure
-------------
Fast kafka api for JVM languages implemented in clojure.
Kafka Version: 0.8.x
Maintainer: https://github.com/gerritjvv
License: Apache 2.0
https://github.com/gerritjvv/kafka-fast
-------------
Wrapper to the Java API for interacting with Kafka
Kafka Version: 0.8.x
Maintainer: https://github.com/pingles
License: Apache 2.0
https://github.com/pingles/clj-kafka/
-------------
Kafka clojure client library
Kafka Version: 0.8.x
Maintainer: Pierre-Yves Ritschard
License: MIT
-------------
Ruby
-------------
ruby-kafka -
Kafka Version: 0.9.x, 0.10.x, 0.11.x
Maintainer: Zendesk/ Daniel Schierbeck
Licence: Apache 2.0
https://github.com/zendesk/ruby-kafka
-------------
Karafka - . Karafka provides higher abstraction layers, including Capistrano, Docker and Heroku support. Based on ruby-kafka.
Kafka Version: 0.9.x, 0.10.x, 0.11.x
Maintainer: Maciej Mensfeld
Licence: MIT
https://github.com/karafka/karafka
-------------
Racecar – A simple framework for writing Kafka consumers in Ruby that integrates nicely with Rails. Based on ruby-kafka.
Kafka Version: 0.9.x, 0.10.x, 0.11.x
Maintainer: Zendesk / Daniel Schierbeck
Licence: Apache 2.0
https://github.com/zendesk/racecar
-------------
DeliveryBoy – A simple way to publish messages to Kafka from Ruby applications. Based on ruby-kafka.
Kafka Version: 0.9.x, 0.10.x, 0.11.x
Maintainer: Zendesk / Daniel Schierbeck
Licence: Apache 2.0
https://github.com/zendesk/delivery_boy
-------------
JRuby wrapper for producers and consumers of the existing API
Kafka Version: 0.8.x
Maintainer: Joseph Lawson
License: Apache 2.0
https://github.com/joekiller/jruby-kafka
-------------
Kafka Version: 0.8.x
Maintainer: Bob Potter
License: MIT
https://github.com/bpot/poseidon
-------------
https://github.com/acrosa/kafka-rb
Kafka Version: 0.7.x
Maintainer: Alejandro Crosa
License:: Apache 2.0
-------------
Event machine client:
https://github.com/groupme/em-kafka
Kafka Version: 0.7.x
-------------
JRuby Event stream processor
https://github.com/wooga/kafkaesque
Kafka Version: 0.7.x
-------------
Node.js
-------------
The node-rdkafka library is a high-performance NodeJS client for Apache Kafka that wraps the native librdkafka library. All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library.
https://github.com/Blizzard/node-rdkafka
Kafka Version: 0.9, 0.10
Node.js >= 4
Maintainer: Blizzard.com
License: MIT
-------------
Kafka-Node is a NodeJS client with Zookeeper integration
Kafka Version: 0.8.x
Maintainer: sohu.com
License: MIT
https://github.com/SOHU-Co/kafka-node/
-------------
Kafka-node is a pure JavaScript implementation for NodeJS Server with Vagrant and Docker support.
Kafka Version: 0.8.x
Maintainer: wurstmeister
License: Apache 2.0
https://github.com/wurstmeister/node-kafka-0.8-plus
-------------
Node-kafka is a node.js wrapper for the C library librdkafka
Kafka Version: 0.8.x
Maintainer: Sutoiku
License: MIT
https://github.com/sutoiku/node-kafka
-------------
kafka-java-bridge is a Nodejs wrapper for the JAVA high level kafka 0.8. consumer API
Kafka Version: 0.8.x
Maintainer: LivePersonInc
License: MIT
https://www.npmjs.com/package/kafka-java-bridge
https://github.com/LivePersonInc/kafka-java-bridge
-------------
Low-level protocol support in node.js.
https://github.com/cainus/Prozess
https://npmjs.org/package/prozess
Kafka Version: 0.7.x
Maintainers:
Gregg Caines
Eric lee
License: MIT
-------------
Alternate node client from Tagged
https://github.com/marcuswestin/node-kafka
Also:
https://github.com/dannycoates/franz-kafka
Kafka Version: 0.7.x
-------------
Proxy (HTTP REST, etc)
-------------
Dory is a producer daemon that supports clients in various programming languages. Clients send messages to Dory using local interprocess communication. Dory then takes full responsibility for reliable message delivery. Monitoring infrastructure can query Dory's web interface for JSON-based status and data quality reports.
https://github.com/dspeterson/dory
Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x
Maintainer: Dave Peterson
License: Apache v.2.0
-------------
Apache Kafka HTTP Endpoint for producing and consuming messages from topics
Kafka Version: 0.8.x
Maintainer: Big Data Open Source Security LLC
C
High performance C library with full protocol support
Maintainer: Magnus Edenhill
License: 2-clause BSD
https://github.com/edenhill/librdkafka
C++
Native C++ library with protocol support for Metadata, Produce, Fetch, and Offset.
Maintainer: David Tompkins
License: Apache 2.0
https://github.com/adobe-research/libkafka
.net
Pure C# client with full protocol support. Includes consumer, producer,
lower level components and gzip support (no snappy)
Maintainer: James Roland
License: Apache 2.0
https://github.com/Jrolandstealthly/dropwizard-kafka-nethttp
Clojure
https://github.com/pingles/clj-kafka/ 0.8 branch
Ruby
Pure Ruby, Consumer and Producer implementations included, GZIP and Snappy compression supported. Ruby 1.9.3 and up (CI runs MRI 2.0, JRuby and Rubinius).
-------------
(Deprecated) Kafka high level Producer and Consumer APIs are very hard to implement right. Rest endpoint gives access to native Scala high level consumer and producer APIs.
Maintainer: Sasha Klizhentas
License: Apache 2.0Maintainer: Bob Potter
License: MIT
https://github.com/bpotmailgun/poseidon
Node.js
Kafka-Node is a NodeJS client with Zookeeper integration
Maintainer: sohu.com
License: MIT
https://github.com/SOHU-Co/kafka-node/
Kafka-node is a pure JavaScript implementation for NodeJS Server with Vagrant and Docker support.
-------------
The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
Kafka Version: 0.8.x, 0.9.x, 0.10.x
Maintainer: Confluent Maintainer: wurstmeister
License: Apache 2.0https
- Docs: http://
...
...
...
...
Node-kafka is a node.js wrapper for the C library libkafkard.
Maintainer: Sutoiku
License: MIT
...
- rest/docs/intro.html
- Code:
...
...
...
-------------
Kafka-Pixy is a local aggregating HTTP proxy to Kafka messaging cluster.
Kafka Version: 0.8.x
Maintainer: Mailgun
...
Alternative Java Clients
Of course the main project maintains a set of jvm-based clients. But here are alternate clients.
Krackle is an optimized Kafka client built by Blackberry.
Maintainer: Blackberry
License: Apache 2.0
https://github.com/blackberry/Krackle/blob/dev/LICENSE
Storm
Port of Apache storm-kafka to >= 0.8.0
-------------
Efficient Kafka REST Proxy for producersMaintainer: wurstmeister
License: Apache 2.0
https://github.com/wurstmeister/storm-kafka-0.8-plus
Scala DSL
A DSL for Scala developers to produce and consume messages with Kafka.
Maintainer: Big Data Open Source Security LLC
License: Apache 2.0
https://github.com/stealthly/scala-kafka
HTTP REST
Apache Kafka HTTP Endpoint for producing and consuming messages from topics
Kafka Version: 0.8.x
Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)
License: Apache 2.0
-------------
Full featured HTTP REST Proxy for Kafka in Elixir with fine grained JWT-based authenticationMaintainer: Big Data Open Source Security LLC
License: Apache 2.0
https://github.com/stealthly/dropwizard-kafka-http
Maintainer: Sasha Klizhentas
License: Apache 2.0
https://github.com/mailgun/kafka-http
JRuby
JRuby wrapper for producers and consumers of the existing API
Maintainer: Joseph Lawson
License: Apache 2.0
https://github.com/joekiller/jruby-kafka
Kafka Version: 0.8.x
Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)
License: Apache 2.0
Perl
-------------
...
Pure Perl, Consumer and Producer implementations included. Zookeeper
integration. GZIP and Snappy compression supported.
Kafka Version: 0.8.x
Maintainer: Sergey Gladkov
License: Artistic Artistic License
https://github.com/TrackingSoft/Kafka
http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm
stdin/stdout
-------------
Generic producer and consumer for stdin and stdout
https://github.com/edenhill/kafkacat
Maintainer: Magnus Edenhill
License: 2-clause BSD
Erlang
.
A minimal, high-performance Kafka client in Erlang
https://github.com/helpshiftedenhill/ekafkafkacat
Maintainer: Helpshift
License: Apache v2
0.1-0.7.x
Java
Maintained with the main code base.
C
Native C library with Consumer, Producer and compression support.
https://github.com/edenhill/librdkafka
Kafka Version: 0.8.x
Maintainer: Maintainer: Magnus Edenhill
License: 2 2-clause BSD
C++
https://github.com/quipo/kafka-cpp
PHP
PHP library with Consumer (simple and Zookeeper-based), Producer and compression support (release notes).
https://github.com/quipo/kafka-php
Maintainer: Lorenzo Alberton
License: Apache v.2.0
Also:
https://github.com/michal-harish/kafka-php
PHP
-------------
PHP extension built on librdkafka
Kafka Version: 0.8.x
Maintainer: Elias Van Ootegem
License: MIT
Log4PHP Appender
https://github.com/dastra/log4php-kafka
Python
We have a wealth of python clients!
Protocol support for Kafka 0.7 in Python. GZip and Snappy compression supported
-------------
PHP client based on librdkafka
Kafka Version: 0.8.x
Maintainer: Arnaud Le Blanc, for Mention.com
License: MIT
Maintainer: David Arthur
License: Apache v.2.0
https://github.com/mumrah/kafka-python/tree/0.7Also:arnaud-lb/php-rdkafka
-------------
PHP library with Consumer (simple and Zookeeper-based), Producer and compression support (release notes).
https://github.com/dsullyquipo/pykafka
Maintainer:: Dan Sulley, LinkedIn
License: Apache 2.0
Python client from Disqus:
https://github.com/getsamsa/samsa
Maintainer:: Keith Bourgoin, Parse.ly
License: Apache 2.0
Kafka Version: 0.7.x
Maintainer: Lorenzo Alberton
License: Apache v.2.0
Also:
https://github.com/michal-harish/kafka-php
Log4PHP Appender
Python client from Urban Airship: https://github.com/urbanairship/pykafkap
Python client from Datadog: https://github.com/datadog/brod
(Producer, Simple Consumer, ZK-Consumer)
Ruby
https://github.com/acrosa/kafka-rb
Maintainer: Alejandro Crosa
License:: Apache 2.0
Event machine client:
https://github.com/groupme/em-kafka
JRuby Event stream processor
https://github.com/wooga/kafkaesque
Clojure
-------------
Rust
-------------
Pure Rust implementation with support for Metadata, Produce, Fetch, and Offset requests. Supports Gzip and Snappy compression
Kafka Version: 0.8.x
Maintainer: Yousuf Fauzan
License: MIT
Alternative Java
-------------
Of course the main project maintains a set of jvm-based clients. But here are alternate clients.
Krackle is an optimized Kafka client built by Blackberry.
Kafka Version: 0.8.x
Maintainer: Blackberry
License: Apache 2.0https://github.com/pingles/clj-kafka
https://github.com/miniway/kafka-clj
Go
blackberry/Krackle/blob/dev/LICENSE
Storm
-------------
Port of Apache storm-kafka to >= 0.8.0
Maintainer: wurstmeister
License: Apache 2.0https://github.com/nuance/kafka
https://github.com/jdamickwurstmeister/storm-kafka.go
Node.js
Low-level protocol support in node.js.
Scala DSL
-------------
A DSL for Scala developers to produce and consume messages with Kafka.
Kafka Version: 0.8.x
Maintainer: Big Data Open Source Security LLC
License: Apache 2.0https://github.com/cainus/Prozess
https://npmjsgithub.orgcom/packagestealthly/prozess
Maintainers:
Gregg Caines
Eric lee
License: MITscala-kafka
Clojure
-------------
Alternate node client from Tagged
https://github.com/marcuswestinpingles/nodeclj-kafkaAlso:
https://github.com/dannycoates/franz-kafka
Erlang
erlkafka is a kafka client written in erlang
https://github.com/milindparikh/erlkafka.git
Maintainer: Milind Parikh
*License: BSD, LGPL
Kafka Version: 0.7.x
Swift
-------------
Pure Swift implementation of Producer + Simple and High-Level Consumer
Kafka Version: 0.9.x
Maintainer: Kellan Cummings, Luke Lau
License: MIT
...
wooga/kafka-erlang
Client Libraries Previously Supported
https://svn.apache.org/repos/asf/incubator/kafka/branches/legacy_client_libraries