How The Kafka Project Handles Clients
Starting with the 0.8 release we are maintaining all but the jvm client external to the main code base. The reason for this is that it allows a small group of implementers who know the language of that client to quickly iterate on their code base on their own release cycle. Having these maintained centrally was becoming a bottleneck as the main committers can't hope to know every possible programming language to be able to perform meaningful code review and testing. This lead to a scenario where the committers were attempting to review and test code they didn't understand.
We are instead moving to the redis/memcached model which seems to work better at supporting a rich ecosystem of high quality clients.
We haven't tried all these clients and can't vouch for them. The normal rules of open source apply.
If you are aware of other clients not listed here (or are the author of such a client), please add it here. Please also feel free to help fill out information on the features the client supports, level of activity of the project, level of documentation, etc.
For 0.8.x
The 0.8 release changes the protocol fairly substantially. This version hasn't been released yet. The new protocol is documented here. A number of clients are in progress, and we will update when they are complete.
Python
Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.
Maintainer: David Arthur
License: Apache v.2.0
https://github.com/mumrah/kafka-python
C
Native C library with Producer support.
https://github.com/edenhill/librdkafka 0.8 branch
Maintainer: Magnus Edenhill
License: 2-clause BSD
Clojure
https://github.com/pingles/clj-kafka/ 0.8 branch
0.1-0.7.x
Java
Maintained with the main code base.
C
Native C library with Consumer, Producer and compression support.
https://github.com/edenhill/librdkafka
Maintainer: Magnus Edenhill
License: 2-clause BSD
C++
https://github.com/quipo/kafka-cpp
PHP
PHP library with Consumer (simple and Zookeeper-based), Producer and compression support (release notes).
https://github.com/quipo/kafka-php
Maintainer: Lorenzo Alberton
License: Apache v.2.0
Also:
https://github.com/michal-harish/kafka-php
Log4PHP Appender
https://github.com/dastra/log4php-kafka
Python
We have a wealth of python clients!
Protocol support for Kafka 0.7 in Python. GZip and Snappy compression supported
Maintainer: David Arthur
License: Apache v.2.0
https://github.com/mumrah/kafka-python/tree/0.7
Also:
https://github.com/dsully/pykafka
Maintainer:: Dan Sulley, LinkedIn
License: Apache 2.0
Python client from Disqus:
https://github.com/getsamsa/samsa
Maintainer:: Keith Bourgoin, Parse.ly
License: Apache 2.0
Python client from Urban Airship:
https://github.com/urbanairship/pykafkap
Python client from Datadog:
https://github.com/datadog/brod
(Producer, Simple Consumer, ZK-Consumer)
Ruby
https://github.com/acrosa/kafka-rb
Maintainer: Alejandro Crosa
License:: Apache 2.0
Event machine client:
https://github.com/groupme/em-kafka
JRuby Event stream processor
https://github.com/wooga/kafkaesque
Clojure
https://github.com/pingles/clj-kafka
https://github.com/miniway/kafka-clj
Go
https://github.com/nuance/kafka
https://github.com/jdamick/kafka.go
Node.js
Low-level protocol support in node.js.
https://github.com/cainus/Prozess
https://npmjs.org/package/prozess
Maintainers:
Gregg Caines
Eric lee
License: MIT
Alternate node client from Tagged
https://github.com/marcuswestin/node-kafka
Also:
https://github.com/dannycoates/franz-kafka
Erlang
erlkafka is a kafka client written in erlang
https://github.com/milindparikh/erlkafka.git
Maintainer: Milind Parikh
*License: BSD, LGPL
Also:
https://github.com/wooga/kafka-erlang
Client Libraries Previously Supported
https://svn.apache.org/repos/asf/incubator/kafka/branches/legacy_client_libraries