How The Kafka Project Handles Clients
Starting with the 0.8 release we are maintaining all but the jvm client external to the main code base. The reason for this is that it allows a small group of implementers who know the language of that client to quickly iterate on their code base on their own release cycle. Having these maintained centrally was becoming a bottleneck as the main committers can't hope to know every possible programming language to be able to perform meaningful code review and testing. This lead to a scenario where the committers were attempting to review and test code they didn't understand.
We are instead moving to the redis/memcached model which seems to work better at supporting a rich ecosystem of high quality clients.
We haven't tried all these clients and can't vouch for them. The normal rules of open source apply.
If you are aware of other clients not listed here (or are the author of such a client), please add it here. Please also feel free to help fill out information on the features the client supports, level of activity of the project, level of documentation, etc.
For 0.8.x
The 0.8 release changes the protocol fairly substantially. The new protocol is documented here.
Producer Daemon
Bruce is a producer daemon that supports clients in various programming languages. Clients write messages to Bruce's UNIX domain datagram socket in a simple binary format. Bruce then takes full responsibility for reliable message delivery. Monitoring infrastructure can query Bruce's web interface for JSON-based status and data quality reports.
Maintainer: Dave Peterson
License: Apache v.2.0
Python
Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.
Maintainer: Dana Powers
License: Apache v.2.0
http://github.com/dpkp/kafka-python
Python driver with full protocol support and balanced consumer implementation. GZIP and Snappy compression supported.
Maintainer: Parse.ly
License: Apache v2.0
https://github.com/Parsely/pykafka
Go (AKA golang)
Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.
Maintainer: Shopify
License: MIT
https://github.com/Shopify/sarama
Enhanced Go Kafka consumer and producer implementations.
Maintainer: Big Data Open Source Security
License: Apache v2.0
https://github.com/stealthly/go_kafka_client
A pure Go implementation of the low level Kafka API.
Maintainer: Big Data Open Source Security
License: Apache v2.0
https://github.com/stealthly/siesta
A fast pure Go Kafka implementation with a clean API.
Maintainer: OptioPay
License: MIT
https://github.com/optiopay/kafka
C/C++
Robust high performance C/C++ library with full protocol support
Maintainer: Magnus Edenhill
License: 2-clause BSD
https://github.com/edenhill/librdkafka
Native C++ library with protocol support for Metadata, Produce, Fetch, and Offset.
Maintainer: David Tompkins
License: Apache 2.0
https://github.com/adobe-research/libkafka
C++ Header-only Kafka Client Library using Boost Asio
Maintainer: Daniel Joos
License: MIT
https://github.com/danieljoos/libkafka-asio
A C++11 asyncronous producer/consumer library for Apache Kafka based on boost asio
Maintainer:
License: Boost Software License - Version 1.0
https://github.com/bitbouncer/csi-kafka
.net
Pure C# client with full protocol support. Includes consumer, producer,
lower level components and gzip support (no snappy)
Maintainer: James Roland
License: Apache 2.0
https://github.com/Jroland/kafka-net
This is a .NET implementation of a client for Kafka using C# for Kafka 0.8. It provides for an implementation that covers most basic functionalities to include a simple Producer and Consumer.
Maintainer: ExactTarget
License: Apache 2.0
https://github.com/ExactTargetDev/kafka-net
Clojure
Fast kafka api for JVM languages implemented in clojure
Maintainer: https://github.com/gerritjvv
License: Apache 2.0
https://github.com/gerritjvv/kafka-fast
Wrapper to the Java API for interacting with Kafka
Maintainer: https://github.com/pingles
License: Apache 2.0
https://github.com/pingles/clj-kafka/
Kafka clojure client library
Maintainer: Pierre-Yves Ritschard
License: MIT
Ruby
Pure Ruby, Consumer and Producer implementations included, GZIP and Snappy compression supported. Ruby 1.9.3 and up (CI runs MRI 2.0, JRuby and Rubinius).
Maintainer: Bob Potter
License: MIT
https://github.com/bpot/poseidon
Karafka - Microframework used to simplify Apache Kafka-based Ruby applications development.
Maintainer: Maciej Mensfeld
Licence: MIT
https://github.com/karafka/karafka
Node.js
Kafka-Node is a NodeJS client with Zookeeper integration
Maintainer: sohu.com
License: MIT
https://github.com/SOHU-Co/kafka-node/
Kafka-node is a pure JavaScript implementation for NodeJS Server with Vagrant and Docker support.
Maintainer: wurstmeister
License: Apache 2.0
https://github.com/wurstmeister/node-kafka-0.8-plus
Node-kafka is a node.js wrapper for the C library librdkafka
Maintainer: Sutoiku
License: MIT
https://github.com/sutoiku/node-kafka
Alternative Java Clients
Of course the main project maintains a set of jvm-based clients. But here are alternate clients.
Krackle is an optimized Kafka client built by Blackberry.
Maintainer: Blackberry
License: Apache 2.0
https://github.com/blackberry/Krackle/blob/dev/LICENSE
Storm
Port of Apache storm-kafka to >= 0.8.0
Maintainer: wurstmeister
License: Apache 2.0
https://github.com/wurstmeister/storm-kafka-0.8-plus
Scala DSL
A DSL for Scala developers to produce and consume messages with Kafka.
Maintainer: Big Data Open Source Security LLC
License: Apache 2.0
https://github.com/stealthly/scala-kafka
HTTP REST
Apache Kafka HTTP Endpoint for producing and consuming messages from topics
Maintainer: Big Data Open Source Security LLC
License: Apache 2.0
https://github.com/stealthly/dropwizard-kafka-http
(Deprecated) Kafka high level Producer and Consumer APIs are very hard to implement right. Rest endpoint gives access to native Scala high level consumer and producer APIs.
Maintainer: Sasha Klizhentas
License: Apache 2.0
https://github.com/mailgun/kafka-http
The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.
Maintainer: Confluent
License: Apache 2.0
- Docs: http://confluent.io/docs/current/kafka-rest/docs/intro.html
- Code: https://github.com/confluentinc/kafka-rest
Kafka-Pixy is a local aggregating HTTP proxy to Kafka messaging cluster.
Maintainer: Mailgun
License: Apache 2.0
https://github.com/mailgun/kafka-pixy
JRuby
JRuby wrapper for producers and consumers of the existing API
Maintainer: Joseph Lawson
License: Apache 2.0
https://github.com/joekiller/jruby-kafka
Perl
Pure Perl, Consumer and Producer implementations included. Zookeeper
integration. GZIP and Snappy compression supported.
Maintainer: Sergey Gladkov
License: Artistic License
https://github.com/TrackingSoft/Kafka
http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm
stdin/stdout
Generic producer and consumer for stdin and stdout
https://github.com/edenhill/kafkacat
Maintainer: Magnus Edenhill
License: 2-clause BSD
Erlang
A minimal, high-performance Kafka client in Erlang
https://github.com/helpshift/ekaf
Maintainer: Helpshift
License: Apache v2
Kafka client library in Erlang. Consumer, Producer but no compression (yet).
https://github.com/klarna/brod
Maintainer: Ivan Dyachkov (Klarna AB)
License: Apache v2
PHP
PHP extension built on librdkafka
Maintainer: Elias Van Ootegem
License: MIT
https://github.com/EVODelavega/phpkafka
PHP client based on librdkafka
Maintainer: Arnaud Le Blanc, for Mention.com
License: MIT
https://github.com/arnaud-lb/php-rdkafka
Rust
Pure Rust implementation with support for Metadata, Produce, Fetch, and Offset requests. Supports Gzip and Snappy compression
Maintainer: Yousuf Fauzan
License: MIT
0.1-0.7.x
Java
Maintained with the main code base.
C
Native C library with Consumer, Producer and compression support.
https://github.com/edenhill/librdkafka
Maintainer: Magnus Edenhill
License: 2-clause BSD
C++
https://github.com/quipo/kafka-cpp
PHP
PHP library with Consumer (simple and Zookeeper-based), Producer and compression support (release notes).
https://github.com/quipo/kafka-php
Maintainer: Lorenzo Alberton
License: Apache v.2.0
Also:
https://github.com/michal-harish/kafka-php
Log4PHP Appender
https://github.com/dastra/log4php-kafka
Python
We have a wealth of python clients!
Protocol support for Kafka 0.7 in Python. GZip and Snappy compression supported
Maintainer: David Arthur
License: Apache v.2.0
https://github.com/mumrah/kafka-python/tree/0.7
Also:
https://github.com/dsully/pykafka
Maintainer:: Dan Sulley, LinkedIn
License: Apache 2.0
Python client from Disqus:
https://github.com/getsamsa/samsa
Maintainer:: Keith Bourgoin, Parse.ly
License: Apache 2.0
Python client from Urban Airship: https://github.com/urbanairship/pykafkap
Python client from Datadog: https://github.com/datadog/brod
(Producer, Simple Consumer, ZK-Consumer)
Ruby
https://github.com/acrosa/kafka-rb
Maintainer: Alejandro Crosa
License:: Apache 2.0
Event machine client:
https://github.com/groupme/em-kafka
JRuby Event stream processor
https://github.com/wooga/kafkaesque
Clojure
https://github.com/pingles/clj-kafka
https://github.com/miniway/kafka-clj
Go
https://github.com/nuance/kafka
https://github.com/jdamick/kafka.go
Node.js
Low-level protocol support in node.js.
https://github.com/cainus/Prozess
https://npmjs.org/package/prozess
Maintainers:
Gregg Caines
Eric lee
License: MIT
Alternate node client from Tagged
https://github.com/marcuswestin/node-kafka
Also:
https://github.com/dannycoates/franz-kafka
Erlang
erlkafka is a kafka client written in erlang
https://github.com/milindparikh/erlkafka.git
Maintainer: Milind Parikh
*License: BSD, LGPL
Also:
https://github.com/wooga/kafka-erlang
Client Libraries Previously Supported
https://svn.apache.org/repos/asf/incubator/kafka/branches/legacy_client_libraries