View Source

How The Kafka Project Handles Clients

Starting with the 0.8 release we are maintaining all but the jvm client external to the main code base. The reason for this is that it allows a small group of implementers who know the language of that client to quickly iterate on their code base on their own release cycle. Having these maintained centrally was becoming a bottleneck as the main committers can't hope to know every possible programming language to be able to perform meaningful code review and testing. This lead to a scenario where the committers were attempting to review and test code they didn't understand.

We are instead moving to the redis/memcached model which seems to work better at supporting a rich ecosystem of high quality clients.

We haven't tried all these clients and can't vouch for them. The normal rules of open source apply.

If you are aware of other clients not listed here (or are the author of such a client), please add it here. Please also feel free to help fill out information on the features the client supports, level of activity of the project, level of documentation, etc.

For 0.8.x

The 0.8 release changes the protocol fairly substantially. The new protocol is documented here.

Producer Daemon

Bruce is a producer daemon that supports clients in various programming languages. Clients write messages to Bruce's UNIX domain datagram socket in a simple binary format. Bruce then takes full responsibility for reliable message delivery. Monitoring infrastructure can query Bruce's web interface for JSON-based status and data quality reports.

Maintainer: Dave Peterson
License: Apache v.2.0

https://github.com/ifwe/bruce

Python

Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

Maintainer: David Arthur
License: Apache v.2.0

https://github.com/mumrah/kafka-python

Go (AKA golang)

Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

Maintainer: Shopify
License: MIT

https://github.com/Shopify/sarama

Another Pure Go implementation built to work as like the "Consumer Group" High Level Consumer with more features including plug-able partition ownership and work fan out.

Maintainer: Big Data Open Source Security
License: Apache v2.0

https://github.com/stealthly/go_kafka_client

C/C++

Robust high performance C/C++ library with full protocol support

Maintainer: Magnus Edenhill
License: 2-clause BSD

https://github.com/edenhill/librdkafka

Native C++ library with protocol support for Metadata, Produce, Fetch, and Offset.

Maintainer: David Tompkins
License: Apache 2.0

https://github.com/adobe-research/libkafka

C++ Header-only Kafka Client Library using Boost Asio

Maintainer: Daniel Joos
License: MIT

https://github.com/danieljoos/libkafka-asio

A C++11 asyncronous producer/consumer library for Apache Kafka based on boost asio

Maintainer: Svante Karlsson
License: Boost Software License - Version 1.0

https://github.com/bitbouncer/csi-kafka

.net

Pure C# client with full protocol support. Includes consumer, producer,

lower level components and gzip support (no snappy)

Maintainer: James Roland

License: Apache 2.0

https://github.com/Jroland/kafka-net

Clojure

Fast kafka api for JVM languages implemented in clojure

Maintainer: https://github.com/gerritjvv

License: Apache 2.0

https://github.com/gerritjvv/kafka-fast

Wrapper to the Java API for interacting with Kafka

Maintainer: https://github.com/pingles

License: Apache 2.0

https://github.com/pingles/clj-kafka/

Ruby

Pure Ruby, Consumer and Producer implementations included, GZIP and Snappy compression supported. Ruby 1.9.3 and up (CI runs MRI 2.0, JRuby and Rubinius).

Maintainer: Bob Potter
License: MIT

https://github.com/bpot/poseidon

Node.js

Kafka-Node is a NodeJS client with Zookeeper integration

Maintainer: sohu.com

License: MIT

https://github.com/SOHU-Co/kafka-node/

Kafka-node is a pure JavaScript implementation for NodeJS Server with Vagrant and Docker support.

Maintainer: wurstmeister
License: Apache 2.0

https://github.com/wurstmeister/node-kafka-0.8-plus

Node-kafka is a node.js wrapper for the C library libkafkard.

Maintainer: Sutoiku

License: MIT

https://github.com/sutoiku/node-kafka

Alternative Java Clients

Of course the main project maintains a set of jvm-based clients. But here are alternate clients.

Krackle is an optimized Kafka client built by Blackberry.

Maintainer: Blackberry

License: Apache 2.0

https://github.com/blackberry/Krackle/blob/dev/LICENSE

Storm

Port of Apache storm-kafka to >= 0.8.0

Maintainer: wurstmeister
License: Apache 2.0

https://github.com/wurstmeister/storm-kafka-0.8-plus

Scala DSL

A DSL for Scala developers to produce and consume messages with Kafka.

Maintainer: Big Data Open Source Security LLC
License: Apache 2.0

https://github.com/stealthly/scala-kafka

HTTP REST

Apache Kafka HTTP Endpoint for producing and consuming messages from topics

Maintainer: Big Data Open Source Security LLC
License: Apache 2.0

https://github.com/stealthly/dropwizard-kafka-http

Maintainer: Sasha Klizhentas
License: Apache 2.0

https://github.com/mailgun/kafka-http

Maintainer: Confluent
License: Apache 2.0

Docs: http://confluent.io/docs/current/kafka-rest/docs/intro.html

Code: https://github.com/confluentinc/kafka-rest

JRuby

JRuby wrapper for producers and consumers of the existing API

Maintainer: Joseph Lawson
License: Apache 2.0

https://github.com/joekiller/jruby-kafka

Perl

Pure Perl, Consumer and Producer implementations included. Zookeeper
integration. GZIP and Snappy compression supported.

Maintainer: Sergey Gladkov
License: Artistic License

https://github.com/TrackingSoft/Kafka
http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm