How The Kafka Project Handles Clients

Starting with the 0.8 release we are maintaining all but the jvm client external to the main code base. The reason for this is that it allows a small group of implementers who know the language of that client to quickly iterate on their code base on their own release cycle. Having these maintained centrally was becoming a bottleneck as the main committers can't hope to know every possible programming language to be able to perform meaningful code review and testing. This lead to a scenario where the committers were attempting to review and test code they didn't understand.

...

If you are aware of other clients not listed here (or are the author of such a client), please add it here. Please also feel free to help fill out information on the features the client supports, level of activity of the project, level of documentation, etc.

For 0.8.x

The 0.8 release changes the protocol fairly substantially. The new protocol is documented here.

Producer Daemon

Dory is a producer daemon that supports clients in various programming languages.  Clients send messages to Dory using local interprocess communication.  Dory then takes full responsibility for reliable message delivery.  Monitoring infrastructure can query Dory's web interface for JSON-based status and data quality reports.

...

C/C++

-------------

Robust high performance C/C++ library with full protocol support

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

https://github.com/edenhill/librdkafka

Python

High performance Python client based on Librdkafka with full protocol support. 

Maintainer: Confluent.io
License: Apache v.2.0

https://github.com/confluentinc/confluent-kafka-python

-------------

Confluent's Apache Kafka client for Python wraps the librdkafka C library, providing full Kafka protocol support with great performance and reliability.

https://github.com/confluentinc/confluent-kafka-python

Go (AKA golang)

-------------

Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

Kafka Version: 0.8.x

Maintainer: Shopify
License: MIT

https://github.com/Shopify/sarama

-------------

Enhanced Go Kafka consumer and producer implementations.

Kafka Version: 0.8.x

Maintainer: Big Data Open Source Security
License: Apache v2.0
https://github.com/stealthly/go_kafka_client

-------------

A pure Go implementation of the low level Kafka API.

Kafka Version: 0.8.x

Maintainer: Big Data Open Source Security
License: Apache v2.0

https://github.com/stealthly/siesta

-------------

A fast pure Go Kafka implementation with a clean API.

Kafka Version: 0.8.x

Maintainer: OptioPay
License: MIT

https://github.com/optiopay/kafka

-------------

https://github.com/nuance/kafka

https://github.com/jdamick/kafka.go

Kafka Version: 0.7.x

-------------

Erlang

-------------

Kafka client library in Erlang. Full support for 0.9+ consumer protocol, very efficient producer implementation.

https://github.com/klarna/brod

Kafka Version: 0.9.x, 0.10.x

Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)

License: Apache 2.0

-------------

...

Python

...

...

...

...

...

Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

Maintainer: Dana Powers
License: Apache v.2.0

http://github.com/dpkp/kafka-python

Python driver with full protocol support and balanced consumer implementation. GZIP and Snappy compression supported.

...

...

Go (AKA golang)

Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

...

...

Enhanced Go Kafka consumer and producer implementations.

Maintainer: Big Data Open Source Security
License: Apache v2.0
https://github.com/stealthly/go_kafka_client

A pure Go implementation of the low level Kafka API.

...

...

...

...

A fast pure Go Kafka implementation with a clean API.

Maintainer: OptioPay
License: MIT

https://github.com/optiopay/kafka

C/C++

Robust high performance C/C++ library with full protocol support

...

...

Native C++ library with protocol support for Metadata, Produce, Fetch, and Offset.

...

C++ Header-only Kafka Client Library using Boost Asio

Maintainer: Daniel Joos
License: MIT

...

...

A C++11 asyncronous producer/consumer library for Apache Kafka based on boost asio

...

...

...

...

libasynckafkaclient - C++ based single threaded asynchronous library

...

...

.net

Pure C# client with full protocol support.  Includes consumer, producer,

lower level components and gzip support (no snappy)

Maintainer: James Roland 

License: Apache 2.0

https://github.com/Jroland/kafka-net

This is a .NET implementation of a client for Kafka using C# for Kafka 0.8. It provides for an implementation that covers most basic functionalities to include a simple Producer and Consumer.

Maintainer: ExactTarget

...

...

...

rdkafka-dotnet is a C# client for Apache Kafka based on librdkafka.

Maintainer: Andreas Heider

License: BSD (2 clause)

https://github.com/ah-/rdkafka-dotnet

.Net implementation of the Apache Kafka Protocol that provides basic functionality through Producer/Consumer classes. The project also offers balanced consumer implementation. The project is a fork from ExactTarget's Kafka-net Client.

Maintainer: Microsoft

...

...

...

...

...

...

kafka-sharp - "High Performance" .NET Kafka Driver

Maintainer: Criteo

...

...

...

Clojure

Fast kafka api for JVM languages implemented in clojure

Maintainer: https://github.com/gerritjvv

...

...

...

Wrapper to the Java API for interacting with Kafka

Maintainer: https://github.com/pingles

...

...

Kafka clojure client library

Maintainer: Pierre-Yves Ritschard

License: MIT

Code, Documentation

Ruby

Pure Ruby, Consumer and Producer implementations included, GZIP and Snappy compression supported. Ruby 1.9.3 and up (CI runs MRI 2.0, JRuby and Rubinius).

...

...

Karafka - Microframework used to simplify Apache Kafka-based Ruby applications development.

...

...

ruby-kaka - A Ruby client library for the Kafka distributed log system

Maintainer: Zendesk/ Daniel Schierbeck

...

...

Node.js

Kafka-Node is a NodeJS client with Zookeeper integration

...

...

...

Kafka-node is a pure JavaScript implementation for NodeJS Server with Vagrant and Docker support.

...

...

Node-kafka is a node.js wrapper for the C library librdkafka

Maintainer: Sutoiku

...

...

...

...

...

kafka-java-bridge is a Nodejs wrapper for the JAVA high level kafka 0.8. consumer API

Maintainer: LivePersonInc

License: MIT

https://www.npmjs.com/package/kafka-java-bridge

https://github.com/LivePersonInc/kafka-java-bridge

Alternative Java Clients

Of course the main project maintains a set of jvm-based clients. But here are alternate clients.

Krackle is an optimized Kafka client built by Blackberry.

Maintainer: Blackberry

...

...

Storm 

Port of Apache storm-kafka to >= 0.8.0

Maintainer: wurstmeister
License: Apache 2.0

https://github.com/wurstmeister/storm-kafka-0.8-plus

Scala DSL 

A DSL for Scala developers to produce and consume messages with Kafka.  

Maintainer: Big Data Open Source Security LLC
License: Apache 2.0

https://github.com/stealthly/scala-kafka

HTTP REST 

Apache Kafka HTTP Endpoint for producing and consuming messages from topics

Maintainer: Big Data Open Source Security LLC
License: Apache 2.0

Proxy (HTTP REST, etc)

-------------

Dory is a producer daemon that supports clients in various programming languages.  Clients send messages to Dory using local interprocess communication.  Dory then takes full responsibility for reliable message delivery.  Monitoring infrastructure can query Dory's web interface for JSON-based status and data quality reports.

https://github.com/stealthly/dropwizard-kafka-http

(Deprecated) Kafka high level Producer and Consumer APIs are very hard to implement right. Rest endpoint gives access to native Scala high level consumer and producer APIs.

Maintainer: Sasha Klizhentas
License: Apache 2.0

https://github.com/mailgun/kafka-http

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.

dspeterson/dory

Kafka Version: 0.8.x

Maintainer: Dave Peterson
License: Apache v.2.0

-------------

Apache Kafka HTTP Endpoint for producing and consuming messages from topics

Kafka Version: 0.8.x

Maintainer: Big Data Open Source Security LLCMaintainer: Confluent 
License: Apache 2.0

...

https://

...

github.

...

com/

...

stealthly/

...

dropwizard-kafka-http

-------------

(Deprecated) Kafka high level Producer and Consumer APIs are very hard to implement right. Rest endpoint gives access to native Scala high level consumer and producer APIs.

Maintainer: Sasha Klizhentas

Kafka-Pixy is a local aggregating HTTP proxy to Kafka messaging cluster.

Maintainer: Mailgun
License: Apache 2.0

https://github.com/mailgun/kafka-pixy-http

-------------

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.

Efficient Kafka REST Proxy for producers

https://github.com/klarna/kastle

Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)
License: Apache 2.0

JRuby 

JRuby wrapper for producers and consumers of the existing API

Kafka Version: 0.8.x, 0.9.x, 0.10.x

Maintainer: Confluent Maintainer: Joseph Lawson
License: Apache 2.0https

• Docs: http://

...

• confluent.

...

• io/docs/

...

• current/

...

Perl

...

• kafka-rest/docs/intro.html
• Code: https://github.com/

...

• confluentinc/kafka-rest

-------------

Kafka-Pixy is a local aggregating HTTP proxy to Kafka messaging cluster.

Kafka Version: 0.8.x

Maintainer: Mailgun
License: Apache 2.0

stdin/stdout

Generic producer and consumer for stdin and stdout

https://github.com/edenhill/kafkacat

Maintainer: Magnus Edenhill
License: 2-clause BSD

Erlang

A minimal, high-performance Kafka client in Erlang

https://github.com/helpshiftmailgun/ekafkafka-pixy

Maintainer: Helpshift
License: Apache v2

-------------

Efficient Kafka REST Proxy for producersKafka client library in Erlang. Full support for 0.9+ consumer protocol, very efficient producer implementation.

https://github.com/klarna/brodkastle

Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)
License: Apache 2.0

PHP

PHP extension built on librdkafka

Maintainer:  Elias Van Ootegem
License: MIT

https://github.com/EVODelavega/phpkafka

PHP client based on librdkafka

 Apache 2.0

JRuby 

-------------

JRuby wrapper for producers and consumers of the existing API

Kafka Version: 0.8.x

Maintainer: Joseph Lawson
License: Apache 2.0

 Maintainer: Arnaud Le Blanc, for Mention.com
License: MIT

https://github.com/arnaud-lbjoekiller/php-rdkafka

Rust

Pure Rust implementation with support for Metadata, Produce, Fetch, and Offset requests. Supports Gzip and Snappy compression

Maintainer:  Yousuf Fauzan
License: MIT

Code
Documentation

0.1-0.7.x

Java

Maintained with the main code base.

C

jruby-kafka

Perl

-------------

Pure Perl, Consumer and Producer implementations included. Zookeeper
integration. GZIP and Snappy compression supported.

Kafka Version: 0.8.x
Maintainer: Sergey Gladkov
License: Artistic License

https://github.com/TrackingSoft/Kafka
http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm

stdin/stdout

-------------

Generic producer and consumer for stdin and stdout.

 Native C library with Consumer, Producer and compression support.

https://github.com/edenhill/librdkafka/edenhill/kafkacat

Kafka Version: 0.8.x

Maintainer: Magnus Edenhill
License: 2 2-clause BSD

C++

https://github.com/quipo/kafka-cpp

PHP

PHP library with Consumer (simple and Zookeeper-based), Producer and compression support (release notes).

https://github.com/quipo/kafka-php

Maintainer: Lorenzo Alberton
License: Apache v.2.0

PHP

-------------

PHP extension built on librdkafka

Kafka Version: 0.8.x

Maintainer:  Elias Van Ootegem
License: MIT

 Also:

https://github.com/michal-harish/kafka-php

Log4PHP Appender

https://github.com/dastra/log4php-kafka

Python

We have a wealth of python clients!

Protocol support for Kafka 0.7 in Python. GZip and Snappy compression supported

/EVODelavega/phpkafka

-------------

PHP client based on librdkafka

Kafka Version: 0.8.x

Maintainer: Arnaud Le Blanc, for Mention.com
License: MIT

 Maintainer: David Arthur
License: Apache v.2.0

https://github.com/mumrah/kafka-python/tree/0.7Also:arnaud-lb/php-rdkafka

-------------

PHP library with Consumer (simple and Zookeeper-based), Producer and compression support (release notes).

https://github.com/dsullyquipo/pykafka
Maintainer:: Dan Sulley, LinkedIn
License: Apache 2.0

Python client from Disqus:
https://github.com/getsamsa/samsa
Maintainer:: Keith Bourgoin, Parse.ly
License: Apache 2.0

kafka-php

Kafka Version: 0.7.x

Maintainer: Lorenzo Alberton
License: Apache v.2.0

Also:

Python client from Urban Airship: https://github.com/urbanairship/pykafkapPython michal-harish/kafka-php

Log4PHP Appender

client from Datadog: https://github.com/datadog/brod
(Producer, Simple Consumer, ZK-Consumer)

Ruby

https://github.com/acrosa/kafka-rb
Maintainer: Alejandro Crosa
License:: Apache 2.0

Event machine client:
https://github.com/groupme/em-kafka

JRuby Event stream processor
https://github.com/wooga/kafkaesque

Clojure

https://github.com/pingles/clj-kafka

https://github.com/miniway/kafka-clj

.com/dastra/log4php-kafka

-------------

Rust

-------------

Pure Rust implementation with support for Metadata, Produce, Fetch, and Offset requests. Supports Gzip and Snappy compression

Kafka Version: 0.8.x

Maintainer:  Yousuf Fauzan
License: MIT

Code
Documentation

Alternative Java

-------------

Of course the main project maintains a set of jvm-based clients. But here are alternate clients.

Krackle is an optimized Kafka client built by Blackberry.

Kafka Version: 0.8.x

Maintainer: Blackberry

License: Apache 2.0

...

https://github.com/nuance/kafka

https://github.com/jdamick/kafka.go

Node.js

blackberry/Krackle/blob/dev/LICENSE

Storm

-------------

Port of Apache storm-kafka to >= 0.8.0

Maintainer: wurstmeister
License: Apache 2.0Low-level protocol support in node.js.

https://github.com/cainus/Prozess
https://npmjs.org/package/prozess
Maintainers:
Gregg Caines
Eric lee
License: MITAlternate node client from Taggedwurstmeister/storm-kafka-0.8-plus

Scala DSL 

-------------

A DSL for Scala developers to produce and consume messages with Kafka.

Kafka Version: 0.8.x

Maintainer: Big Data Open Source Security LLC
License: Apache 2.0

https://github.com/marcuswestin/node-kafkaAlso:stealthly/scala-kafka

Clojure

-------------

https://github.com/dannycoatespingles/franzclj-kafka

...

erlkafka is a kafka client written in erlang

https://github.com/milindparikh/erlkafka.git

Maintainer: Milind Parikh
*License: BSD, LGPL

miniway/kafka-clj

Kafka Version: 0.7.x

 Also:
https://github.com/wooga/kafka-erlang

Client Libraries Previously Supported

https://svn.apache.org/repos/asf/incubator/kafka/branches/legacy_client_libraries