Table of Contents

How The Kafka Project Handles Clients

Starting with the 0.8 release we are maintaining all but the jvm client external to the main code base. The reason for this is that it allows a small group of implementers who know the language of that client to quickly iterate on their code base on their own release cycle. Having these maintained centrally was becoming a bottleneck as the main committers can't hope to know every possible programming language to be able to perform meaningful code review and testing. This lead to a scenario where the committers were attempting to review and test code they didn't understand.

...

If you are aware of other clients not listed here (or are the author of such a client), please add it here. Please also feel free to help fill out information on the features the client supports, level of activity of the project, level of documentation, etc.

For 0.8.x

The 0.8 release changes the protocol fairly substantially. The new protocol is documented here.

Producer Daemon

Bruce is a producer daemon that supports clients in various programming languages. Clients write messages to Bruce's UNIX domain datagram socket in a simple binary format. Bruce then takes full responsibility for reliable message delivery. Monitoring infrastructure can query Bruce's web interface for JSON-based status and data quality reports.

Maintainer: Dave Peterson
License: Apache v.2.0

https://github.com/ifwe/bruce

Python

Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

Maintainer: Dana Powers
License: Apache v.2.0

http://github.com/dpkp/kafka-python

Python driver with full protocol support and balanced consumer implementation. GZIP and Snappy compression supported.

...

C/C++

-------------

Robust high performance C/C++ library with full protocol support

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

https://github.com/edenhill/librdkafka

Kafka Version: 0.7.x, 0.8.x, 0.9.x, 0.10.x, 0.11.x

Maintainers: Magnus Edenhill

License: 2-clause BSD

-------------

Native C++ library with protocol support for Metadata, Produce, Fetch, and Offset.

Kafka Version: 0.8.x

Maintainer: David Tompkins
License: Apache 2.0

https://github.com/adobe-research/libkafka

-------------

C++ Header-only Kafka Client Library using Boost Asio

Kafka Version: 0.8.x

Maintainer: Daniel Joos
License: MIT

https://github.com/danieljoos/libkafka-asio

-------------

A C++11 asyncronous producer/consumer library for Apache Kafka based on boost asio

Kafka Version: 0.8.x

Maintainer: Svante Karlsson
License: Boost Software License - Version 1.0

https://github.com/bitbouncer/csi-kafka

-------------

libasynckafkaclient - C++ based single threaded asynchronous library

Kafka Version: 0.8.x

Maintainer: Vijay Jadhav
License: 2-clause BSD

https://github.com/GSLabDev/libasynckafkaclient

-------------

https://github.com/quipo/kafka-cpp

Kafka Version: 0.7.x

-------------

Python

-------------

High performance Python client based on Librdkafka with full protocol support.

https://github.com/confluentinc/confluent-kafka-python

Docs: http://docs.confluent.io/current/clients/index.html

Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x

Maintainers: Confluent

License: Apache 2.0

-------------

Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP, LZ4, and Snappy compression supported.

http://github.com/dpkp/kafka-python

Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x, 1.0.x

Maintainer: Dana Powers
License: Apache 2.0

-------------

Python driver with full protocol support, Pythonic API, and managed balanced consumer implementation.
https://github.com/Parsely/pykafka
Kafka Version: 0.8.x,0.9.x,0.10.x
Maintainer: Parse.ly
License: Apache 2.0

-------------

Protocol support for Kafka 0.7 in Python. GZip and Snappy compression supported

Kafka Version: 0.7.x

Maintainer: David Arthur
License: Apache v.2.0

https://github.com/mumrah/kafka-python/tree/0.7

-------------

Also:
https://github.com/dsully/pykafka

Kafka Version: 0.7.x

Maintainer:: Dan Sulley, LinkedIn
License: Apache 2.0

-------------

Python client from Disqus:
https://github.com/getsamsa/samsa

Kafka Version: 0.7.x

Maintainer:: Keith Bourgoin, Parse.ly
License: Apache 2.0

-------------

Python client from Urban Airship: https://github.com/urbanairship/pykafkap

Kafka Version: 0.7.x

-------------

Python client from Datadog: https://github.com/datadog/brod
(Producer, Simple Consumer, ZK-Consumer)

Kafka Version: 0.7.x

-------------

Go (AKA golang)

-------------

Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

Kafka Version: 0.8.x

Maintainer: Shopify
License: MIT

https://github.com/Shopify/sarama

-------------

Enhanced Go Kafka consumer and producer implementations.

Kafka Version: 0.8.x

Maintainer: Big Data Open Source Security
License: Apache v2.0
https://github.com/stealthly/go_kafka_client

-------------

A pure Go implementation of the low level Kafka API.

Kafka Version: 0.8.x

Maintainer: Big Data Open Source Security
License: Apache v2.0

https://github.com/stealthly/siesta

-------------

A fast pure Go Kafka implementation with a clean API.

Kafka Version: 0.8.x

Maintainer: OptioPay
License: MIT

https://github.com/optiopay/kafka

-------------

https://github.com/nuance/kafka

https://github.com/jdamick/kafka.go

Kafka Version: 0.7.x

-------------

confluent-kafka-go: Confluent's Kafka client for Golang wraps the librdkafka C library, providing full Kafka protocol support with great performance and reliability.

The Golang bindings provides a high-level Producer and Consumer with support for the balanced consumer groups of Apache Kafka 0.9 and above.

Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x

Maintainer: Confluent

License: Apache v2.0

https://github.com/confluentinc/confluent-kafka-go

Docs: http://docs.confluent.io/current/clients/index.html

-------------

Erlang

-------------

Kafka client library in Erlang. Full support for 0.9+ consumer protocol, very efficient producer implementation.

https://github.com/klarna/brod

Kafka Version: 0.9.x, 0.10.x

Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)

License: Apache 2.0

-------------

A minimal, high-performance Kafka client in Erlang

https://github.com/helpshift/ekaf

Kafka Version: 0.8.x

Maintainer: Helpshift
License: Apache v2

-------------

erlkafka is a kafka client written in erlang

https://github.com/milindparikh/erlkafka.git

Kafka Version: 0.7.x

Maintainer: Milind Parikh
*License: BSD, LGPL

Also:
https://github.com/wooga/kafka-erlang

.NET

-------------

A fully featured .NET client for Apache Kafka based on librdkafka (a fork of rdkafka-dotnet).

Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x

Maintainer: Confluent Inc. (original author Andreas Heider)

License: Apache 2.0

https://github.com/confluentinc/confluent-kafka-dotnet

-------------

Pure C# client with full protocol support. Includes consumer, producer,

lower level components and gzip support (no snappy)

Kafka Version: 0.8.x

Maintainer: James Roland

License: Apache 2.0

https://github.com/Jroland/kafka-net

-------------

This is a .NET implementation of a client for Kafka using C# for Kafka 0.8. It provides for an implementation that covers most basic functionalities to include a simple Producer and Consumer.

Kafka Version: 0.8.x

Maintainer: ExactTarget

License: Apache 2.0

https://github.com/ExactTargetDev/kafka-net

-------------

.Net implementation of the Apache Kafka Protocol that provides basic functionality through Producer/Consumer classes. The project also offers balanced consumer implementation. The project is a fork from ExactTarget's Kafka-net Client.

Kafka Version: 0.8.x, 0.9.x

Maintainer: Microsoft

License: Apache 2.0

https://github.com/Microsoft/Kafkanet

-------------

C# client, asynchronous, all 3 compressions supported (read and write), tracks leader partition changes transparently, long time in production.

Kafka Version: 0.8.x

Maintainer: Vadim Chekan
License: Apache-2.0
https://github.com/ntent-ad/kafka4net

-------------

kafka-sharp - "High Performance" .NET Kafka Driver

Kafka Version: 0.8.x

Maintainer: Criteo

License: Apache 2.0

https://github.com/criteo/kafka-sharp

-------------

kafunk - Fully-featured, native F# client which works on .NET Core

Kafka Version: 0.8.x, 0.9.x, 0.10.x

Maintainer: Jet.com

License: Apache 2.0

https://github.com/jet/kafunk

-------------

Clojure

-------------

Fast kafka api for JVM languages implemented in clojure.

Kafka Version: 0.8.x

Maintainer: https://github.com/gerritjvv

License: Apache 2.0

https://github.com/

...

Go (AKA golang)

Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

gerritjvv/kafka-fast

-------------

Wrapper to the Java API for interacting with Kafka

Kafka Version: 0.8.x

Maintainer: https://github.com/pingles

License: Apache 2.0

...

https://github.com

...

/pingles/clj-kafka/

-------------

Kafka clojure client library

Enhanced Go Kafka consumer and producer implementations.

Maintainer: Big Data Open Source Security
License: Apache v2.0
https://github.com/stealthly/go_kafka_client

A pure Go implementation of the low level Kafka API.

Maintainer: Big Data Open Source Security
License: Apache v2.0

https://github.com/stealthly/siesta

A fast pure Go Kafka implementation with a clean API.

Maintainer: OptioPay
License: MIT

https://github.com/optiopay/kafka

C/C++

Robust high performance C/C++ library with full protocol support

...

Kafka Version: 0.8.x

Maintainer: Pierre-Yves Ritschard

License: MIT

Code, Documentation

-------------

Ruby

-------------

ruby-kafka - A pure Ruby client library for the Kafka distributed log system.

Kafka Version: 0.9.x, 0.10.x, 0.11.x

Maintainer: Zendesk/ Daniel Schierbeck

Licence: Apache 2.0

https://github.com/

...

Native C++ library with protocol support for Metadata, Produce, Fetch, and Offset.

Maintainer: David Tompkins
License: Apache 2.0

https://github.com/adobe-research/libkafka

C++ Header-only Kafka Client Library using Boost Asio

zendesk/ruby-kafka

-------------

Karafka - Framework used to simplify Apache Kafka based Ruby and Rails applications development. Karafka provides higher abstraction layers, including Capistrano, Docker and Heroku support. Based on ruby-kafka.

Kafka Version: 0.9.x, 0.10.x, 0.11.x

Maintainer: Maciej Mensfeld
Licence

...

: MIT

https://github.com/

...

A C++11 asyncronous producer/consumer library for Apache Kafka based on boost asio

Maintainer: Svante Karlsson
License: Boost Software License - Version 1.0

https://github.com/bitbouncer/csi-kafka

libasynckafkaclient - C++ based single threaded asynchronous library

...

karafka/karafka

-------------

Racecar – A simple framework for writing Kafka consumers in Ruby that integrates nicely with Rails. Based on ruby-kafka.

Kafka Version: 0.9.x, 0.10.x, 0.11.x

Maintainer: Zendesk / Daniel Schierbeck

Licence: Apache 2.0

https://github.com/

...

zendesk/

...

.net

Pure C# client with full protocol support. Includes consumer, producer,

lower level components and gzip support (no snappy)

Maintainer: James Roland

License: Apache 2.0

https://github.com/Jroland/kafka-net

This is a .NET implementation of a client for Kafka using C# for Kafka 0.8. It provides for an implementation that covers most basic functionalities to include a simple Producer and Consumer.

Maintainer: ExactTarget

...

racecar

-------------

DeliveryBoy – A simple way to publish messages to Kafka from Ruby applications. Based on ruby-kafka.

Kafka Version: 0.9.x, 0.10.x, 0.11.x

Maintainer: Zendesk / Daniel Schierbeck

Licence: Apache 2.0

https://github.com/

...

rdkafka-dotnet is a C# client for Apache Kafka based on librdkafka.

Maintainer: Andreas Heider

License: BSD (2 clause)

https://github.com/ah-/rdkafka-dotnet

.Net implementation of the Apache Kafka Protocol that provides basic functionality through Producer/Consumer classes. The project also offers balanced consumer implementation. The project is a fork from ExactTarget's Kafka-net Client.

...

zendesk/delivery_boy

-------------

JRuby wrapper for producers and consumers of the existing API

Kafka Version: 0.8.x

Maintainer: Joseph Lawson
License: Apache 2.0

https://github.com

...

/joekiller/jruby-kafka

-------------

[Unmaintained] Pure Ruby, Consumer and Producer implementations included, GZIP and Snappy compression supported. Ruby 1.9.3 and up (CI runs MRI 2.0, JRuby and Rubinius).

Kafka Version: 0.8.x

Maintainer: Bob Potter
License: MIT

https://github.com/

...

bpot/

...

poseidon

kafka-sharp - "High Performance" .NET Kafka Driver

Maintainer: Criteo

...

-------------

https://github.com/acrosa/kafka-rb

Kafka Version: 0.7.x

Maintainer: Alejandro Crosa
License:: Apache 2.0

-------------

Event machine client:
https://github.com/

...

groupme/em-kafka

...

Clojure

Fast kafka api for JVM languages implemented in clojure

...

Kafka Version: 0.7.x

-------------

JRuby Event stream processor

https://github.com/wooga/

...

License: Apache 2.0

https://github.com/gerritjvv/kafka-fast

Wrapper to the Java API for interacting with Kafka

Maintainer: https://github.com/pingles

...

kafkaesque

Kafka Version: 0.7.x

-------------

Node.js

-------------

The node-rdkafka library is a high-performance NodeJS client for Apache Kafka that wraps the native librdkafka library. All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library.

https://github.com/

...

Blizzard/

...

Kafka clojure client library

Maintainer: Pierre-Yves Ritschard

License: MIT

Code, Documentation

Ruby

Pure Ruby, Consumer and Producer implementations included, GZIP and Snappy compression supported. Ruby 1.9.3 and up (CI runs MRI 2.0, JRuby and Rubinius).

Maintainer: Bob Potter
License: MIT

https://github.com/bpot/poseidon

Karafka - Microframework used to simplify Apache Kafka-based Ruby applications development.

...

node-rdkafka

Kafka Version: 0.9, 0.10

Node.js >= 4

Maintainer: Blizzard.com

License: MIT

-------------

Kafka-Node is a NodeJS client with Zookeeper integration

Kafka Version: 0.8.x

Maintainer: sohu.com

License: MIT

https://github.com

...

/SOHU-Co/kafka-node/

...

-------------

Kafka-node is a pure JavaScript implementation for NodeJS Server with Vagrant and Docker support.

Kafka Version: 0.8.x

Maintainer:

...

wurstmeister
License: Apache 2.0

https://github.com/

...

wurstmeister/

...

node-kafka-0.8-plus

-------------

Node

...

Kafka-Node is a NodeJS client with Zookeeper integration

-kafka is a node.js wrapper for the C library librdkafka

Kafka Version: 0.8.x

Maintainer: Sutoiku

...

License: MIT

https://github.com

...

Kafka-node is a pure JavaScript implementation for NodeJS Server with Vagrant and Docker support.

...

/sutoiku/node-kafka

-------------

kafka-java-bridge is a Nodejs wrapper for the JAVA high level kafka 0.8. consumer API

Kafka Version: 0.8.x

Maintainer: LivePersonInc

License: MIT

https://

...

Node-kafka is a node.js wrapper for the C library librdkafka

Maintainer: Sutoiku

License: MIT

www.npmjs.com

...

/package/kafka-java-bridge

...

...

...

kafka-java-bridge is a Nodejs wrapper for the JAVA high level kafka 0.8. consumer API

Maintainer: LivePersonInc

-------------

Low-level protocol support in node.js.

...

https://

...

github.

...

com/

...

cainus/Prozess
https://

...

npmjs.

...

org/

...

Alternative Java Clients

Of course the main project maintains a set of jvm-based clients. But here are alternate clients.

Krackle is an optimized Kafka client built by Blackberry.

Maintainer: Blackberry

...

Storm

Port of Apache storm-kafka to >= 0.8.0

Maintainer: wurstmeister
License: Apache 2.0

package/prozess

Kafka Version: 0.7.x

Maintainers:
Gregg Caines
Eric lee
License: MIT

-------------

Alternate node client from Tagged
https://github.com/

...

marcuswestin/node-kafka

Also:

https://github.com

...

/dannycoates/franz-kafka

Kafka Version: 0.7.x

-------------

Proxy (HTTP REST, etc)

-------------

Dory is a producer daemon that supports clients in various programming languages. Clients send messages to Dory using local interprocess communication. Dory then takes full responsibility for reliable message delivery. Monitoring infrastructure can query Dory's web interface for JSON-based status and data quality reports.

...

Scala DSL

A DSL for Scala developers to produce and consume messages with Kafka.

Maintainer: Big Data Open Source Security LLC
License: Apache 2.0

https://github.com/stealthly/scala-kafka

HTTP REST

Apache Kafka HTTP Endpoint for producing and consuming messages from topics

Maintainer: Big Data Open Source Security LLC
License: Apache 2.0

https://github.com/stealthly/dropwizard-kafka-http

(Deprecated) Kafka high level Producer and Consumer APIs are very hard to implement right. Rest endpoint gives access to native Scala high level consumer and producer APIs.

Maintainer: Sasha Klizhentas
License: Apache 2.0

https://github.com/mailgun/kafka-http

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.

dspeterson/dory

Kafka Version: 0.8.x, 0.9.x, 0.10.x, 0.11.x

Maintainer: Dave Peterson
License: Apache v.2.0

-------------

Apache Kafka HTTP Endpoint for producing and consuming messages from topics

Kafka Version: 0.8.x

Maintainer: Big Data Open Source Security LLCMaintainer: Confluent
License: Apache 2.0

...

...

...

...

...

...

dropwizard-kafka-http

-------------

(Deprecated) Kafka high level Producer and Consumer APIs are very hard to implement right. Rest endpoint gives access to native Scala high level consumer and producer APIs.

Maintainer: Sasha Klizhentas

Kafka-Pixy is a local aggregating HTTP proxy to Kafka messaging cluster.

Maintainer: Mailgun
License: Apache 2.0

https://github.com/mailgun/kafka-pixyhttp

Efficient Kafka REST Proxy for producers

https://github.com/klarna/kastle

Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)
License: Apache 2.0

JRuby

JRuby wrapper for producers and consumers of the existing API

Maintainer: Joseph Lawson
License: Apache 2.0

-------------

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster.

Kafka Version: 0.8.x, 0.9.x, 0.10.x

Maintainer: Confluent
License: Apache 2.0

...

confluentinc/

...

kafka-

...

Perl

rest

-------------

Kafka-Pixy is a local aggregating HTTP proxy to Kafka messaging cluster.

Kafka Version: 0.8.x

Maintainer: Mailgun
License: Apache 2.0Pure Perl, Consumer and Producer implementations included. Zookeeper
integration. GZIP and Snappy compression supported.
Maintainer: Sergey Gladkov
License: Artistic License

https://github.com/TrackingSoft/Kafka
http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm

stdin/stdout

mailgun/kafka-pixy

-------------

Efficient Kafka REST Proxy for producersGeneric producer and consumer for stdin and stdout

https://github.com/edenhillklarna/kafkacatkastle

Maintainer: Magnus EdenhillLicense: 2-clause BSD

Erlang

A minimal, high-performance Kafka client in Erlang

https://github.com/helpshift/ekaf

Maintainer: Helpshift
License: Apache v2

Kafka Version: 0.8.x

Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)
License: Apache 2.0

-------------

Full featured HTTP REST Proxy for Kafka in Elixir with fine grained JWT-based authenticationKafka client library in Erlang. Full support for 0.9+ consumer protocol, very efficient producer implementation.

https://github.com/klarna/brodkastlex

Kafka Version: 0.8.x

Maintainers: Ivan Dyachkov (Klarna AB), Shi Zaiming (Klarna AB)
License: Apache 2.0

PHP

PHP extension built on librdkafka

.0

Perl

-------------

Pure Perl, Consumer and Producer implementations included. Zookeeper
integration. GZIP and Snappy compression supported.

Kafka Version: 0.8.x
Maintainer: Sergey Gladkov
License: Artistic LicenseMaintainer: Elias Van Ootegem
License: MIT

https://github.com/EVODelavega/phpkafka