Child pages
  • Clients
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 42 Next »

How The Kafka Project Handles Clients

Starting with the 0.8 release we are maintaining all but the jvm client external to the main code base. The reason for this is that it allows a small group of implementers who know the language of that client to quickly iterate on their code base on their own release cycle. Having these maintained centrally was becoming a bottleneck as the main committers can't hope to know every possible programming language to be able to perform meaningful code review and testing. This lead to a scenario where the committers were attempting to review and test code they didn't understand.

We are instead moving to the redis/memcached model which seems to work better at supporting a rich ecosystem of high quality clients.

We haven't tried all these clients and can't vouch for them. The normal rules of open source apply.

If you are aware of other clients not listed here (or are the author of such a client), please add it here. Please also feel free to help fill out information on the features the client supports, level of activity of the project, level of documentation, etc.

For 0.8.x

The 0.8 release changes the protocol fairly substantially. This version hasn't been released yet. The new protocol is documented here. A number of clients are in progress, and we will update when they are complete.

Python

Pure Python implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

Maintainer: David Arthur
License: Apache v.2.0

https://github.com/mumrah/kafka-python

Go (AKA golang)

Pure Go implementation with full protocol support. Consumer and Producer implementations included, GZIP and Snappy compression supported.

License: MIT

https://github.com/Shopify/sarama

C

High performance C library with full protocol support

https://github.com/edenhill/librdkafka

Maintainer: Magnus Edenhill
License: 2-clause BSD

C++

Native C++ library with protocol support for Metadata, Produce, Fetch, and Offset.

https://github.com/adobe-research/libkafka

Maintainer: David Tompkins
License: Apache 2.0

Clojure

https://github.com/pingles/clj-kafka/ 0.8 branch

Ruby

Pure Ruby, Consumer and Producer implementations included, GZIP and Snappy compression supported. Ruby 1.9.3 and up (CI runs MRI 2.0, JRuby and Rubinius).

Maintainer: Bob Potter
License: MIT

https://github.com/bpot/poseidon

Node.js

Kafka-Node is a NodeJS client with Zookeeper integration

Maintainersohu.com

License: MIT

https://github.com/SOHU-Co/kafka-node/

Kafka-node is a pure JavaScript implementation for NodeJS Server with Vagrant and Docker support.

Maintainerwurstmeister
License: Apache 2.0

https://github.com/wurstmeister/node-kafka-0.8-plus

Node-kafka is a node.js wrapper for the C library libkafkard.

Maintainer: Sutoiku

License: MIT

https://github.com/sutoiku/node-kafka

Storm 

Port of Apache storm-kafka to >= 0.8.0

Maintainerwurstmeister
License: Apache 2.0

https://github.com/wurstmeister/storm-kafka-0.8-plus

Scala DSL 

A DSL for Scala developers to produce and consume messages with Kafka.  

MaintainerBig Data Open Source Security LLC
License: Apache 2.0

https://github.com/stealthly/scala-kafka

HTTP REST 

Apache Kafka HTTP Endpoint for producing and consuming messages from topics

MaintainerBig Data Open Source Security LLC
License: Apache 2.0

https://github.com/stealthly/dropwizard-kafka-http

Maintainer: Sasha Klizhentas
License: Apache 2.0

https://github.com/mailgun/kafka-http

JRuby 

JRuby wrapper for producers and consumers of the existing API

Maintainer: Joseph Lawson
License: Apache 2.0

https://github.com/joekiller/jruby-kafka

Perl

Pure Perl, Consumer and Producer implementations included. Zookeeper
integration. GZIP and Snappy compression not supported.

Maintainer: Sergey Gladkov, <sgladkov@trackingsoft.com>
License: Artistic License

https://github.com/TrackingSoft/Kafka
http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm

stdin/stdout

Generic producer and consumer for stdin and stdout

https://github.com/edenhill/kafkacat

Maintainer: Magnus Edenhill
License: 2-clause BSD

0.1-0.7.x

Java

Maintained with the main code base.

C

Native C library with Consumer, Producer and compression support.

https://github.com/edenhill/librdkafka

Maintainer: Magnus Edenhill
License: 2-clause BSD

C++

https://github.com/quipo/kafka-cpp

PHP

PHP library with Consumer (simple and Zookeeper-based), Producer and compression support (release notes).

https://github.com/quipo/kafka-php

Maintainer: Lorenzo Alberton
License: Apache v.2.0

Also:

https://github.com/michal-harish/kafka-php

Log4PHP Appender

https://github.com/dastra/log4php-kafka

Python

We have a wealth of python clients!

Protocol support for Kafka 0.7 in Python. GZip and Snappy compression supported

Maintainer: David Arthur
License: Apache v.2.0

https://github.com/mumrah/kafka-python/tree/0.7

Also:
https://github.com/dsully/pykafka
Maintainer:: Dan Sulley, LinkedIn
License: Apache 2.0

Python client from Disqus:
https://github.com/getsamsa/samsa
Maintainer:: Keith Bourgoin, Parse.ly
License: Apache 2.0

Python client from Urban Airship: https://github.com/urbanairship/pykafkap

Python client from Datadog: https://github.com/datadog/brod
(Producer, Simple Consumer, ZK-Consumer)

Ruby

https://github.com/acrosa/kafka-rb
Maintainer: Alejandro Crosa
License:: Apache 2.0

Event machine client:
https://github.com/groupme/em-kafka

JRuby Event stream processor
https://github.com/wooga/kafkaesque

Clojure

https://github.com/pingles/clj-kafka

https://github.com/miniway/kafka-clj

Go

https://github.com/nuance/kafka

https://github.com/jdamick/kafka.go

Node.js

Low-level protocol support in node.js.

https://github.com/cainus/Prozess
https://npmjs.org/package/prozess
Maintainers:
Gregg Caines
Eric lee
License: MIT

Alternate node client from Tagged
https://github.com/marcuswestin/node-kafka

Also:
https://github.com/dannycoates/franz-kafka

Erlang

erlkafka is a kafka client written in erlang

https://github.com/milindparikh/erlkafka.git

Maintainer: Milind Parikh
*License: BSD, LGPL

Also:
https://github.com/wooga/kafka-erlang

Client Libraries Previously Supported

https://svn.apache.org/repos/asf/incubator/kafka/branches/legacy_client_libraries

  • No labels