I am working on a proposal to satisfy the Restful Proxy project listed on the Projects page. (http://incubator.apache.org/kafka/projects.html)
My working knowledge of Kafka is about 3 hours old and I have actually written zero code yet, it is my job to do so for Box so I will update this as my knowledge increases. There may be some fundamental issues with this proposal at this time.
General Idea
Create a Restful API which will allow producers and consumers to communicate with a Kafka server via HTTP calls
Consumers
URL Format:
GET /messages/<topic>/<consumer>/<offset> - Retrieve the messages for a consumer, from the topic, starting with an offset
DELETE /messages/<topic>/<consumer>/<offset> - ACK consumption of message for a consumer, topic, offset
Producers
PUT /messages/<topic>
Request Body will become the message
Known Missing Items:
- I know that this does not take into account partitions.
- It would be nice to be able to implement the async producer and have an HTTP callback happen from within the RESTServer (producer specifies a URL to be requested as callback)
- If an HTTP call back wasn't available, have the service return a task ID (MD5 of the message or something), and then allow the client to query the server to see if the callback from Kafka has happened yet.