Restful API Proposal
Skip to end of metadata
Go to start of metadata

I am working on a proposal to satisfy the Restful Proxy project listed on the Projects page. (http://incubator.apache.org/kafka/projects.html)

My working knowledge of Kafka is about 3 hours old and I have actually written zero code yet, it is my job to do so for Box so I will update this as my knowledge increases. There may be some fundamental issues with this proposal at this time.

General Idea

Create a Restful API which will allow producers and consumers to communicate with a Kafka server via HTTP calls

Consumers

URL Format:

GET /messages/<topic>/<consumer>/<offset> - Retrieve the messages for a consumer, from the topic, starting with an offset

DELETE /messages/<topic>/<consumer>/<offset> - ACK consumption of message for a consumer, topic, offset

Producers

PUT /messages/<topic>

Request Body will become the message

Known Missing Items:

  • I know that this does not take into account partitions.
  • It would be nice to be able to implement the async producer and have an HTTP callback happen from within the RESTServer (producer specifies a URL to be requested as callback)
  • If an HTTP call back wasn't available, have the service return a task ID (MD5 of the message or something), and then allow the client to query the server to see if the callback from Kafka has happened yet.
Labels
  • No labels
  1. This is great. A few comments:

    • I am not much of a REST person, but maybe the PUT should be a POST since it appends?
    • The PUT/POST will definitely need to support multiple messages in a single batch for efficiency. This could be done either with HTTP multipart perhaps or just by allowing one to post a full message set.
    • Another approach to the API would be to have the consumer stream messages to the client. This might be more natural.
    • It is worth considering whether offset management should be exposed in this api or hidden, or perhaps both depending on the options given.
    • It would be good to allow plugin APIs for optional serialization and deserialization by the REST server.
  2. Hello, did you reach any results in your KAFKA HTTP REST API proposal?