New Wire Format Proposal

Some comments below:

1. coorelation id: Not sure if we need this. It seems that it's simpler if we allow each client to make just 1 outstanding request. If so, we don't need a coorelation id.
2. max_wait/min_size: I can definitely see that max_wait is very useful. There are a couple of problems with min_size. First, it's a bit hard to implement, especially with multifetch. Second, it's not clear how useful it is. I feel it's much simpler for consumers to just reason about delay in time.
3. required_acks: For generality, we need this field. However, I suggest that we use 0 to mean waiting for acks from all replicas that are currently synced up. The async behavior probably doesn't need a response.
4. We will need a way to distinguish btw a FetchRequest made by a follower replica and that made by a consumer. One way to do that is to add a flag field in the fetch request. By default, the flag is 0. The follower replica can set the flag to a non-zero value.
5. For a multifetch request, currently there is an error code per response, in addition to the error code for the whole multifetch. We can probably do the same thing for multiproduce.
6. About removing fetch and keeping only multifetch, one minor concern is that sync produce makes most sense on a single topic (probably with a single message too). That's probably the main use case when a response needs to be sent to the producer. If we have both fetch and multifetch, one thing that we can do is to only allow fetch to send data synchronously and get a response back.

Additional comments:

Agree with Jun on points 1, 2 and 3. Regarding point 4 Jun, I'm not sure I see the distinction between a replica follower and a regular consumer. Followers only fetch when catching up to the primary replica, this behaviour is the same as an old-fashioned consumer, no? Maybe I'm missing something.

One point of my own is:

• The new ProduceRequest has required_acks at the top level, meaning that the entire request needs an ack from that many brokers. What if the replica_count for some topic "A" in the request has < required_ack replicas but topic "B" meets the requirement?

Interesting stuff !

Two comments:

1. If we do end up with a correlation ID, shouldn't it be included in the FetchRequest? Currently, it's in the FetchResponse but not in the request, which I think doesn't really make sense...
2. A general question about required_acks and replication in general (sorry if it's off-topic...). What is supposed to happen if, for example, a topic is expected to have a replication factor of 3 but only 2 brokers are able to record a given message (the original one + 1 replica)? I guess the ProduceResponse should include an error that indicates that the ProduceRequest has failed, but will the two replicas be deleted from the brokers that did record the message successfully? It might be tricky to ensure those deletions while maintaining a high-throughput ... The problem is that if those messages are not deleted from the 2 replicas that have them then the Producer might retry sending the same message again and the brokers would then contain duplicate messages...

Felix,

1. It's actually there. Just happen to be in the same line as the previous field. I fixed this.
2. That's a good question. If the client insists on 3 successful writes from each of the 3 replicas but only 2 are successful, I think the contract should be the following. The producer request will return with an error code. This just means that the message is published, but under replicated. The under replicated messages will be copied to other replicas over time (when brokers are up). The producer, when getting an error, can take appropriate actions on its own, such as stopping sending new requests or issuing a warning, etc.

3. For required_acks, how about we use -1 to designate no ack and 0 to designate ack from all current in-sync replicas? A positive value will designate the # of replicas with successful writes. Related to that, if an ack is needed, do we need to specify the timeout in the request?

Jun--

1. The correlation id enables the ability of the client to multiplex requests, it doesn't require that. I don't recommend we implement this yet in the scala client BUT we may well want to do this. Once we add produce requests that block on other replicas this will become somewhat slow, so tying up a full connection is not desirable. We will have to see how this plays out, but adding this now is just enabling this style of client.

2. The goal of min_size is to allow high throughput with low latency. Currently polling on a topic getting a continuous trickle of messages will lead to very small fetches. To work around this the only ability the consumer has is to basically sleep for some guessed period of time and hope that some data accumulates. But setting this sleep time correctly is impossible. This is a way to allows the consumer to precisely specify the size they want and the minimum timeout. This does add to implementation complexity so I am open to the idea of passing on this, but I think it would be nice to have.

3. Required_acks--isn't it a bit unintuitive to have 0 acks mean all in_sync replicas and not 0 acks are required? Here is how I see it, you really need to specify a timeout for the acknowledgement and the number of acknowledgements you want. I don't think your replication requirement is likely to depend on whether replicas are in sync or not.

4. I agree we need a replica_id field in the fetch request which would be set to -1 for consumers outside the cluster.

5. I agree we need both per-fetch error code and a global. I can't think of a use for the global error right now, though, can anyone else?

6. I agree that the multi-* requests are less intuitive than the single topic/partition variety. I think the idea here, though, is to make the network-level api as general as possible even if it is less intuitive.

Jun--The timeout should definitely, definitely be on the client not the server. Different applications have different latency requirements and we definitely don't want to get into the business of tuning this for each user.

Felix--Added the correlation id to FetchRequest, that was a typo.

Jun, Felix--The contract should definitely be just that we could not meet the replication requirements you gave but at least the master got the message. If a replica times out or crashes it is impossible for us to know if this replica got the message or not. Ideally we should tell the client sorry, we could only replicate to N replicas which is less than the M replicas you required, but right now we don't have a way to send back N so it might be fine just to say that we couldn't meet the required replicas in the time limit.

One more thing I forgot that I think we should add is an optional client id. This would make it easier to correlate requests, exceptions, and metrics collected on the server side with the client that sent the request. We can do this now on a per-connection basis, but it can be a huge pain to correlate connections to applications. This token would allow that.

I will update the proposal with the following:

• global error
• producer timeout
• replica id
• client id

Question regarding the FetchRequest format, specifically in the topic_offset field. Can we model the offset_data structure like the topic_data_struct/partition_data_struct of the produce requests? Though I realize introducing another new data structure adds complexity, I feel it adds clarity; rather than matching an offset to a partition based purely on indices into the array, that information is captured in a trivial class (or maybe even a simple tuple). It also protects against any potential transmission problems in case the offset or partition array get's truncated. Just a thought, I'm open to discuss

Regarding acking in ProduceRequest; it would probably need a similar field in the ProduceResponse to reflect the number of actual succesful acks.

  errors: [int16] // per-partition errors, one for each message set sent (or all -1 if none)
  offsets: [int64] // the offsets for each off the message sets supplied, in the order given in the request
  acks: [int16] // per-partition ack counts, one for each message set sent, or -1 if acks were not requested.

This goes in line with previous discussions here on what to do when the desired number of acks cannot be satisfied (including timeout) and delegates the decision to the producer application, where it belongs.

So typical behaviour of the producer would perhaps be:

   if (actual_ackcount == 0)
     warn_and_attempt_resend
   else if (actual_ackcount < desired_ackcount)
     warn_but_proceed_with_next

Regarding client_id:

Sending the client_id in every message is a bit excessive since it shouldnt change during the lifetime of a connection (right?).
How about having an (optional) Hello/Register message sent on the initialization of the connection?

Something along the lines of:

    Hello {
       version: int16            // protocol version, no need to duplicate that either, if Hello is required, but that might not be desirable.

       client_id: string         // "weblogconsumer4/pid19242@backend19.company.com", host uuid, etc..
       replica_id: string        // ...?

       (and possibly, for administrative convenience:)
       softwareversion: string   // "webloganalyst 1.2.0")
       
    }
    HelloReply {
       broker_id: string         // "broker4/28923@hub.company.com", ..
       (...possibly..)
       softwareversion: string   // "kafka 0.8.1"

    }

For simplicity a Hello message must only be accepted as the first message on a new connection, and not sending a Hello messages makes the broker generate its own unique client_id ("192.168.0.3:23593", ...).

If the Hello handshake is made to be required then the version_id from the generic respone and reply headers could be moved here instead. Thus only checking the version once for each connection. Albeit checking a int16 for each request isnt that much work in the context.

Regardless of Hello message or not; what if the client_id collides with an existing connection?

• assume the new connection is from a new instance of the same application, replacing a malfunct/dying connection that is yet to be torn down by TCP timeout. This closes the old connection.
• or, reject the new connection, probably not preferable because of the point above.
• append something unique to the client_id and let both connections persist.

i kind of like keeping the client_id on each request. if we wanted to have an encrypted transmission over the socket the client_id value can be turned into the symmetrically encrypted data's "password" in the transmission which would could encrypt with the first client id (with certificates and private keys) it makes a lot of possibilities in implementation, lots of good uses.

too much discussion of conditionals if we wanted to-do hold off until 0.8.1 or 0.9 or however we wanted to plan for releases ahead -> better to have discussion on JIRA

Hey Magnus,

Moving some fields into a connection handshake would certainly be possible. That is true of the versioning and the client id. I don't have a strong opinion either way. As an optimization it is probably not major. The only downside is that the versioning information has to be maintained with the connection which would somewhat couple the network layer and the api layer.

Our intention for client_id was not actually a unique per-connection identifier, which I think TCP already provides. Rather the problem we have right now is that in a highly multi-tenant environment it is difficult to aggregate metrics or trace errors to a logical application. The client id was meant to be a logical identifier potentially used across many servers, e.g. "foobar-front-end". This then becomes an aggregation level for metrics, quotas, as well as being helpful in error reporting.

I could go either way on returning the ack count. I think the important distinction from the client pov, though, is whether the message is (1) known to have been committed, (2) written at least once but not to sufficient servers to be guaranteed, (3) definitely not written, or (4) unknown. Case (4) would be in the case of a socket timeout or network error in which no response is received. We should think through if we have correctly captured at least these cases.