Java

From Thrift

Avro's specific implementation can be very similar to thrift's IDL / generation paradigm. For an example of project that simultaneously supports both Java and Thrift, see Flume http://github.com/cloudera/flume/tree/master/src/ which has both Avro and Thrift Sources.

Source files

Here we see an avro IDL file and a thrift source file that represent the same RPC service.

flumeconfig.avdl
@namespace("com.cloudera.flume.conf.avro")

protocol FlumeReportAvroServer{
 enum FlumeNodeState { HELLO, IDLE, CONFIGURING,
    ACTIVE, ERROR, LOST, DECOMMISSIONED }
 record AvroFlumeConfigData {
    long timestamp;
    string sourceConfig;
    string sinkConfig;
    long sourceVersion;
    long sinkVersion;
    string flowID;
 }
 record FlumeReportAvro {
    map<string> stringMetrics;
    map<long> longMetrics;
    map<double> doubleMetrics;
 }

 boolean heartbeat(string logicalNode, string physicalNode, string host,
    FlumeNodeState s, long timestamp);
 union { AvroFlumeConfigData, null } getConfig(string physNode);
 array<string> getLogicalNodes(string physNode);
 void acknowledge(string ackid);
 boolean checkAck(string ackid);
 void putReports(map<FlumeReportAvro> reports);
}
flumeconfig.thrift
namespace java com.cloudera.flume.conf.thrift

typedef i64 Timestamp

 enum FlumeNodeState {
    HELLO =0,
    IDLE =1,
    CONFIGURING =2,
    ACTIVE=3,
    ERROR =4,
    LOST=5,
    DECOMMISSIONED=6
 }

 struct ThriftFlumeConfigData {
    1: Timestamp timestamp,
    2: string sourceConfig,
    3: string sinkConfig,
    4: i64 sourceVersion,
    5: i64 sinkVersion,
    6: string flowID
 }

 struct FlumeReport {
    3: map<string, string> stringMetrics,
    4: map<string, i64> longMetrics,
    5: map<string, double> doubleMetrics
 }

service FlumeClientServer {
 bool heartbeat(1:string logicalNode, 4:string physicalNode, 5:string host, 2:FlumeNodeState s, 3:i64 timestamp),
 ThriftFlumeConfigData getConfig(1:string sourceId),
 list<string> getLogicalNodes(1: string physNode),
 void acknowledge(1:string ackid),
 bool checkAck(1:string ackid),
 void putReports(1:map<string, FlumeReport> reports)
}

Two key differences are that fields are not numbered in Avro in the same way they are numbered in Thrift and that maps are only allowed to have String keys, so they just require one parameter when defined.

Building clients and servers

On the client side, Avro generates a client class against which you can make RPC calls. This example shows how to instantiate a client that request over HTTP transport.

  URL url = new URL("http", SERVER_HOST, SERVER_PORT, "/");
  trans = new HttpTransceiver(url);
  FlumeReportAvroServer masterClient = (FlumeReportAvroServer) 
    SpecificRequestor.getClient(FlumeReportAvroServer.class, trans);

This is a similar, but not exactly equivalent thrift code segment:

  TTransport masterTransport = new TSocket(SERVER_HOST, SERVER_PORT);
  TProtocol protocol = new TBinaryProtocol(masterTransport);
  masterTransport.open();
  FlumeClientServer.Iface masterClient = new Client(protocol);

On the server side, Avro creates an interface (similar to Thrift) that your server must implement. This will contain the method signatures from your IDL file.

public class MasterClientServerAvro implements
  FlumeReportAvroServer {

In thrift we have:

public class MasterClientServerThrift extends ThriftServer implements
    FlumeClientServer.Iface {
Nonstandard data types

Avro has some quirky data-types that will cause hiccups if directly copying your thrift code.

Utf8 In older versions of Avro's (1.3.3 and earlier), function signatures that involve strings use Utf8() not String(). Your client and server implementations will expect to pass and receive Utf8() instances, so you will need to translate this type to and from String on your own.

Arrays If your function accepts or returns an Array type, you cannot simply pass a Java array. Instead, it will expect an implementing class of Avro's own GenericArray. To create an array of Strings, for example, use 

GenericArray<Utf8> out = new GenericData.Array<Utf8>(
        str.size(), Schema.createArray(Schema.create(Type.STRING)));
  • No labels