Bindy
Available as of Camel 2.0
The idea that the developers has followed to design this component was to allow the parsing/binding of non structured data (or to be more precise non-XML data)
to Java Bean using annotations. Using Bindy, you can bind data like :
- CSV record,
- Fixedlength record,
- FIX messages,
- or any other non-structured data
to one or many Plain Old Java Object (POJO) and to convert the data according to the type of the java property. POJO can be linked together and relation one to many is available in some cases. Moreover, for data type like Date, Double, Float, Integer, Short, Long and BigDecimal, you can provide the pattern to apply during the formatting of the property.
For the BigDecimal number, you can also define the precision and the decimal or grouping separators
Type |
Format Type |
Pattern example |
Link |
---|---|---|---|
Date |
DateFormat |
"dd-MM-yyyy" |
http://java.sun.com/j2se/1.5.0/docs/api/java/text/SimpleDateFormat.html |
Decimal* |
Decimalformat |
"##.###.###" |
http://java.sun.com/j2se/1.5.0/docs/api/java/text/DecimalFormat.html |
Decimal* = Double, Integer, Float, Short, Long
Format supported
This first release only support comma separated values fields and key value pair fields (e.g. : FIX messages).
To work with camel-bindy, you must first define your model in a package (e.g. com.acme.model) and for each model class (e.g. Order, Client, Instrument, ...) associate the required annotations (described hereafter) with Class or property name.
Annotations
The annotations created allow to map different concept of your model to the POJO like :
- Type of record (csv, key value pair (e.g. FIX message), fixed length ...),
- Link (to link object in another object),
- DataField and their properties (int, type, ...),
- KeyValuePairField (for key = value format like we have in FIX financial messages),
- Section (to identify header, body and footer section),
- OneToMany
This section will describe them :
1. CsvRecord
The CsvRecord annotation is used to identified the root class of the model. It represents a record = a line of a CSV file and can be linked to several children model classes.
Annotation name |
Record type |
Level |
---|---|---|
CsvRecord |
csv |
Class |
Parameter name |
type |
Info |
---|---|---|
separator |
string |
mandatory - can be ',' or ';' or 'anything' |
skipFirstLine |
boolean |
optional - default value = false - allow to skip the first line of the CSV file |
crlf |
string |
optional - default value = WINDOWS - allow to define the carriage return character to use |
generateHeaderColumns |
boolean |
optional - default value = false - uses to generate the header columns of the CSV generates |
isOrdered |
boolean |
optional - default value = false - allow to change the order of the fields when CSV is generated |
|
|
This annotation is associated to the root class of the model and must be declared one time. |
case 1 : separator = ','
The separator used to segregate the fields in the CSV record is ',' :
10, J, Pauline, M, XD12345678, Fortis Dynamic 15/15, 2500, USD,08-01-2009
@CsvRecord( separator = "," ) public Class Order { ... }
case 2 : separator = ';'
Compare to the previous case, the separator here is ';' instead of ',' :
10; J; Pauline; M; XD12345678; Fortis Dynamic 15/15; 2500; USD; 08-01-2009
@CsvRecord( separator = ";" ) public Class Order { ... }
case 3 : separator & skipfirstline
The feature is interesting when the client wants to have in the first line of the file, the name of the data fields :
order id, client id, first name, last name, isin code, instrument name, quantity, currency, date
To inform bindy that this first line must be skipped during the parsing process, then we use the attribute :
@CsvRecord(separator = ",", skipFirstLine = true) public Class Order { ... }
case 4 : generateHeaderColumns
To add at the first line of the CSV generated, the attribute generateHeaderColumns must be set to true in the annotation like this :
@CsvRecord( generateHeaderColumns = true ) public Class Order { ... }
As a result, Bindy during the unmarshaling process will generate CSV like this :
order id, client id, first name, last name, isin code, instrument name, quantity, currency, date
10, J, Pauline, M, XD12345678, Fortis Dynamic 15/15, 2500, USD,08-01-2009
case 5 : carriage return
If the platform where camel-bindy will run is not Windows but Macintosh or Unix, than you can change the crlf property like this. Three values are available : WINDOWS, UNIX or MAC
@CsvRecord(separator = ",", crlf="MAC") public Class Order { ... }
case 6 : isOrdered
Sometimes, the order to follow during the creation of the CSV record from the model is different from the order used during the parsing. Then, in this case, we can use the attribute isOrdered = true to indicate this in combination with attribute 'position' of the DataField annotation.
@CsvRecord(isOrdered = true) public Class Order { @DataField(pos = 1, position = 11) private int orderNr; @DataField(pos = 2, position = 10) private String clientNr; ... }
Remark : pos is used to parse the file, stream while positions is used to generate the CSV
2. Link
The link annotation will allow to link objects together.
Annotation name |
Record type |
Level |
---|---|---|
Link |
all |
Class & Property |
Parameter name |
type |
Info |
---|---|---|
linkType |
LinkType |
optional - by default the value is LinkType.oneToOne - so you are not obliged to mention it |
|
|
Only one-to-one relation is allowed. |
e.g : If the model Class Client is linked to the Order class, then use annotation Link in the Order class like this :
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @Link private Client client; ...
AND for the class Client :
@Link public class Client { ... }
3. DataField
The DataField annotation defines the property of the field. Each datafield is identified by its position in the record, a type (string, int, date, ...) and optionally of a pattern
Annotation name |
Record type |
Level |
---|---|---|
DataField |
all |
Property |
Parameter name |
type |
Info |
---|---|---|
pos |
int |
mandatory - digit number starting from 1 to ... |
pattern |
string |
optional - default value = "" - will be used to format Decimal, Date, ... |
length |
int |
optional - represents the length of the field for fixed length format |
precision |
int |
optional - represents the precision to be used when the Decimal number will be formatted/parsed |
pattern |
string |
optional - default value = "" - is used by the Java Formater (SimpleDateFormat by example) to format/validate data |
position |
int |
optional - must be used when the position of the field in the CSV generated must be different compare to pos |
required |
boolean |
optional - default value = "false" |
case 1 : pos
This parameter/attribute represents the position of the field in the csv record
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @DataField(pos = 5) private String isinCode; ... }
As you can see in this example the position starts at '1' but continues at '5' in the class Order. The numbers from '2' to '4' are defined in the class Client (see here after).
public class Client { @DataField(pos = 2) private String clientNr; @DataField(pos = 3) private String firstName; @DataField(pos = 4) private String lastName; ... }
case 2 : pattern
The pattern allows to enrich or validates the format of your data
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @DataField(pos = 5) private String isinCode; @DataField(name = "Name", pos = 6) private String instrumentName; @DataField(pos = 7, precision = 2) private BigDecimal amount; @DataField(pos = 8) private String currency; @DataField(pos = 9, pattern = "dd-MM-yyyy") -- pattern used during parsing or when the date is created private Date orderDate; ... }
case 3 : precision
The precision is helpful when you want to define the decimal part of your number
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @Link private Client client; @DataField(pos = 5) private String isinCode; @DataField(name = "Name", pos = 6) private String instrumentName; @DataField(pos = 7, precision = 2) -- precision private BigDecimal amount; @DataField(pos = 8) private String currency; @DataField(pos = 9, pattern = "dd-MM-yyyy") private Date orderDate; ... }
case 4 : Different position in output
The position attribute will inform bindy how to place the field in the CSV record generated. By default, the position used corresponds to the position defined with the attribute 'pos'. If the position is different (that means that we have an asymetric processus comparing marshaling from unmarshaling) than we can use 'position' to indicate this.
Here is an example
@CsvRecord(separator = ",") public class Order { @DataField(pos = 1) private int orderNr; @DataField(pos = 5) private String isinCode; @DataField(name = "Name", pos = 6) private String instrumentName; @DataField(pos = 7, precision = 2) -- precision private BigDecimal amount; @DataField(pos = 8) private String currency; @DataField(pos = 9, pattern = "dd-MM-yyyy") private Date orderDate; ... }
This attribute of the annotation @DataField must be used in combination with attribute isOrdered = true of the annotation @CsvRecord
4. Message
The Message annotation is used to identified the class of your model who will contain key value pairs fields. This kind of format is used mainly in Financial Exchange Protocol Messages (FIX). Nevertheless, this annotation can be used for any other format where data are identified by keys. The key pair values are separated each other by a separator which can be a special character like a tab delimitor (unicode representation : \u0009) or a start of heading (unicode representation : \u0001)
"FIX information"
More information about FIX can be found on this web site : http://www.fixprotocol.org/. To work with FIX messages, the model must contain a Header and Trailer classes linked to the root message class which could be a Order class. This is not mandatory but will be very helpful when you will use camel-bindy in combination with camel-fix which is a Fix gateway based on quickFix project http://www.quickfixj.org/.
Annotation name |
Record type |
Level |
---|---|---|
Message |
key value pair |
Class |
Parameter name |
type |
Info |
---|---|---|
pairSeparator |
string |
mandatory - can be '=' or ';' or 'anything' |
keyValuePairSeparair |
string |
mandatory - can be '\u0001', '\u0009', '#' or 'anything' |
crlf |
string |
optional - default value = WINDOWS - allow to define the carriage return character to use |
type |
string |
optional - define the type of message (e.g. FIX, EMX, ...) |
version |
string |
optional - version of the message (e.g. 4.1) |
|
|
This annotation is associated to the message class of the model and must be declared one time. |
case 1 : separator = 'u0001'
The separator used to segregate the key value pair fields in a FIX message is the ASCII '01' character or in unicode format '\u0001'. This character must be escaped a second time to avoid a java runtime error. Here is an example :
8=FIX.4.1 9=20 34=1 35=0 49=INVMGR 56=BRKR 1=BE.CHM.001 11=CHM0001-01 22=4 ...
and how to use the annotation
@Message(keyValuePairSeparator = "=", pairSeparator = "\u0001", type="FIX", version="4.1") public class Order { ... }
Look at test cases
The ASCII character like tab, ... cannot be displayed in WIKI page. So, have a look to the test case of camel-bindy to see exactly how the FIX message looks like (src\test\data\fix\fix.txt) and the Order, Trailer, Header classes (src\test\java\org\apache\camel\dataformat\bindy\model\fix\simple\Order.java)
5. KeyValuePairField
The KeyValuePairField annotation defines the property of a key value pair field. Each KeyValuePairField is identified by a tag (= key) and its value associated, a type (string, int, date, ...), optionaly a pattern and if the field is required
Annotation name |
Record type |
Level |
---|---|---|
KeyValuePairField |
Key Value Pair - FIX |
Property |
Parameter name |
type |
Info |
---|---|---|
tag |
int |
mandatory - digit number identifying the field in the message - must be unique |
pattern |
string |
optional - default value = "" - will be used to format Decimal, Date, ... |
precision |
int |
optional - digit number - represents the precision to be used when the Decimal number will be formatted/parsed |
required |
boolean |
optional |
case 1 : tag
This parameter represents the key of the field in the message
@Message(keyValuePairSeparator = "=", pairSeparator = "\u0001", type="FIX", version="4.1") public class Order { @Link Header header; @Link Trailer trailer; @KeyValuePairField(tag = 1) // Client reference private String Account; @KeyValuePairField(tag = 11) // Order reference private String ClOrdId; @KeyValuePairField(tag = 22) // Fund ID type (Sedol, ISIN, ...) private String IDSource; @KeyValuePairField(tag = 48) // Fund code private String SecurityId; @KeyValuePairField(tag = 54) // Movement type ( 1 = Buy, 2 = sell) private String Side; @KeyValuePairField(tag = 58) // Free text private String Text; ... }
Using the Java DSL
The next step consists in instantiating the DataFormat bindy class associated with this record type and providing Java package name(s) as parameter.
For example the following uses the class CsvBindyFormat (who correspond to the class associated with the CSV record type) which is configured with "com.acme.model"
package name to initialize the model objects configured in this package.
DataFormat bindy = new CsvBindyDataFormat("com.acme.model"); from("file://inbox"). unmarshal(bindy). to("bean:handleOrder");
The Camel route will pick-up files in the inbox directory, unmarshall CSV records in a collection of model objects and send the collection
to the bean referenced by 'handleOrder'.
The collection is a list of Map. Each Map of the list contains the objects of the model. Each object can be retrieve using its class name.
int count = 0; List<Map<String, Object>> models = new ArrayList<Map<String, Object>>(); Map<String, Object> model = new HashMap<String, Object>(); models = (List<Map<String, Object>>) exchange.getIn().getBody(); Iterator<Map<String, Object>> it = models.iterator(); while(it.hasNext()){ model = it.next(); for(String key : model.keySet()) { Object obj = model.get(key); LOG.info("Count : " + count + ", " + obj.toString()); } count++; } LOG.info("Nber of CSV records received by the csv bean : " + count);
To generate CSV records from a collection of model objects, you create the following route :
from("bean:handleOrder") marshal(bindy) to("file://outbox")
You can if you prefer use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file. e.g.
from("file://inbox"). unmarshal("myBindyDataFormat"). to("bean:handleOrder");
Unit test
Here is two examples showing how to marshall or unmarshall a CSV file with Camel
package org.apache.camel.dataformat.bindy.csv; import java.math.BigDecimal; import java.util.ArrayList; import java.util.Calendar; import java.util.GregorianCalendar; import java.util.HashMap; import java.util.List; import java.util.Map; import org.apache.camel.EndpointInject; import org.apache.camel.Produce; import org.apache.camel.ProducerTemplate; import org.apache.camel.builder.RouteBuilder; import org.apache.camel.component.mock.MockEndpoint; import org.apache.camel.dataformat.bindy.model.complex.twoclassesandonelink.Client; import org.apache.camel.dataformat.bindy.model.complex.twoclassesandonelink.Order; import org.apache.camel.spring.javaconfig.SingleRouteCamelConfiguration; import org.junit.Test; import org.springframework.config.java.annotation.Bean; import org.springframework.config.java.annotation.Configuration; import org.springframework.config.java.test.JavaConfigContextLoader; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.AbstractJUnit4SpringContextTests; @ContextConfiguration(locations = "org.apache.camel.dataformat.bindy.csv.BindyComplexCsvMarshallTest$ContextConfig", loader = JavaConfigContextLoader.class) public class BindyComplexCsvMarshallTest extends AbstractJUnit4SpringContextTests { private List<Map<String, Object>> models = new ArrayList<Map<String, Object>>(); private String result = "10,A1,Julia,Roberts,BE123456789,Belgium Ventage 10/12,150,USD,14-01-2009"; @Produce(uri = "direct:start") private ProducerTemplate template; @EndpointInject(uri = "mock:result") private MockEndpoint resultEndpoint; @Test public void testMarshallMessage() throws Exception { resultEndpoint.expectedBodiesReceived(result); template.sendBody(generateModel()); resultEndpoint.assertIsSatisfied(); } private List<Map<String, Object>> generateModel() { Map<String, Object> model = new HashMap<String, Object>(); Order order = new Order(); order.setOrderNr(10); order.setAmount(new BigDecimal("150")); order.setIsinCode("BE123456789"); order.setInstrumentName("Belgium Ventage 10/12"); order.setCurrency("USD"); Calendar calendar = new GregorianCalendar(); calendar.set(2009, 0, 14); order.setOrderDate(calendar.getTime()); Client client = new Client(); client.setClientNr("A1"); client.setFirstName("Julia"); client.setLastName("Roberts"); order.setClient(client); model.put(order.getClass().getName(), order); model.put(client.getClass().getName(), client); models.add(0, model); return models; } @Configuration public static class ContextConfig extends SingleRouteCamelConfiguration { BindyCsvDataFormat camelDataFormat = new BindyCsvDataFormat("org.apache.camel.dataformat.bindy.model.complex.twoclassesandonelink"); @Override @Bean public RouteBuilder route() { return new RouteBuilder() { @Override public void configure() { from("direct:start").marshal(camelDataFormat).to("mock:result"); } }; } } }
package org.apache.camel.dataformat.bindy.csv; import org.apache.camel.EndpointInject; import org.apache.camel.builder.RouteBuilder; import org.apache.camel.component.mock.MockEndpoint; import org.apache.camel.spring.javaconfig.SingleRouteCamelConfiguration; import org.junit.Test; import org.springframework.config.java.annotation.Bean; import org.springframework.config.java.annotation.Configuration; import org.springframework.config.java.test.JavaConfigContextLoader; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.AbstractJUnit4SpringContextTests; @ContextConfiguration(locations = "org.apache.camel.dataformat.bindy.csv.BindyComplexCsvUnmarshallTest$ContextConfig", loader = JavaConfigContextLoader.class) public class BindyComplexCsvUnmarshallTest extends AbstractJUnit4SpringContextTests { @EndpointInject(uri = "mock:result") private MockEndpoint resultEndpoint; @Test public void testUnMarshallMessage() throws Exception { resultEndpoint.expectedMessageCount(1); resultEndpoint.assertIsSatisfied(); } @Configuration public static class ContextConfig extends SingleRouteCamelConfiguration { BindyCsvDataFormat csvBindyDataFormat = new BindyCsvDataFormat("org.apache.camel.dataformat.bindy.model.complex.twoclassesandonelink"); @Override @Bean public RouteBuilder route() { return new RouteBuilder() { @Override public void configure() { from("file://src/test/data?noop=true").unmarshal(csvBindyDataFormat).to("mock:result"); } }; } } }
In this example, BindyCsvDataFormat class has been instantiated in a traditional way but it is also possible to provide information directly to the function (un)marshal like this where BindyType corresponds to the Bindy DataFormat class to instantiate and the parameter contains the list of package names.
public static class ContextConfig extends SingleRouteCamelConfiguration { @Override @Bean public RouteBuilder route() { return new RouteBuilder() { @Override public void configure() { from("direct:start") .marshal().bindy(BindyType.Csv, "org.apache.camel.dataformat.bindy.model.simple.oneclass") .to("mock:result"); } }; } }
Using Spring XML
This is really easy to use Spring as your favorite DSL language to declare the routes to be used for camel-bindy. The following example shows two routes where the first will pick-up records from files, unmarshal the content and bind it to their model. The result is then send to a pojo (doing nothing special) and place them into a queue.
The second route will extract the pojos from the queue and marshal the content to generate a file containing the csv record
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd"> <bean id="bindyDataformat" class="org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat"> <constructor-arg value="org.apache.camel.bindy.model" /> </bean> <bean id="csv" class="org.apache.camel.bindy.csv.HandleOrderBean" /> <!-- Queuing engine - ActiveMq - work locally in mode virtual memory --> <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent"> <property name="brokerURL" value="vm://localhost:61616"/> </bean> <camelContext xmlns="http://camel.apache.org/schema/spring"> <jmxAgent id="agent" disabled="false" /> <route> <from uri="file://src/data/csv/?noop=true" /> <unmarshal ref="bindyDataformat" /> <to uri="bean:csv" /> <to uri="activemq:queue:in" /> </route> <route> <from uri="activemq:queue:in" /> <marshal ref="bindyDataformat" /> <to uri="file://src/data/csv/out/" /> </route> </camelContext> </beans>
Be careful
Please verify that your model classes implements serializable otherwise the queue manager will raise an error
Dependencies
To use Bindy in your camel routes you need to add the a dependency on camel-bindy which implements this data format.
If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-bindy</artifactId> <version>2.1.0</version> </dependency>