Data Format

Camel supports a pluggable DataFormat to allow messages to be marshalled to and from binary or text formats to support a kind of Message Translator.

The following data formats are currently supported:

And related is the following:

• DataFormat Component for working with Data Formats as if it was a regular Component supporting Endpoints and URIs.
• Dozer Type Conversion using Dozer for type converting POJOs

Unmarshalling

If you receive a message from one of the Camel Components such as File, HTTP or JMS you often want to unmarshal the payload into some bean so that you can process it using some Bean Integration or perform Predicate evaluation and so forth. To do this use the unmarshal word in the DSL in Java or the Xml Configuration.

For example

DataFormat jaxb = new JaxbDataFormat("com.acme.model");

from("activemq:My.Queue").
  unmarshal(jaxb).
  to("mqseries:Another.Queue");

The above uses a named DataFormat of jaxb which is configured with a number of Java package names. You can if you prefer use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file.

You can also use the DSL itself to define the data format as you use it. For example the following uses Java serialization to unmarshal a binary file then send it as an ObjectMessage to ActiveMQ

from("file://foo/bar").
  unmarshal().serialization().
  to("activemq:Some.Queue");

Marshalling

Marshalling is the opposite of unmarshalling, where a bean is marshalled into some binary or textual format for transmission over some transport via a Camel Component. Marshalling is used in the same way as unmarshalling above; in the DSL you can use a DataFormat instance, you can configure the DataFormat dynamically using the DSL or you can refer to a named instance of the format in the Registry.

The following example unmarshals via serialization then marshals using a named JAXB data format to perform a kind of Message Translator

from("file://foo/bar").
  unmarshal().serialization(). 
  marshal("jaxb").
  to("activemq:Some.Queue");

Using Spring XML

This example shows how to configure the data type just once and reuse it on multiple routes

You can also define reusable data formats as Spring beans

<bean id="myJaxb" class="org.apache.camel.model.dataformat.JaxbDataFormat">
  <property name="prettyPrint" value="true"/>
  <property name="contextPath" value="org.apache.camel.example"/>
</bean>  

Serialization

Serialization is a Data Format which uses the standard Java Serialization mechanism to unmarshal a binary payload into Java objects or to marshal Java objects into a binary blob.
For example the following uses Java serialization to unmarshal a binary file then send it as an ObjectMessage to ActiveMQ

from("file://foo/bar").
  unmarshal().serialization().
  to("activemq:Some.Queue");

Dependencies

This data format is provided in camel-core so no additional dependencies is needed.

JAXB

JAXB is a Data Format which uses the JAXB2 XML marshalling standard which is included in Java 6 to unmarshal an XML payload into Java objects or to marshal Java objects into an XML payload.

Using the Java DSL

For example the following uses a named DataFormat of jaxb which is configured with a number of Java package names to initialize the JAXBContext.

You can if you prefer use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file. e.g.

Using Spring XML

The following example shows how to use JAXB to unmarshal using Spring configuring the jaxb data type{snippet:id=example|lang=xml|url=camel/trunk/components/camel-jaxb/src/test/resources/org/apache/camel/example/springDataFormat.xml}This example shows how to configure the data type just once and reuse it on multiple routes.{snippet:id=example|lang=xml|url=camel/trunk/components/camel-jaxb/src/test/resources/org/apache/camel/example/marshalAndUnmarshalWithRef.xml}

Partial marshalling/unmarshalling

This feature is new to Camel 2.2.0.
JAXB 2 supports marshalling and unmarshalling XML tree fragments. By default JAXB looks for @XmlRootElement annotation on given class to operate on whole XML tree. This is useful but not always - sometimes generated code does not have @XmlRootElement annotation, sometimes you need unmarshall only part of tree.
In that case you can use partial unmarshalling. To enable this behaviours you need set property partClass. Camel will pass this class to JAXB's unmarshaler.{snippet:id=example|lang=xml|url=camel/trunk/components/camel-jaxb/src/test/resources/org/apache/camel/example/springDataFormatPartial.xml}For marshalling you have to add partNamespace attribute with QName of destination namespace. Example of Spring DSL you can find above.

Fragment

This feature is new to Camel 2.8.0.
JaxbDataFormat has new property fragment which can set the the Marshaller.JAXB_FRAGMENT encoding property on the JAXB Marshaller. If you don't want the JAXB Marshaller to generate the XML declaration, you can set this option to be true. The default value of this property is false.

Ignoring the NonXML Character

This feature is new to Camel 2.2.0.
JaxbDataFromat supports to ignore the NonXML Character, you just need to set the filterNonXmlChars property to be true, JaxbDataFormat will replace the NonXML character with " " when it is marshaling or unmarshaling the message. You can also do it by setting the Exchange property Exchange.FILTER_NON_XML_CHARS.

This feature has been tested with Woodstox 3.2.9 and Sun JDK 1.6 StAX implementation.

New for Camel 2.12.1
JaxbDataFormat now allows you to customize the XMLStreamWriter used to marshal the stream to XML. Using this configuration, you can add your own stream writer to completely remove, escape, or replace non-xml characters.

The following example shows using the Spring DSL and also enabling Camel's NonXML filtering:

Working with the ObjectFactory

If you use XJC to create the java class from the schema, you will get an ObjectFactory for you JAXB context. Since the ObjectFactory uses JAXBElement to hold the reference of the schema and element instance value, jaxbDataformat will ignore the JAXBElement by default and you will get the element instance value instead of the JAXBElement object form the unmarshaled message body.
If you want to get the JAXBElement object form the unmarshaled message body, you need to set the JaxbDataFormat object's ignoreJAXBElement property to be false.

Setting encoding

You can set the encoding option to use when marshalling. Its the Marshaller.JAXB_ENCODING encoding property on the JAXB Marshaller.
You can setup which encoding to use when you declare the JAXB data format. You can also provide the encoding in the Exchange property Exchange.CHARSET_NAME. This property will overrule the encoding set on the JAXB data format.

In this Spring DSL we have defined to use iso-8859-1 as the encoding:{snippet:id=example|lang=xml|url=camel/trunk/components/camel-jaxb/src/test/resources/org/apache/camel/example/springDataFormatWithEncoding.xml}

Controlling namespace prefix mapping

Available as of Camel 2.11

When marshalling using JAXB or SOAP then the JAXB implementation will automatic assign namespace prefixes, such as ns2, ns3, ns4 etc. To control this mapping, Camel allows you to refer to a map which contains the desired mapping.

Notice this requires having JAXB-RI 2.1 or better (from SUN) on the classpath, as the mapping functionality is dependent on the implementation of JAXB, whether its supported.

For example in Spring XML we can define a Map with the mapping. In the mapping file below, we map SOAP to use soap as prefix. While our custom namespace "http://www.mycompany.com/foo/2" is not using any prefix.

To use this in JAXB or SOAP you refer to this map, using the namespacePrefixRef attribute as shown below. Then Camel will lookup in the Registry a java.util.Map with the id "myMap", which was what we defined above.

Schema validation

Available as of Camel 2.11

The JAXB Data Format supports validation by marshalling and unmarshalling from/to XML. Your can use the prefix classpath:, file:* or *http: to specify how the resource should by resolved. You can separate multiple schema files by using the ',' character.

Using the Java DSL, you can configure it in the following way:

You can do the same using the XML DSL:

Camel will create and pool the underling SchemaFactory instances on the fly, because the SchemaFactory shipped with the JDK is not thread safe.
However, if you have a SchemaFactory implementation which is thread safe, you can configure the JAXB data format to use this one:

Schema Location

Available as of Camel 2.14

The JAXB Data Format supports to specify the SchemaLocation when marshaling the XML. 

Using the Java DSL, you can configure it in the following way:

You can do the same using the XML DSL:

Marshal data that is already XML

Available as of Camel 2.14.1

Dependencies

To use JAXB in your camel routes you need to add the a dependency on camel-jaxb which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

XmlBeans

XmlBeans is a Data Format which uses the XmlBeans library to unmarshal an XML payload into Java objects or to marshal Java objects into an XML payload.

from("activemq:My.Queue").
  unmarshal().xmlBeans().
  to("mqseries:Another.Queue");

Dependencies

To use XmlBeans in your camel routes you need to add the dependency on camel-xmlbeans which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-xmlbeans</artifactId>
  <version>x.x.x</version>
  <!-- use the same version as your Camel core version -->
</dependency>

XStream

XStream is a Data Format which uses the XStream library to marshal and unmarshal Java objects to and from XML.

To use XStream in your camel routes you need to add the a dependency on camel-xstream which implements this data format.

Maven users will need to add the following dependency to their pom.xml for this component:

Using the Java DSL

If you would like to configure the XStream instance used by the Camel for the message transformation, you can simply pass a reference to that instance on the DSL level.

XMLInputFactory and XMLOutputFactory

The XStream library uses the javax.xml.stream.XMLInputFactory and javax.xml.stream.XMLOutputFactory, you can control which implementation of this factory should be used.

The Factory is discovered using this algorithm:
1. Use the javax.xml.stream.XMLInputFactory , javax.xml.stream.XMLOutputFactory system property.
2. Use the lib/xml.stream.properties file in the JRE_HOME directory.
3. Use the Services API, if available, to determine the classname by looking in the META-INF/services/javax.xml.stream.XMLInputFactory, META-INF/services/javax.xml.stream.XMLOutputFactory files in jars available to the JRE.
4. Use the platform default XMLInputFactory,XMLOutputFactory instance.

How to set the XML encoding in Xstream DataFormat?

From Camel 2.2.0, you can set the encoding of XML in Xstream DataFormat by setting the Exchange's property with the key Exchange.CHARSET_NAME, or setting the encoding property on Xstream from DSL or Spring config.

{snippet:id=e1|lang=xml|url=camel/trunk/components/camel-xstream/src/test/resources/org/apache/camel/dataformat/xstream/SpringMarshalListTest.xml}

Setting the type permissions of Xstream DataFormat

In Camel, one can always use its own processing step in the route to filter and block certain XML documents to be routed to the XStream's unmarhall step. From Camel 2.16.1, 2.15.5, you can set XStream's type permissions to automatically allow or deny the instantiation of certain types.

The default type permissions setting used by Camel denies all types except for those from java.lang and java.util packages. This setting can be changed by setting System property org.apache.camel.xstream.permissions. Its value is a string of comma-separated permission terms, each representing a type being allowed or denied, depending on whether the term is prefixed with '+' (note '+' may be omitted) or with '-', respectively.

Each term may contain a wildcard character '*'. For example, value "-*,java.lang.*,java.util.*" indicates denying all types except for java.lang.* and java.util.* classes. Setting this value to an empty string "" reverts to the default XStream's type permissions handling which denies certain blacklisted classes and allow others.

The type permissions setting can be extended at an individual XStream DataFormat instance by setting its type permissions property.

CSV

The CSV Data Format uses Apache Commons CSV to handle CSV payloads (Comma Separated Values) such as those exported/imported by Excel.

As of Camel 2.15.0, it now uses the Apache Commons CSV 1.1 which is based on a completely different set of options.

Available options until Camel 2.15

Available options as of Camel 2.15

<csv >
    <header>orderId</header>
    <header>amount</header>
</csv>

Marshalling a Map to CSV

The component allows you to marshal a Java Map (or any other message type that can be converted in a Map) into a CSV payload.

Map<String, Object> body = new LinkedHashMap<>();
body.put("foo", "abc");
body.put("bar", 123);

from("direct:start")
    .marshal().csv()
    .to("mock:result");

<route>
    <from uri="direct:start" />
    <marshal>
        <csv />
    </marshal>
    <to uri="mock:result" />
</route>

abc,123

Unmarshalling a CSV message into a Java List

Unmarshalling will transform a CSV messsage into a Java List with CSV file lines (containing another List with all the field values).

An example: we have a CSV file with names of persons, their IQ and their current activity.

Jack Dalton, 115, mad at Averell
Joe Dalton, 105, calming Joe
William Dalton, 105, keeping Joe from killing Averell
Averell Dalton, 80, playing with Rantanplan
Lucky Luke, 120, capturing the Daltons

We can now use the CSV component to unmarshal this file:

from("file:src/test/resources/?fileName=daltons.csv&noop=true")
    .unmarshal().csv()
    .to("mock:daltons");

The resulting message will contain a List<List<String>> like...

List<List<String>> data = (List<List<String>>) exchange.getIn().getBody();
for (List<String> line : data) {
    LOG.debug(String.format("%s has an IQ of %s and is currently %s", line.get(0), line.get(1), line.get(2)));
}

Marshalling a List<Map> to CSV

Available as of Camel 2.1

If you have multiple rows of data you want to be marshalled into CSV format you can now store the message payload as a List<Map<String, Object>> object where the list contains a Map for each row.

File Poller of CSV, then unmarshaling

Given a bean which can handle the incoming data...

// Some comments here
public void doHandleCsvData(List<List<String>> csvData)
{
    // do magic here
}

... your route then looks as follows

<route>
        <!-- poll every 10 seconds -->
        <from uri="file:///some/path/to/pickup/csvfiles?delete=true&amp;consumer.delay=10000" />
        <unmarshal><csv /></unmarshal>
        <to uri="bean:myCsvHandler?method=doHandleCsvData" />
</route>

Marshaling with a pipe as delimiter

Map<String, Object> body = new LinkedHashMap<>();
body.put("foo", "abc");
body.put("bar", 123);

// Camel version < 2.15
CsvDataFormat oldCSV = new CsvDataFormat();
oldCSV.setDelimiter("|");
from("direct:start")
    .marshal(oldCSV)
    .to("mock:result")
 
// Camel version >= 2.15
from("direct:start")
    .marshal(new CsvDataFormat().setDelimiter('|'))
    .to("mock:result")

<route>
  <from uri="direct:start" />
  <marshal>
    <csv delimiter="|" />
  </marshal>
  <to uri="mock:result" />
</route>

abc|123

Using autogenColumns, configRef and strategyRef attributes inside XML DSL

Available as of Camel 2.9.2 / 2.10 and deleted for Camel 2.15

You can customize the CSV Data Format to make use of your own CSVConfig and/or CSVStrategy. Also note that the default value of the autogenColumns option is true. The following example should illustrate this customization.

<route>
  <from uri="direct:start" />
  <marshal>
    <!-- make use of a strategy other than the default one which is 'org.apache.commons.csv.CSVStrategy.DEFAULT_STRATEGY' -->
    <csv autogenColumns="false" delimiter="|" configRef="csvConfig" strategyRef="excelStrategy" />
  </marshal>
  <convertBodyTo type="java.lang.String" />
  <to uri="mock:result" />
</route>

<bean id="csvConfig" class="org.apache.commons.csv.writer.CSVConfig">
  <property name="fields">
    <list>
      <bean class="org.apache.commons.csv.writer.CSVField">
        <property name="name" value="orderId" />
      </bean>
      <bean class="org.apache.commons.csv.writer.CSVField">
        <property name="name" value="amount" />
      </bean>
    </list>
  </property>
</bean>

<bean id="excelStrategy" class="org.springframework.beans.factory.config.FieldRetrievingFactoryBean">
  <property name="staticField" value="org.apache.commons.csv.CSVStrategy.EXCEL_STRATEGY" />
</bean>

Using skipFirstLine option while unmarshaling

Available as of Camel 2.10 and deleted for Camel 2.15

You can instruct the CSV Data Format to skip the first line which contains the CSV headers. Using the Spring/XML DSL:

<route>
  <from uri="direct:start" />
  <unmarshal>
    <csv skipFirstLine="true" />
  </unmarshal>
  <to uri="bean:myCsvHandler?method=doHandleCsv" />
</route>

Or the Java DSL:

CsvDataFormat csv = new CsvDataFormat();
csv.setSkipFirstLine(true);

from("direct:start")
  .unmarshal(csv)
.to("bean:myCsvHandler?method=doHandleCsv");

Unmarshaling with a pipe as delimiter

Using the Spring/XML DSL:

<route>
  <from uri="direct:start" />
  <unmarshal>
    <csv delimiter="|" />
  </unmarshal>
  <to uri="bean:myCsvHandler?method=doHandleCsv" />
</route>

Or the Java DSL:

CsvDataFormat csv = new CsvDataFormat();
CSVStrategy strategy = CSVStrategy.DEFAULT_STRATEGY;
strategy.setDelimiter('|');
csv.setStrategy(strategy);

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");

CsvDataFormat csv = new CsvDataFormat();
csv.setDelimiter("|");

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");

CsvDataFormat csv = new CsvDataFormat();
CSVConfig csvConfig = new CSVConfig();
csvConfig.setDelimiter(";");
csv.setConfig(csvConfig);

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");

CSVConfig csvConfig = new CSVConfig();
csvConfig.setDelimiter(';');

Dependencies

To use CSV in your Camel routes you need to add a dependency on camel-csv, which implements this data format.

If you use Maven you can just add the following to your pom.xml, substituting the version number for the latest and greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-csv</artifactId>
  <version>x.x.x</version>
</dependency>

Options

Marshal

In this example we marshal the file content to String object in UTF-8 encoding.

from("file://data.csv").marshal().string("UTF-8").to("jms://myqueue");

Unmarshal

In this example we unmarshal the payload from the JMS queue to a String object using UTF-8 encoding, before its processed by the newOrder processor.

from("jms://queue/order").unmarshal().string("UTF-8").processRef("newOrder");

Dependencies

This data format is provided in camel-core so no additional dependencies is needed.

HL7 DataFormat

The HL7 component ships with a HL7 data format that can be used to marshal or unmarshal HL7 model objects.

• marshal = from Message to byte stream (can be used when responding using the HL7 MLLP codec)
• unmarshal = from byte stream to Message (can be used when receiving streamed data from the HL7 MLLP

To use the data format, simply instantiate an instance and invoke the marshal or unmarshal operation in the route builder:

  DataFormat hl7 = new HL7DataFormat();
  ...
  from("direct:hl7in").marshal(hl7).to("jms:queue:hl7out");

In the sample above, the HL7 is marshalled from a HAPI Message object to a byte stream and put on a JMS queue.
The next example is the opposite:

  DataFormat hl7 = new HL7DataFormat();
  ...
  from("jms:queue:hl7out").unmarshal(hl7).to("patientLookupService");

Here we unmarshal the byte stream into a HAPI Message object that is passed to our patient lookup service.

There is a shorthand syntax in Camel for well-known data formats that are commonly used.
Then you don't need to create an instance of the HL7DataFormat object:

  from("direct:hl7in").marshal().hl7().to("jms:queue:hl7out");
  from("jms:queue:hl7out").unmarshal().hl7().to("patientLookupService");

EDI DataFormat

We encourage end users to look at the Smooks which supports EDI and Camel natively.

Flatpack DataFormat

The Flatpack component ships with the Flatpack data format that can be used to format between fixed width or delimited text messages to a List of rows as Map.

• marshal = from List<Map<String, Object>> to OutputStream (can be converted to String)
• unmarshal = from java.io.InputStream (such as a File or String) to a java.util.List as an org.apache.camel.component.flatpack.DataSetList instance.
  The result of the operation will contain all the data. If you need to process each row one by one you can split the exchange, using Splitter.

Notice: The Flatpack library does currently not support header and trailers for the marshal operation.

Options

The data format has the following options:

Usage

To use the data format, simply instantiate an instance and invoke the marshal or unmarshal operation in the route builder:

  FlatpackDataFormat fp = new FlatpackDataFormat();
  fp.setDefinition(new ClassPathResource("INVENTORY-Delimited.pzmap.xml"));
  ...
  from("file:order/in").unmarshal(df).to("seda:queue:neworder");

The sample above will read files from the order/in folder and unmarshal the input using the Flatpack configuration file INVENTORY-Delimited.pzmap.xml that configures the structure of the files. The result is a DataSetList object we store on the SEDA queue.

FlatpackDataFormat df = new FlatpackDataFormat();
df.setDefinition(new ClassPathResource("PEOPLE-FixedLength.pzmap.xml"));
df.setFixed(true);
df.setIgnoreFirstRecord(false);

from("seda:people").marshal(df).convertBodyTo(String.class).to("jms:queue:people");

In the code above we marshal the data from a Object representation as a List of rows as Maps. The rows as Map contains the column name as the key, and the the corresponding value. This structure can be created in Java code from e.g. a processor. We marshal the data according to the Flatpack format and convert the result as a String object and store it on a JMS queue.

Dependencies

To use Flatpack in your camel routes you need to add the a dependency on camel-flatpack which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-flatpack</artifactId>
  <version>x.x.x</version>
</dependency>

JSON

JSON is a Data Format to marshal and unmarshal Java objects to and from JSON.

For JSON to object marshalling, Camel provides integration with three popular JSON libraries:

• The XStream library and Jettsion
• The Jackson library
• Camel 2.10: The GSon library

Every library requires adding the special camel component (see "Dependency..." paragraphs further down). By default Camel uses the XStream library.

Using JSON Data Format With the XStream Library

Using JSON Data Format With the Jackson Library

Using JSON Data Format With the GSON Library

Using JSON in Spring DSL

When using Data Format in Spring DSL you need to declare the data formats first. This is done in the DataFormats XML tag.

And then you can refer to this id in the route:

Excluding POJO Fields From Marshalling

As of Camel 2.10
When marshaling a POJO to JSON you might want to exclude certain fields from the JSON output. With Jackson you can use JSON views to accomplish this.

First create one or more marker classes:{snippet:id=marker|lang=java|url=camel/trunk/components/camel-jackson/src/test/java/org/apache/camel/component/jackson/Views.java}Second, use the marker classes with the @JsonView annotation to include/exclude certain fields. The annotation also works on getters:{snippet:id=jsonview|lang=java|url=camel/trunk/components/camel-jackson/src/test/java/org/apache/camel/component/jackson/TestPojoView.java}Finally, use the Camel JacksonDataFormat to marshal the above POJO to JSON.{snippet:id=format|lang=java|url=camel/trunk/components/camel-jackson/src/test/java/org/apache/camel/component/jackson/JacksonMarshalViewTest.java}Note: the height field is missing in the resulting JSON.

The GSON library supports a similar feature through the notion of ExclusionStrategies:{snippet:id=strategy|lang=java|url=camel/trunk/components/camel-gson/src/test/java/org/apache/camel/component/gson/GsonMarshalExclusionTest.java}The GsonDataFormat accepts an ExclusionStrategy in its constructor:{snippet:id=format|lang=java|url=camel/trunk/components/camel-gson/src/test/java/org/apache/camel/component/gson/GsonMarshalExclusionTest.java}The line above will exclude fields annotated with @ExcludeAge when marshaling to JSON.

Configuring Field Naming Policy

Available as of Camel 2.11

The GSON library supports specifying policies and strategies for mapping from JSON to POJO fields. A common naming convention is to map JSON fields using lower case with underscores.

We may have this JSON string

Which we want to map to a POJO that has getter/setters as:

Then we can configure the org.apache.camel.component.gson.GsonDataFormat in a Spring XML files as shown below. Notice we use fieldNamingPolicy property to set the field mapping. This property is an enum from GSon com.google.gson.FieldNamingPolicy which has a number of predefined mappings.

If you need full control you can use the property FieldNamingStrategy and implement a custom com.google.gson.FieldNamingStrategy where you can control the mapping.

And use it in Camel routes by referring to its bean id as shown:

Include/Exclude Fields Using the jsonView Attribute With JacksonDataFormat

Available as of Camel 2.12

As an example of using this attribute you can instead of:

Directly specify your JSON view inside the Java DSL as:

And the same in XML DSL:

Setting Serialization Include Option for Jackson Marshal

Available as of Camel 2.13.3/2.14

If you want to marshal a POJO to JSON, and the POJO has some fields with null values. And you want to skip these null values, then you need to set either an annotation on the POJO, 

But this requires you to include that annotation in your POJO source code. You can also configure the Camel JsonDataFormat to set the include option, as shown below:

Or from XML DSL you configure this as

Unmarshaling from JSON to POJO with Dynamic Class Name

Available as of Camel 2.14

If you use Jackson to unmarshal JSON to POJO, then you can now specify a header in the message that indicate which class name to unmarshal to. The header has key CamelJacksonUnmarshalType if that header is present in the message, then Jackson will use that as FQN for the POJO class to unmarshal the JSON payload as. Notice that behavior is enabled out of the box from Camel 2.14. 

 For JMS end users there is the JMSType header from the JMS spec that indicates that also. To enable support for JMSType you would need to turn that on, on the Jackson data format as shown:

Or from XML DSL you configure this as

Unmarshaling From JSON to List<Map> or List<pojo>

Available as of Camel 2.14

If you are using Jackson to unmarshal JSON to a list of map/POJO, you can now specify this by setting useList="true" or use the org.apache.camel.component.jackson.ListJacksonDataFormat.

For example, with Java you can do as shown below:

And if you use XML DSL then you configure to use list using useList attribute as shown below:

And you can specify the pojo type also

Using Custom Jackson's ObjectMapper

Available from Camel 2.17

You can use custom Jackson ObjectMapper instance, can be configured as shown below.

Where myMapper is the id of the custom instance that Camel will lookup in the Registry.

Using Custom Jackson Modules

Available as of Camel 2.15

You can use custom Jackson modules by specifying the class names of those using the moduleClassNames option as shown below.

When using moduleClassNames then the custom Jackson modules are not configured, by created using default constructor and used as-is. If a custom module needs any custom configuration, then an instance of the module can be created and configured, and then use modulesRefs to refer to the module as shown below:

 Multiple modules can be specified separated by comma, such as moduleRefs="myJacksonModule,myOtherModule".

Enabling or Disable Features Using Jackson

Available as of Camel 2.15

Jackson has a number of features you can enable or disable, which its ObjectMapper uses. For example to disable failing on unknown properties when marshaling, you can configure this using the disableFeatures:

You can disable multiple features by separating the values using comma. The values for the features must be the name of the enums from Jackson from the following enum classes

• com.fasterxml.jackson.databind.SerializationFeature
• com.fasterxml.jackson.databind.DeserializationFeature
• com.fasterxml.jackson.databind.MapperFeature

To enable a feature use the enableFeatures options instead.

From Java code you can use the type safe methods from camel-jackson module:

Converting Maps to POJO Using Jackson

Available since Camel 2.16. Jackson ObjectMapper can be used to convert maps to POJO objects. Jackson component comes with the data converter that can be used to convert java.util.Map instance to non-String, non-primitive and non-Number objects.

If there is a single ObjectMapper instance available in the Camel registry, it will used by the converter to perform the conversion. Otherwise the default mapper will be used.  

Formatted JSON marshalling (pretty-printing)

Available as of Camel 2.16

Using the prettyPrint option one can output a well formatted JSON while marshaling:

And in Java DSL:

Please note that as of Camel 2.16 there are five different overloaded json() DSL methods which support the prettyPrint option in combination with other settings for JsonLibrary, unmarshalType, jsonView etc. 

 Integrating Jackson with Camel's TypeConverters

 Available as of Camel 2.17

The camel-jackson module allows to integrate Jackson as a Type Converter in the Camel registry. This works in similar ways that camel-jaxb integrates with the type converter as well. However camel-jackson must be explicit enabled, which is done by setting some options on the CamelContext properties, as shown below:

The camel-jackson type converter integrates with JAXB which means you can annotate POJO class with JAXB annotations that Jackson can leverage. 

Dependencies for XStream

To use JSON in your camel routes you need to add the a dependency on camel-xstream which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

Dependencies for Jackson

To use JSON in your camel routes you need to add the a dependency on camel-jackson which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

Dependencies for GSON

To use JSON in your camel routes you need to add the a dependency on camel-gson which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

Options

Marshal

In this example we marshal a regular text/XML payload to a compressed payload employing zip compression Deflater.BEST_COMPRESSION and send it an ActiveMQ queue called MY_QUEUE.

from("direct:start").marshal().zip(Deflater.BEST_COMPRESSION).to("activemq:queue:MY_QUEUE");

Alternatively if you would like to use the default setting you could send it as

from("direct:start").marshal().zip().to("activemq:queue:MY_QUEUE");

Unmarshal

In this example we unmarshal a zipped payload from an ActiveMQ queue called MY_QUEUE to its original format, and forward it for processing to the UnZippedMessageProcessor. Note that the compression Level employed during the marshalling should be identical to the one employed during unmarshalling to avoid errors.

from("activemq:queue:MY_QUEUE").unmarshal().zip().process(new UnZippedMessageProcessor()); 

Dependencies

This data format is provided in camel-core so no additional dependencies are needed.

Bindy

The goal of this component is to allow the parsing/binding of non-structured data (or to be more precise non-XML data)
to/from Java Beans that have binding mappings defined with annotations. Using Bindy, you can bind data from sources such as :

• CSV records,
• Fixed-length records,
• FIX messages,
• or almost any other non-structured data

to one or many Plain Old Java Object (POJO). Bindy converts the data according to the type of the java property. POJOs can be linked together with one-to-many relationships available in some cases. Moreover, for data type like Date, Double, Float, Integer, Short, Long and BigDecimal, you can provide the pattern to apply during the formatting of the property.

For the BigDecimal numbers, you can also define the precision and the decimal or grouping separators.

Decimal* = Double, Integer, Float, Short, Long

To work with camel-bindy, you must first define your model in a package (e.g. com.acme.model) and for each model class (e.g. Order, Client, Instrument, ...) add the required annotations (described hereafter) to the Class or field.

Annotations

The annotations created allow to map different concept of your model to the POJO like :

• Type of record (csv, key value pair (e.g. FIX message), fixed length ...),
• Link (to link object in another object),
• DataField and their properties (int, type, ...),
• KeyValuePairField (for key = value format like we have in FIX financial messages),
• Section (to identify header, body and footer section),
• OneToMany

This section will describe them :

1. CsvRecord

The CsvRecord annotation is used to identified the root class of the model. It represents a record = a line of a CSV file and can be linked to several children model classes.

case 1 : separator = ','

The separator used to segregate the fields in the CSV record is ',' :

10, J, Pauline, M, XD12345678, Fortis Dynamic 15/15, 2500, USD,08-01-2009

@CsvRecord( separator = "," )
public Class Order {
...
}

case 2 : separator = ';'

Compare to the previous case, the separator here is ';' instead of ',' :

10; J; Pauline; M; XD12345678; Fortis Dynamic 15/15; 2500; USD; 08-01-2009

@CsvRecord( separator = ";" )
public Class Order {
...
}

case 3 : separator = '|'

Compare to the previous case, the separator here is '|' instead of ';' :

10| J| Pauline| M| XD12345678| Fortis Dynamic 15/15| 2500| USD| 08-01-2009

@CsvRecord( separator = "\\|" )
public Class Order {
...
}

case 4 : separator = '\",\"'
Applies for Camel 2.8.2 or older

When the field to be parsed of the CSV record contains ',' or ';' which is also used as separator, we whould find another strategy
to tell camel bindy how to handle this case. To define the field containing the data with a comma, you will use simple or double quotes
as delimiter (e.g : '10', 'Street 10, NY', 'USA' or "10", "Street 10, NY", "USA").
Remark : In this case, the first and last character of the line which are a simple or double quotes will removed by bindy

"10","J","Pauline"," M","XD12345678","Fortis Dynamic 15,15" 2500","USD","08-01-2009"

@CsvRecord( separator = "\",\"" )
public Class Order {
...
}

From Camel 2.8.3/2.9 or never bindy will automatic detect if the record is enclosed with either single or double quotes and automatic remove those quotes when unmarshalling from CSV to Object. Therefore do not include the quotes in the separator, but simple do as below:

"10","J","Pauline"," M","XD12345678","Fortis Dynamic 15,15" 2500","USD","08-01-2009"

@CsvRecord( separator = "," )
public Class Order {
...
}

Notice that if you want to marshal from Object to CSV and use quotes, then you need to specify which quote character to use, using the quote attribute on the @CsvRecord as shown below:

@CsvRecord( separator = ",", quote = "\"" )
public Class Order {
...
}

case 5 : separator & skipfirstline

The feature is interesting when the client wants to have in the first line of the file, the name of the data fields :

order id, client id, first name, last name, isin code, instrument name, quantity, currency, date

To inform bindy that this first line must be skipped during the parsing process, then we use the attribute :

@CsvRecord(separator = ",", skipFirstLine = true)
public Class Order {
...
}

case 6 : generateHeaderColumns

To add at the first line of the CSV generated, the attribute generateHeaderColumns must be set to true in the annotation like this :

@CsvRecord( generateHeaderColumns = true )
public Class Order {
...
}

As a result, Bindy during the unmarshaling process will generate CSV like this :

order id, client id, first name, last name, isin code, instrument name, quantity, currency, date
10, J, Pauline, M, XD12345678, Fortis Dynamic 15/15, 2500, USD,08-01-2009

case 7 : carriage return

If the platform where camel-bindy will run is not Windows but Macintosh or Unix, than you can change the crlf property like this. Three values are available : WINDOWS, UNIX or MAC

@CsvRecord(separator = ",", crlf="MAC")
public Class Order {
...
}

Additionally, if for some reason you need to add a different line ending character, you can opt to specify it using the crlf parameter. In the following example, we can end the line with a comma followed by the newline character:

@CsvRecord(separator = ",", crlf=",\n")
public Class Order {
...
}

case 8 : isOrdered

Sometimes, the order to follow during the creation of the CSV record from the model is different from the order used during the parsing. Then, in this case, we can use the attribute isOrdered = true to indicate this in combination with attribute 'position' of the DataField annotation.

@CsvRecord(isOrdered = true)
public Class Order {

   @DataField(pos = 1, position = 11)
   private int orderNr;

   @DataField(pos = 2, position = 10)
   private String clientNr;

...
}

Remark : pos is used to parse the file, stream while positions is used to generate the CSV

2. Link

The link annotation will allow to link objects together.

e.g : If the model Class Client is linked to the Order class, then use annotation Link in the Order class like this :

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @Link
    private Client client;
...

AND for the class Client :

@Link
public class Client {
...
}

3. DataField

The DataField annotation defines the property of the field. Each datafield is identified by its position in the record, a type (string, int, date, ...) and optionally of a pattern

case 1 : pos

This parameter/attribute represents the position of the field in the csv record

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @DataField(pos = 5)
    private String isinCode;

...
}

As you can see in this example the position starts at '1' but continues at '5' in the class Order. The numbers from '2' to '4' are defined in the class Client (see here after).

public class Client {

    @DataField(pos = 2)
    private String clientNr;

    @DataField(pos = 3)
    private String firstName;

    @DataField(pos = 4)
    private String lastName;
...
}

case 2 : pattern

The pattern allows to enrich or validates the format of your data

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @DataField(pos = 5)
    private String isinCode;

    @DataField(name = "Name", pos = 6)
    private String instrumentName;

    @DataField(pos = 7, precision = 2)
    private BigDecimal amount;

    @DataField(pos = 8)
    private String currency;

    @DataField(pos = 9, pattern = "dd-MM-yyyy") -- pattern used during parsing or when the date is created
    private Date orderDate;
...
}

case 3 : precision

The precision is helpful when you want to define the decimal part of your number

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @Link
    private Client client;

    @DataField(pos = 5)
    private String isinCode;

    @DataField(name = "Name", pos = 6)
    private String instrumentName;

    @DataField(pos = 7, precision = 2) -- precision
    private BigDecimal amount;

    @DataField(pos = 8)
    private String currency;

    @DataField(pos = 9, pattern = "dd-MM-yyyy")
    private Date orderDate;
...
}

case 4 : Position is different in output

The position attribute will inform bindy how to place the field in the CSV record generated. By default, the position used corresponds to the position defined with the attribute 'pos'. If the position is different (that means that we have an asymetric processus comparing marshaling from unmarshaling) than we can use 'position' to indicate this.

Here is an example

@CsvRecord(separator = ",")
public class Order {
@CsvRecord(separator = ",", isOrdered = true)
public class Order {

    // Positions of the fields start from 1 and not from 0

    @DataField(pos = 1, position = 11)
    private int orderNr;

    @DataField(pos = 2, position = 10)
    private String clientNr;

    @DataField(pos = 3, position = 9)
    private String firstName;

    @DataField(pos = 4, position = 8)
    private String lastName;

    @DataField(pos = 5, position = 7)
    private String instrumentCode;

    @DataField(pos = 6, position = 6)
    private String instrumentNumber;
...
}

case 5 : required

If a field is mandatory, simply use the attribute 'required' setted to true

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @DataField(pos = 2, required = true)
    private String clientNr;

    @DataField(pos = 3, required = true)
    private String firstName;

    @DataField(pos = 4, required = true)
    private String lastName;
...
}

If this field is not present in the record, than an error will be raised by the parser with the following information :

Some fields are missing (optional or mandatory), line :

case 6 : trim

If a field has leading and/or trailing spaces which should be removed before they are processed, simply use the attribute 'trim' setted to true

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1, trim = true)
    private int orderNr;

    @DataField(pos = 2, trim = true)
    private Integer clientNr;

    @DataField(pos = 3, required = true)
    private String firstName;

    @DataField(pos = 4)
    private String lastName;
...
}

case 7 : defaultValue

If a field is not defined then uses the value indicated by the defaultValue attribute

@CsvRecord(separator = ",")
public class Order {

    @DataField(pos = 1)
    private int orderNr;

    @DataField(pos = 2)
    private Integer clientNr;

    @DataField(pos = 3, required = true)
    private String firstName;

    @DataField(pos = 4, defaultValue = "Barin")
    private String lastName;
...
}

4. FixedLengthRecord

The FixedLengthRecord annotation is used to identified the root class of the model. It represents a record = a line of a file/message containing data fixed length formatted and can be linked to several children model classes. This format is a bit particular beause data of a field can be aligned to the right or to the left.
When the size of the data does not fill completely the length of the field, we can then add 'padd' characters.

case 1 : Simple fixed length record

This simple example shows how to design the model to parse/format a fixed message

10A9PaulineMISINXD12345678BUYShare2500.45USD01-08-2009

   @FixedLengthRecord(length=54, paddingChar=' ')
    public static class Order {

        @DataField(pos = 1, length=2)
        private int orderNr;

        @DataField(pos = 3, length=2)
        private String clientNr;

        @DataField(pos = 5, length=7)
        private String firstName;

        @DataField(pos = 12, length=1, align="L")
        private String lastName;

        @DataField(pos = 13, length=4)
        private String instrumentCode;

        @DataField(pos = 17, length=10)
        private String instrumentNumber;

        @DataField(pos = 27, length=3)
        private String orderType;

        @DataField(pos = 30, length=5)
        private String instrumentType;

        @DataField(pos = 35, precision = 2, length=7)
        private BigDecimal amount;

        @DataField(pos = 42, length=3)
        private String currency;

        @DataField(pos = 45, length=10, pattern = "dd-MM-yyyy")
        private Date orderDate;
        ...

case 2 : Fixed length record with alignment and padding

This more elaborated example show how to define the alignment for a field and how to assign a padding character which is ' ' here''

10A9 PaulineM ISINXD12345678BUYShare2500.45USD01-08-2009

   @FixedLengthRecord(length=60, paddingChar=' ')
    public static class Order {

        @DataField(pos = 1, length=2)
        private int orderNr;

        @DataField(pos = 3, length=2)
        private String clientNr;

        @DataField(pos = 5, length=9)
        private String firstName;

        @DataField(pos = 14, length=5, align="L")   // align text to the LEFT zone of the block
        private String lastName;

        @DataField(pos = 19, length=4)
        private String instrumentCode;

        @DataField(pos = 23, length=10)
        private String instrumentNumber;

        @DataField(pos = 33, length=3)
        private String orderType;

        @DataField(pos = 36, length=5)
        private String instrumentType;

        @DataField(pos = 41, precision = 2, length=7)
        private BigDecimal amount;

        @DataField(pos = 48, length=3)
        private String currency;

        @DataField(pos = 51, length=10, pattern = "dd-MM-yyyy")
        private Date orderDate;
        ...

case 3 : Field padding

Sometimes, the default padding defined for record cannnot be applied to the field as we have a number format where we would like to padd with '0' instead of ' '. In this case, you can use in the model the attribute paddingField to set this value.

10A9 PaulineM ISINXD12345678BUYShare000002500.45USD01-08-2009

    @FixedLengthRecord(length = 65, paddingChar = ' ')
    public static class Order {

        @DataField(pos = 1, length = 2)
        private int orderNr;

        @DataField(pos = 3, length = 2)
        private String clientNr;

        @DataField(pos = 5, length = 9)
        private String firstName;

        @DataField(pos = 14, length = 5, align = "L")
        private String lastName;

        @DataField(pos = 19, length = 4)
        private String instrumentCode;

        @DataField(pos = 23, length = 10)
        private String instrumentNumber;

        @DataField(pos = 33, length = 3)
        private String orderType;

        @DataField(pos = 36, length = 5)
        private String instrumentType;

        @DataField(pos = 41, precision = 2, length = 12, paddingChar = '0')
        private BigDecimal amount;

        @DataField(pos = 53, length = 3)
        private String currency;

        @DataField(pos = 56, length = 10, pattern = "dd-MM-yyyy")
        private Date orderDate;
        ...

case 4: Fixed length record with delimiter

Fixed-length records sometimes have delimited content within the record. The firstName and lastName fields are delimited with the '^' character in the following example:

10A9Pauline^M^ISINXD12345678BUYShare000002500.45USD01-08-2009

    @FixedLengthRecord()
    public static class Order {

        @DataField(pos = 1, length = 2)
        private int orderNr;

        @DataField(pos = 2, length = 2)
        private String clientNr;

        @DataField(pos = 3, delimiter = "^")
        private String firstName;

        @DataField(pos = 4, delimiter = "^")
        private String lastName;

        @DataField(pos = 5, length = 4)
        private String instrumentCode;

        @DataField(pos = 6, length = 10)
        private String instrumentNumber;

        @DataField(pos = 7, length = 3)
        private String orderType;

        @DataField(pos = 8, length = 5)
        private String instrumentType;

        @DataField(pos = 9, precision = 2, length = 12, paddingChar = '0')
        private BigDecimal amount;

        @DataField(pos = 10, length = 3)
        private String currency;

        @DataField(pos = 11, length = 10, pattern = "dd-MM-yyyy")
        private Date orderDate;

case 5 : Fixed length record with record-defined field length

Occasionally a fixed-length record may contain a field that define the expected length of another field within the same record. In the following example the length of the instrumentNumber field value is defined by the value of instrumentNumberLen field in the record.

10A9Pauline^M^ISIN10XD12345678BUYShare000002500.45USD01-08-2009

    @FixedLengthRecord()
    public static class Order {

        @DataField(pos = 1, length = 2)
        private int orderNr;

        @DataField(pos = 2, length = 2)
        private String clientNr;

        @DataField(pos = 3, delimiter = "^")
        private String firstName;

        @DataField(pos = 4, delimiter = "^")
        private String lastName;

        @DataField(pos = 5, length = 4)
        private String instrumentCode;

        @DataField(pos = 6, length = 2, align = "R", paddingChar = '0')
        private int instrumentNumberLen;
        
        @DataField(pos = 7, lengthPos=6)
        private String instrumentNumber;

        @DataField(pos = 8, length = 3)
        private String orderType;

        @DataField(pos = 9, length = 5)
        private String instrumentType;

        @DataField(pos = 10, precision = 2, length = 12, paddingChar = '0')
        private BigDecimal amount;

        @DataField(pos = 11, length = 3)
        private String currency;

        @DataField(pos = 12, length = 10, pattern = "dd-MM-yyyy")
        private Date orderDate;

case 6 : Fixed length record with header and footer

Bindy will discover fixed-length header and footer records that are configured as part of the model – provided that the annotated classes exist either in the same package as the primary @FixedLengthRecord class, or within one of the configured scan packages. The following text illustrates two fixed-length records that are bracketed by a header record and footer record.

101-08-2009
10A9 PaulineM ISINXD12345678BUYShare000002500.45USD01-08-2009
10A9 RichN ISINXD12345678BUYShare000002700.45USD01-08-2009
9000000002

@FixedLengthRecord(hasHeader = true, hasFooter = true)
public class Order {

    @DataField(pos = 1, length = 2)
    private int orderNr;

    @DataField(pos = 2, length = 2)
    private String clientNr;

    @DataField(pos = 3, length = 9)
    private String firstName;

    @DataField(pos = 4, length = 5, align = "L")
    private String lastName;

    @DataField(pos = 5, length = 4)
    private String instrumentCode;

    @DataField(pos = 6, length = 10)
    private String instrumentNumber;

    @DataField(pos = 7, length = 3)
    private String orderType;

    @DataField(pos = 8, length = 5)
    private String instrumentType;

    @DataField(pos = 9, precision = 2, length = 12, paddingChar = '0')
    private BigDecimal amount;

    @DataField(pos = 10, length = 3)
    private String currency;

    @DataField(pos = 11, length = 10, pattern = "dd-MM-yyyy")
    private Date orderDate;
...
}


@FixedLengthRecord(isHeader = true)
public  class OrderHeader {
    @DataField(pos = 1, length = 1)
    private int recordType = 1;
    
    @DataField(pos = 2, length = 10, pattern = "dd-MM-yyyy")
    private Date recordDate;
    
...
}


@FixedLengthRecord(isFooter = true)
public class OrderFooter {
    
    @DataField(pos = 1, length = 1)
    private int recordType = 9;
    
    @DataField(pos = 2, length = 9, align = "R", paddingChar = '0')
    private int numberOfRecordsInTheFile;

...
}

case 7 : Skipping content when parsing a fixed length record. (Camel 2.11.1)

It is common to integrate with systems that provide fixed-length records containing more information than needed for the target use case. It is useful in this situation to skip the declaration and parsing of those fields that we do not need. To accomodate this, Bindy will skip forward to the next mapped field within a record if the 'pos' value of the next declared field is beyond the cursor position of the last parsed field. Using absolute 'pos' locations for the fields of interest (instead of ordinal values) causes Bindy to skip content between two fields.

Similarly, it is possible that none of the content beyond some field is of interest. In this case, you can tell Bindy to skip parsing of everything beyond the last mapped field by setting the ignoreTrailingChars property on the @FixedLengthRecord declaration.

@FixedLengthRecord(ignoreTrailingChars = true)
public static class Order {

        @DataField(pos = 1, length = 2)
        private int orderNr;

        @DataField(pos = 3, length = 2)
        private String clientNr;

    ... any characters that appear beyond the last mapped field will be ignored

}

5. Message

The Message annotation is used to identified the class of your model who will contain key value pairs fields. This kind of format is used mainly in Financial Exchange Protocol Messages (FIX). Nevertheless, this annotation can be used for any other format where data are identified by keys. The key pair values are separated each other by a separator which can be a special character like a tab delimitor (unicode representation : \u0009) or a start of heading (unicode representation : \u0001)

case 1 : separator = 'u0001'

The separator used to segregate the key value pair fields in a FIX message is the ASCII '01' character or in unicode format '\u0001'. This character must be escaped a second time to avoid a java runtime error. Here is an example :

8=FIX.4.1 9=20 34=1 35=0 49=INVMGR 56=BRKR 1=BE.CHM.001 11=CHM0001-01 22=4 ...

and how to use the annotation

@Message(keyValuePairSeparator = "=", pairSeparator = "\u0001", type="FIX", version="4.1")
public class Order {
...
}

6. KeyValuePairField

The KeyValuePairField annotation defines the property of a key value pair field. Each KeyValuePairField is identified by a tag (= key) and its value associated, a type (string, int, date, ...), optionaly a pattern and if the field is required

case 1 : tag

This parameter represents the key of the field in the message

@Message(keyValuePairSeparator = "=", pairSeparator = "\u0001", type="FIX", version="4.1")
public class Order {

    @Link Header header;

    @Link Trailer trailer;

    @KeyValuePairField(tag = 1) // Client reference
    private String Account;

    @KeyValuePairField(tag = 11) // Order reference
    private String ClOrdId;

    @KeyValuePairField(tag = 22) // Fund ID type (Sedol, ISIN, ...)
    private String IDSource;

    @KeyValuePairField(tag = 48) // Fund code
    private String SecurityId;

    @KeyValuePairField(tag = 54) // Movement type ( 1 = Buy, 2 = sell)
    private String Side;

    @KeyValuePairField(tag = 58) // Free text
    private String Text;

...
}

case 2 : Different position in output

If the tags/keys that we will put in the FIX message must be sorted according to a predefine order, then use the attribute 'position' of the annotation @KeyValuePairField

@Message(keyValuePairSeparator = "=", pairSeparator = "\\u0001", type = "FIX", version = "4.1", isOrdered = true)
public class Order {

    @Link Header header;

    @Link Trailer trailer;

    @KeyValuePairField(tag = 1, position = 1) // Client reference
    private String account;

    @KeyValuePairField(tag = 11, position = 3) // Order reference
    private String clOrdId;

...
}

7. Section

In FIX message of fixed length records, it is common to have different sections in the representation of the information : header, body and section. The purpose of the annotation @Section is to inform bindy about which class of the model represents the header (= section 1), body (= section 2) and footer (= section 3)

Only one attribute/parameter exists for this annotation.

case 1 : Section

A. Definition of the header section

@Section(number = 1)
public class Header {

    @KeyValuePairField(tag = 8, position = 1) // Message Header
    private String beginString;

    @KeyValuePairField(tag = 9, position = 2) // Checksum
    private int bodyLength;
...
}

B. Definition of the body section

@Section(number = 2)
@Message(keyValuePairSeparator = "=", pairSeparator = "\\u0001", type = "FIX", version = "4.1", isOrdered = true)
public class Order {

    @Link Header header;

    @Link Trailer trailer;

    @KeyValuePairField(tag = 1, position = 1) // Client reference
    private String account;

    @KeyValuePairField(tag = 11, position = 3) // Order reference
    private String clOrdId;

C. Definition of the footer section

@Section(number = 3)
public class Trailer {

    @KeyValuePairField(tag = 10, position = 1)
    // CheckSum
    private int checkSum;

    public int getCheckSum() {
        return checkSum;
    }

8. OneToMany

The purpose of the annotation @OneToMany is to allow to work with a List<?> field defined a POJO class or from a record containing repetitive groups.

The relation OneToMany ONLY WORKS in the following cases :

• Reading a FIX message containing repetitive groups (= group of tags/keys)
• Generating a CSV with repetitive data

case 1 : Generating CSV with repetitive data

Here is the CSV output that we want :

Claus,Ibsen,Camel in Action 1,2010,35
Claus,Ibsen,Camel in Action 2,2012,35
Claus,Ibsen,Camel in Action 3,2013,35
Claus,Ibsen,Camel in Action 4,2014,35

Remark : the repetitive data concern the title of the book and its publication date while first, last name and age are common

and the classes used to modeling this. The Author class contains a List of Book.

@CsvRecord(separator=",")
public class Author {

	@DataField(pos = 1)
	private String firstName;

	@DataField(pos = 2)
	private String lastName;

	@OneToMany
	private List<Book> books;

	@DataField(pos = 5)
	private String Age;
...


public class Book {

	@DataField(pos = 3)
	private String title;

	@DataField(pos = 4)
	private String year;

Very simple isn't it !!!

case 2 : Reading FIX message containing group of tags/keys

Here is the message that we would like to process in our model :

"8=FIX 4.19=2034=135=049=INVMGR56=BRKR"
"1=BE.CHM.00111=CHM0001-0158=this is a camel - bindy test"
"22=448=BE000124567854=1"
"22=548=BE000987654354=2"
"22=648=BE000999999954=3"
"10=220"

tags 22, 48 and 54 are repeated

and the code

public class Order {

    @Link Header header;

    @Link Trailer trailer;

    @KeyValuePairField(tag = 1) // Client reference
    private String account;

    @KeyValuePairField(tag = 11) // Order reference
    private String clOrdId;

    @KeyValuePairField(tag = 58) // Free text
    private String text;

    @OneToMany(mappedTo = "org.apache.camel.dataformat.bindy.model.fix.complex.onetomany.Security")
    List<Security> securities;
...

public class Security {

    @KeyValuePairField(tag = 22) // Fund ID type (Sedol, ISIN, ...)
    private String idSource;

    @KeyValuePairField(tag = 48) // Fund code
    private String securityCode;

    @KeyValuePairField(tag = 54) // Movement type ( 1 = Buy, 2 = sell)
    private String side;

Using the Java DSL

The next step consists in instantiating the DataFormat bindy class associated with this record type and providing Java package name(s) as parameter.

For example the following uses the class BindyCsvDataFormat (who correspond to the class associated with the CSV record type) which is configured with "com.acme.model"
package name to initialize the model objects configured in this package.

// Camel 2.15 or older (configure by package name)
DataFormat bindy = new BindyCsvDataFormat("com.acme.model");

 
// Camel 2.16 onwards (configure by class name)
DataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);

Setting locale

Bindy supports configuring the locale on the dataformat, such as 

// Camel 2.15 or older (configure by package name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model");
// Camel 2.16 onwards (configure by class name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);

bindy.setLocale("us");

Or to use the platform default locale then use "default" as the locale name. Notice this requires Camel 2.14/2.13.3/2.12.5.

// Camel 2.15 or older (configure by package name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model");
// Camel 2.16 onwards (configure by class name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);

bindy.setLocale("default");

for older releases you can set it using Java code as shown

// Camel 2.15 or older (configure by package name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat("com.acme.model");
// Camel 2.16 onwards (configure by class name)
BindyCsvDataFormat bindy = new BindyCsvDataFormat(com.acme.model.MyModel.class);


bindy.setLocale(Locale.getDefault().getISO3Country());

Unmarshaling

from("file://inbox")
  .unmarshal(bindy)
  .to("direct:handleOrders");

Alternatively, you can use a named reference to a data format which can then be defined in your Registry e.g. your Spring XML file:

from("file://inbox")
  .unmarshal("myBindyDataFormat")
  .to("direct:handleOrders");

The Camel route will pick-up files in the inbox directory, unmarshall CSV records into a collection of model objects and send the collection
to the route referenced by 'handleOrders'.

The collection returned is a List of Map objects. Each Map within the list contains the model objects that were marshalled out of each line of the CSV. The reason behind this is that each line can correspond to more than one object. This can be confusing when you simply expect one object to be returned per line.

Each object can be retrieve using its class name.

    List<Map<String, Object>> unmarshaledModels = (List<Map<String, Object>>) exchange.getIn().getBody();

    int modelCount = 0;
    for (Map<String, Object> model : unmarshaledModels) {
	  for (String className : model.keySet()) {
	     Object obj = model.get(className);
	     LOG.info("Count : " + modelCount + ", " + obj.toString());
	  }
	 modelCount++;
    }

    LOG.info("Total CSV records received by the csv bean : " + modelCount);

Assuming that you want to extract a single Order object from this map for processing in a route, you could use a combination of a Splitter and a Processor as per the following:

from("file://inbox")
    .unmarshal(bindy)
    .split(body())
        .process(new Processor() {
            public void process(Exchange exchange) throws Exception {
                Message in = exchange.getIn();
                Map<String, Object> modelMap = (Map<String, Object>) in.getBody();
                in.setBody(modelMap.get(Order.class.getCanonicalName()));
            }
        })
        .to("direct:handleSingleOrder")
    .end();

Take care of the fact that Bindy uses CHARSET_NAME property or the CHARSET_NAME header as define in the Exchange interface to do a characterset conversion of the inputstream received for unmarshalling. In some producers (e.g. file-endpoint) you can define a characterset. The characterset conversion can already been done by this producer. Sometimes you need to remove this property or header from the exchange before sending it to the unmarshal. If you don't remove it the conversion might be done twice which might lead to unwanted results.

from("file://inbox?charset=Cp922")
  .removeProperty(Exchange.CHARSET_NAME)
  .unmarshal("myBindyDataFormat")
  .to("direct:handleOrders");

Marshaling

To generate CSV records from a collection of model objects, you create the following route :

from("direct:handleOrders")
   .marshal(bindy)
   .to("file://outbox")

Using Spring XML

This is really easy to use Spring as your favorite DSL language to declare the routes to be used for camel-bindy. The following example shows two routes where the first will pick-up records from files, unmarshal the content and bind it to their model. The result is then send to a pojo (doing nothing special) and place them into a queue.

The second route will extract the pojos from the queue and marshal the content to generate a file containing the csv record. The example above is for using Camel 2.16 onwards.

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://camel.apache.org/schema/spring
       http://camel.apache.org/schema/spring/camel-spring.xsd">

        <!-- Queuing engine - ActiveMq - work locally in mode virtual memory -->
	<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
		<property name="brokerURL" value="vm://localhost:61616"/>
	</bean>


	<camelContext xmlns="http://camel.apache.org/schema/spring">

 
        <dataFormats>
          <bindy id="bindyDataformat" type="Csv" classType="org.apache.camel.bindy.model.Order"/>
        </dataFormats>

		<route>
			<from uri="file://src/data/csv/?noop=true" />
			<unmarshal ref="bindyDataformat" />
			<to uri="bean:csv" />
			<to uri="activemq:queue:in" />
		</route>

		<route>
			<from uri="activemq:queue:in" />
			<marshal ref="bindyDataformat" />
			<to uri="file://src/data/csv/out/" />
		</route>
	</camelContext>
</beans>

Dependencies

To use Bindy in your camel routes you need to add the a dependency on camel-bindy which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-bindy</artifactId>
  <version>x.x.x</version>
</dependency>

Options

There are no options provided for this data format.

Marshal

In this example we marshal a regular text/XML payload to a compressed payload employing gzip compression format and send it an ActiveMQ queue called MY_QUEUE.

from("direct:start").marshal().gzip().to("activemq:queue:MY_QUEUE");

Unmarshal

In this example we unmarshal a gzipped payload from an ActiveMQ queue called MY_QUEUE to its original format, and forward it for processing to the UnGZippedMessageProcessor.

from("activemq:queue:MY_QUEUE").unmarshal().gzip().process(new UnGZippedMessageProcessor()); 

Dependencies

This data format is provided in camel-core so no additional dependencies is needed.

Castor

Available as of Camel 2.1

Castor is a Data Format which uses the Castor XML library to unmarshal an XML payload into Java objects or to marshal Java objects into an XML payload.

As usually you can use either Java DSL or Spring XML to work with Castor Data Format.

Using the Java DSL

from("direct:order").
  marshal().castor().
  to("activemq:queue:order");

For example the following uses a named DataFormat of Castor which uses default Castor data binding features.

CastorDataFormat castor = new CastorDataFormat ();

from("activemq:My.Queue").
  unmarshal(castor).
  to("mqseries:Another.Queue");

If you prefer to use a named reference to a data format which can then be defined in your Registry such as via your Spring XML file. e.g.

from("activemq:My.Queue").
  unmarshal("mycastorType").
  to("mqseries:Another.Queue");

If you want to override default mapping schema by providing a mapping file you can set it as follows.

CastorDataFormat castor = new CastorDataFormat ();
castor.setMappingFile("mapping.xml");

Also if you want to have more control on Castor Marshaller and Unmarshaller you can access them as below.

castor.getMarshaller();
castor.getUnmarshaller();

Using Spring XML

The following example shows how to use Castor to unmarshal using Spring configuring the castor data type

<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
  <route>
    <from uri="direct:start"/>
    <unmarshal>
      <castor validation="true" />
    </unmarshal>
    <to uri="mock:result"/>
  </route>
</camelContext>

This example shows how to configure the data type just once and reuse it on multiple routes. You have to set the <castor> element directly in <camelContext>.

<camelContext>
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
  <dataFormats>
    <castor id="myCastor"/>
  </dataFormats>

  <route>
    <from uri="direct:start"/>
    <marshal ref="myCastor"/>
    <to uri="direct:marshalled"/>
  </route>
  <route>
    <from uri="direct:marshalled"/>
    <unmarshal ref="myCastor"/>
    <to uri="mock:result"/>
  </route>

</camelContext>

Options

Castor supports the following options

Dependencies

To use Castor in your camel routes you need to add the a dependency on camel-castor which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-castor</artifactId>
  <version>x.x.x</version>
</dependency>

Protobuf - Protocol Buffers

"Protocol Buffers - Google's data interchange format"

Camel provides a Data Format to serialse between Java and the Protocol Buffer protocol. The project's site details why you may wish to choose this format over xml. Protocol Buffer is language-neutral and platform-neutral, so messages produced by your Camel routes may be consumed by other language implementations.

API Site
Protobuf Implementation
Protobuf Java Tutorial

Protobuf overview

This quick overview of how to use Protobuf. For more detail see the complete tutorial

Defining the proto format

The first step is to define the format for the body of your exchange. This is defined in a .proto file as so:

package org.apache.camel.component.protobuf;

option java_package = "org.apache.camel.component.protobuf";
option java_outer_classname = "AddressBookProtos";

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

message AddressBook {
  repeated Person person = 1;
}

Generating Java classes

The Protobuf SDK provides a compiler which will generate the Java classes for the format we defined in our .proto file. You can run the compiler for any additional supported languages you require.

protoc --java_out=. ./addressbook.proto

This will generate a single Java class named AddressBookProtos which contains inner classes for Person and AddressBook. Builders are also implemented for you. The generated classes implement com.google.protobuf.Message which is required by the serialisation mechanism. For this reason it important that only these classes are used in the body of your exchanges. Camel will throw an exception on route creation if you attempt to tell the Data Format to use a class that does not implement com.google.protobuf.Message. Use the generated builders to translate the data from any of your existing domain classes.

Java DSL

You can use create the ProtobufDataFormat instance and pass it to Camel DataFormat marshal and unmarsha API like this.

   ProtobufDataFormat format = new ProtobufDataFormat(Person.getDefaultInstance());

   from("direct:in").marshal(format);
   from("direct:back").unmarshal(format).to("mock:reverse");

Or use the DSL protobuf() passing the unmarshal default instance or default instance class name like this.

   // You don't need to specify the default instance for protobuf marshaling               
   from("direct:marshal").marshal().protobuf();
   from("direct:unmarshalA").unmarshal().
       protobuf("org.apache.camel.dataformat.protobuf.generated.AddressBookProtos$Person").
       to ("mock:reverse");
                
   from("direct:unmarshalB").unmarshal().protobuf(Person.getDefaultInstance()).to("mock:reverse");

Spring DSL

The following example shows how to use Castor to unmarshal using Spring configuring the protobuf data type

<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
  <route>
    <from uri="direct:start"/>
    <unmarshal>
      <protobuf instanceClass="org.apache.camel.dataformat.protobuf.generated.AddressBookProtos$Person" />
    </unmarshal>
    <to uri="mock:result"/>
  </route>
</camelContext>

Dependencies

To use Protobuf in your camel routes you need to add the a dependency on camel-protobuf which implements this data format.

If you use maven you could just add the following to your pom.xml, substituting the version number for the latest & greatest release (see the download page for the latest versions).

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-protobuf</artifactId>
  <version>2.2.0</version>
</dependency>

SOAP DataFormat

Available as of Camel 2.3

SOAP is a Data Format which uses JAXB2 and JAX-WS annotations to marshal and unmarshal SOAP payloads. It provides the basic features of Apache CXF without need for the CXF Stack.

ElementNameStrategy

An element name strategy is used for two purposes. The first is to find a xml element name for a given object and soap action when marshaling the object into a SOAP message. The second is to find an Exception class for a given soap fault name.

If you have generated the web service stub code with cxf-codegen or a similar tool then you probably will want to use the ServiceInterfaceStrategy. In the case you have no annotated service interface you should use QNameStrategy or TypeNameStrategy.

Using the Java DSL

The following example uses a named DataFormat of soap which is configured with the package com.example.customerservice to initialize the JAXBContext. The second parameter is the ElementNameStrategy. The route is able to marshal normal objects as well as exceptions. (Note the below just sends a SOAP Envelope to a queue. A web service provider would actually need to be listening to the queue for a SOAP call to actually occur, in which case it would be a one way SOAP request. If you need request reply then you should look at the next example.)

SoapJaxbDataFormat soap = new SoapJaxbDataFormat("com.example.customerservice", new ServiceInterfaceStrategy(CustomerService.class));
from("direct:start")
  .marshal(soap)
  .to("jms:myQueue");

Using SOAP 1.2

Available as of Camel 2.11

SoapJaxbDataFormat soap = new SoapJaxbDataFormat("com.example.customerservice", new ServiceInterfaceStrategy(CustomerService.class));
soap.setVersion("1.2");
from("direct:start")
  .marshal(soap)
  .to("jms:myQueue");

When using XML DSL there is a version attribute you can set on the <soapjaxb> element.

    <!-- Defining a ServiceInterfaceStrategy for retrieving the element name when marshalling -->
    <bean id="myNameStrategy" class="org.apache.camel.dataformat.soap.name.ServiceInterfaceStrategy">
    	<constructor-arg value="com.example.customerservice.CustomerService"/>
	<constructor-arg value="true"/>
    </bean>

And in the Camel route

<route>
  <from uri="direct:start"/>
  <marshal>
    <soapjaxb contentPath="com.example.customerservice" version="1.2" elementNameStrategyRef="myNameStrategy"/>
  </marshal>
  <to uri="jms:myQueue"/>
</route>

Multi-part Messages

Available as of Camel 2.8.1

Multi-part SOAP messages are supported by the ServiceInterfaceStrategy. The ServiceInterfaceStrategy must be initialized with a service interface definition that is annotated in accordance with JAX-WS 2.2 and meets the requirements of the Document Bare style. The target method must meet the following criteria, as per the JAX-WS specification: 1) it must have at most one in or in/out non-header parameter, 2) if it has a return type other than void it must have no in/out or out non-header parameters, 3) if it it has a return type of void it must have at most one in/out or out non-header parameter.

The ServiceInterfaceStrategy should be initialized with a boolean parameter that indicates whether the mapping strategy applies to the request parameters or response parameters.

ServiceInterfaceStrategy strat =  new ServiceInterfaceStrategy(com.example.customerservice.multipart.MultiPartCustomerService.class, true);
SoapJaxbDataFormat soapDataFormat = new SoapJaxbDataFormat("com.example.customerservice.multipart", strat);

Multi-part Request

The payload parameters for a multi-part request are initiazlied using a BeanInvocation object that reflects the signature of the target operation. The camel-soap DataFormat maps the content in the BeanInvocation to fields in the SOAP header and body in accordance with the JAX-WS mapping when the marshal() processor is invoked.

BeanInvocation beanInvocation = new BeanInvocation();

// Identify the target method
beanInvocation.setMethod(MultiPartCustomerService.class.getMethod("getCustomersByName", 
    GetCustomersByName.class, com.example.customerservice.multipart.Product.class));

// Populate the method arguments
GetCustomersByName getCustomersByName = new GetCustomersByName();
getCustomersByName.setName("Dr. Multipart");
                
Product product = new Product();
product.setName("Multiuse Product");
product.setDescription("Useful for lots of things.");
                
Object[] args = new Object[] {getCustomersByName, product};

// Add the arguments to the bean invocation
beanInvocation.setArgs(args);

// Set the bean invocation object as the message body
exchange.getIn().setBody(beanInvocation); 

Multi-part Response

A multi-part soap response may include an element in the soap body and will have one or more elements in the soap header. The camel-soap DataFormat will unmarshall the element in the soap body (if it exists) and place it onto the body of the out message in the exchange. Header elements will not be marshaled into their JAXB mapped object types. Instead, these elements are placed into the camel out message header org.apache.camel.dataformat.soap.UNMARSHALLED_HEADER_LIST. The elements will appear either as element instance values, or as JAXBElement values, depending upon the setting for the ignoreJAXBElement property. This property is inherited from camel-jaxb.

You can also have the camel-soap DataFormate ignore header content all-together by setting the ignoreUnmarshalledHeaders value to true.

Holder Object mapping

JAX-WS specifies the use of a type-parameterized javax.xml.ws.Holder object for In/Out and Out parameters. A Holder object may be used when building the BeanInvocation, or you may use an instance of the parameterized-type directly. The camel-soap DataFormat marshals Holder values in accordance with the JAXB mapping for the class of the Holder's value. No mapping is provided for Holder objects in an unmarshalled response.

Examples

Webservice client

The following route supports marshalling the request and unmarshalling a response or fault.

String WS_URI = "cxf://http://myserver/customerservice?serviceClass=com.example.customerservice&dataFormat=MESSAGE";
SoapJaxbDataFormat soapDF = new SoapJaxbDataFormat("com.example.customerservice", new ServiceInterfaceStrategy(CustomerService.class));
from("direct:customerServiceClient")
  .onException(Exception.class)
    .handled(true)
    .unmarshal(soapDF)
  .end()
  .marshal(soapDF)
  .to(WS_URI)
  .unmarshal(soapDF);

The below snippet creates a proxy for the service interface and makes a SOAP call to the above route.

import org.apache.camel.Endpoint;
import org.apache.camel.component.bean.ProxyHelper;
...

Endpoint startEndpoint = context.getEndpoint("direct:customerServiceClient");
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
// CustomerService below is the service endpoint interface, *not* the javax.xml.ws.Service subclass
CustomerService proxy = ProxyHelper.createProxy(startEndpoint, classLoader, CustomerService.class);
GetCustomersByNameResponse response = proxy.getCustomersByName(new GetCustomersByName());

Webservice Server

Using the following route sets up a webservice server that listens on jms queue customerServiceQueue and processes requests using the class CustomerServiceImpl. The customerServiceImpl of course should implement the interface CustomerService. Instead of directly instantiating the server class it could be defined in a spring context as a regular bean.

SoapJaxbDataFormat soapDF = new SoapJaxbDataFormat("com.example.customerservice", new ServiceInterfaceStrategy(CustomerService.class));
CustomerService serverBean = new CustomerServiceImpl();
from("jms://queue:customerServiceQueue")
  .onException(Exception.class)
    .handled(true)
    .marshal(soapDF)
  .end()
  .unmarshal(soapDF)
  .bean(serverBean)
  .marshal(soapDF);

Dependencies

To use the SOAP dataformat in your camel routes you need to add the following dependency to your pom.

<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-soap</artifactId>
  <version>2.3.0</version>
</dependency>

Crypto

Available as of Camel 2.3
PGP Available as of Camel 2.9

The Crypto Data Format integrates the Java Cryptographic Extension into Camel, allowing simple and flexible encryption and decryption of messages using Camel's familiar marshall and unmarshal formatting mechanism. It assumes marshalling to mean encryption to cyphertext and unmarshalling to mean decryption back to the original plaintext. This data format implements only symmetric (shared-key) encryption and decyption.

Options

Basic Usage

At its most basic all that is required to encrypt/decrypt an exchange is a shared secret key. If one or more instances of the Crypto data format are configured with this key the format can be used to encrypt the payload in one route (or part of one) and decrypted in another. For example, using the Java DSL as follows:{snippet:id=basic|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/CryptoDataFormatTest.java}In Spring the dataformat is configured first and then used in routes

Specifying the Encryption Algorithm

Changing the algorithm is a matter of supplying the JCE algorithm name. If you change the algorithm you will need to use a compatible key.{snippet:id=algorithm|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/CryptoDataFormatTest.java}A list of the available algorithms in Java 7 is available via the Java Cryptography Architecture Standard Algorithm Name Documentation.

Specifying an Initialization Vector

Some crypto algorithms, particularly block algorithms, require configuration with an initial block of data known as an Initialization Vector. In the JCE this is passed as an AlgorithmParameterSpec when the Cipher is initialized. To use such a vector with the CryptoDataFormat you can configure it with a byte[] containing the required data e.g.{snippet:id=init-vector|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/CryptoDataFormatTest.java}or with spring, suppling a reference to a byte[]{snippet:id=init-vector|lang=xml|url=camel/trunk/components/camel-crypto/src/test/resources/org/apache/camel/component/crypto/SpringCryptoDataFormatTest.xml}The same vector is required in both the encryption and decryption phases. As it is not necessary to keep the IV a secret, the DataFormat allows for it to be inlined into the encrypted data and subsequently read out in the decryption phase to initialize the Cipher. To inline the IV set the /oinline flag.{snippet:id=inline-init-vector|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/CryptoDataFormatTest.java}or with spring.{snippet:id=inline|lang=xml|url=camel/trunk/components/camel-crypto/src/test/resources/org/apache/camel/component/crypto/SpringCryptoDataFormatTest.xml}For more information of the use of Initialization Vectors, consult

• http://en.wikipedia.org/wiki/Initialization_vector
• http://www.herongyang.com/Cryptography/
• http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation

Hashed Message Authentication Codes (HMAC)

To avoid attacks against the encrypted data while it is in transit the CryptoDataFormat can also calculate a Message Authentication Code for the encrypted exchange contents based on a configurable MAC algorithm. The calculated HMAC is appended to the stream after encryption. It is separated from the stream in the decryption phase. The MAC is recalculated and verified against the transmitted version to insure nothing was tampered with in transit.For more information on Message Authentication Codes see http://en.wikipedia.org/wiki/HMAC{snippet:id=hmac|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/CryptoDataFormatTest.java}or with spring.{snippet:id=hmac|lang=xml|url=camel/trunk/components/camel-crypto/src/test/resources/org/apache/camel/component/crypto/SpringCryptoDataFormatTest.xml}By default the HMAC is calculated using the HmacSHA1 mac algorithm though this can be easily changed by supplying a different algorithm name. See here for how to check what algorithms are available through the configured security providers{snippet:id=hmac-algorithm|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/CryptoDataFormatTest.java}or with spring.{snippet:id=hmac-algorithm|lang=xml|url=camel/trunk/components/camel-crypto/src/test/resources/org/apache/camel/component/crypto/SpringCryptoDataFormatTest.xml}

Supplying Keys Dynamically

When using a Recipient list or similar EIP the recipient of an exchange can vary dynamically. Using the same key across all recipients may neither be feasible or desirable. It would be useful to be able to specify keys dynamically on a per exchange basis. The exchange could then be dynamically enriched with the key of its target recipient before being processed by the data format. To facilitate this the DataFormat allow for keys to be supplied dynamically via the message headers below

• CryptoDataFormat.KEY "CamelCryptoKey"

{snippet:id=key-in-header|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/CryptoDataFormatTest.java}or with spring.{snippet:id=header-key|lang=xml|url=camel/trunk/components/camel-crypto/src/test/resources/org/apache/camel/component/crypto/SpringCryptoDataFormatTest.xml}

PGP Message

The PGP Data Formater can create and decrypt/verify PGP Messages of the following PGP packet structure (entries in brackets are optional and ellipses indicate repetition, comma represents  sequential composition, and vertical bar separates alternatives):

    Public Key Encrypted Session Key ..., Symmetrically Encrypted Data | Sym. Encrypted and Integrity Protected Data, (Compressed Data,) (One Pass Signature ...,) Literal Data, (Signature ...,)

Since Camel 2.16.0 the Compressed Data packet is optional, before it was mandatory.

PGPDataFormat Options

PGPDataFormat Message Headers

You can override the PGPDataFormat options by applying below headers into message dynamically.

Encrypting with PGPDataFormat

The following sample uses the popular PGP format for encrypting/decrypting files using the Bouncy Castle Java libraries:{snippet:id=pgp-format|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/PGPDataFormatTest.java}The following sample performs signing + encryption, and then signature verification + decryption. It uses the same keyring for both signing and encryption, but you can obviously use different keys:{snippet:id=pgp-format-signature|lang=java|url=camel/trunk/components/camel-crypto/src/test/java/org/apache/camel/converter/crypto/PGPDataFormatTest.java}Or using Spring:{snippet:id=pgp-xml-basic|lang=xml|url=camel/trunk/components/camel-crypto/src/test/resources/org/apache/camel/component/crypto/SpringPGPDataFormatTest.xml}

To work with the previous example you need the following

• A public keyring file which contains the public keys used to encrypt the data
• A private keyring file which contains the keys used to decrypt the data
• The keyring password

Managing your keyring

To manage the keyring, I use the command line tools, I find this to be the simplest approach in managing the keys. There are also Java libraries available from http://www.bouncycastle.org/java.html if you would prefer to do it that way.

1. Install the command line utilities on linux

   apt-get install gnupg

2. Create your keyring, entering a secure password

   gpg --gen-key

3. If you need to import someone elses public key so that you can encrypt a file for them.

   gpg --import <filename.key

4. The following files should now exist and can be used to run the example

   ls -l ~/.gnupg/pubring.gpg ~/.gnupg/secring.gpg

PGP Decrypting/Verifying of Messages Encrypted/Signed by Different Private/Public Keys

Since Camel 2.12.2.

A PGP Data Formater can decrypt/verify messages which have been encrypted by different public keys or signed by different private keys. Just, provide the corresponding private keys in the secret keyring, the corresponding public keys in the public keyring, and the passphrases in the passphrase accessor.

• The functionality is especially useful to support the key exchange. If you want to exchange the private key for decrypting you can accept for a period of time messages which are either encrypted with the old or new corresponding public key. Or if the sender wants to exchange his signer private key, you can accept for a period of time, the old or new signer key.
• Technical background: The PGP encrypted data contains a Key ID of the public key which was used to encrypt the data. This Key ID can be used to locate the private key in the secret keyring to decrypt the data. The same mechanism is also used to locate the public key for verifying a signature. Therefore you no longer must specify User IDs for the unmarshaling.

Restricting the Signer Identities during PGP Signature Verification

Since Camel 2.12.3.

If you verify a signature you not only want to verify the correctness of the signature but you also want check that the signature comes from a certain identity or a specific set of identities. Therefore it is possible to restrict the number of public keys from the public keyring which can be used for the verification of a signature.  

• If the PGP content has several signatures the verification is successful as soon as one signature can be verified.
• If you do not want to restrict the signer identities for verification then do not specify the signature key User IDs. In this case all public keys in the public keyring are taken into account.

Several Signatures in One PGP Data Format

Since Camel 2.12.3.

The PGP specification allows that one PGP data format can contain several signatures from different keys. Since Camel 2.13.3 it is possible to create such kind of PGP content via specifying signature User IDs which relate to several private keys in the secret keyring.

Support of Sub-Keys and Key Flags in PGP Data Format Marshaler

Since Camel 2.12.3.
An OpenPGP V4 key can have a primary key and sub-keys. The usage of the keys is indicated by the so called Key Flags. For example, you can have a primary key with two sub-keys; the primary key shall only be used for certifying other keys (Key Flag 0x01), the first sub-key  shall only be used for signing (Key Flag 0x02), and the second sub-key shall only be used for encryption (Key Flag 0x04 or 0x08). The PGP Data Format marshaler takes into account these Key Flags of the primary key and sub-keys in order to determine the right key for signing and encryption. This is necessary because the primary key and its sub-keys have the same User IDs.

Support of Custom Key Accessors

Since Camel 2.13.0.
You can implement custom key accessors for encryption/signing. The above PGPDataFormat class selects in a certain predefined way the keys which should be used for signing/encryption or verifying/decryption. If you have special requirements how your keys should be selected you should use the PGPKeyAccessDataFormat class instead and implement the interfaces PGPPublicKeyAccessor and PGPSecretKeyAccessor as beans. There are default implementations DefaultPGPPublicKeyAccessor and DefaultPGPSecretKeyAccessor which cache the keys, so that not every time the keyring is parsed when the processor is called.

PGPKeyAccessDataFormat has the same options as PGPDataFormat except password, keyFileName, encryptionKeyRing, signaturePassword, signatureKeyFileName, and signatureKeyRing.

Dependencies

To use the Crypto dataformat in your camel routes you need to add the following dependency to your pom.

See Also

• Data Format
• Crypto (Digital Signatures)
• http://www.bouncycastle.org/java.html