This tutorial is a work in progress. |
So there's a company, which we'll call Acme. Acme sells widgets, in a fairly unusual way. Their customers are responsible for telling Acme what they purchased. The customer enters into their own systems (ERP or whatever) which widgets they bought from Acme. Then at some point, their systems emit a record of the sale which needs to go to Acme so Acme can bill them for it. Obviously, everyone wants this to be as automated as possible, so there needs to be integration between the customer's system and Acme.
Sadly, Acme's sales people are, technically speaking, doormats. They tell all their prospects, "you can send us the data in whatever format, using whatever protocols, whatever. You just can't change once it's up and running."
The result is pretty much what you'd expect. Taking a random sample of 3 customers:
Now on the Acme side, all this has to be converted to a canonical XML format and submitted to the Acme accounting system via JMS. Then the Acme accounting system does its stuff and sends an XML reply via JMS, with a summary of what it processed (e.g. 3 line items accepted, line item #2 in error, total invoice $123.45). Finally, that data needs to be formatted into an e-mail, and sent to a contact at the customer in question ("Dear Joyce, we received an invoice on 1/2/08. We accepted 3 line items totaling $123.45, though there was an error with line items #2 [invalid quantity ordered]. Thank you for your business. Love, Acme.").
So it turns out Camel can handle all this:
This tutorial will cover all that, plus setting up tests along the way.
Before starting, you should be familiar with:
You'll learn:
You may choose to treat this as a hands-on tutorial, and work through building the code and configuration files yourself. Each of the sections gives detailed descriptions of the steps that need to be taken to get the components and routes working in Camel, and takes you through tests to make sure they are working as expected.
But each section also links to working copies of the source and configuration files, so if you don't want the hands-on approach, you can simply review and/or download the finished files.
Here's more or less what the integration process looks like.
First, the input from the customers to Acme:
And then, the output from Acme to the customers:
To get through this scenario, we're going to break it down into smaller pieces, implement and test those, and then try to assemble the big scenario and test that.
Here's what we'll try to accomplish:
List<List<String>>
to the above JAXB POJOs
We'll use Maven for this project as there will eventually be quite a few dependencies and it's nice to have Maven handle them for us. You should have a current version of Maven (e.g. 2.0.9) installed.
You can start with a pretty empty project directory and a Maven POM file, or use a simple JAR archetype to create one.
Here's a sample POM. We've added a dependency on camel-core, and set the compile version to 1.5 (so we can use annotations):
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>org.apache.camel.tutorial</groupId> <artifactId>business-partners</artifactId> <version>1.0-SNAPSHOT</version> <name>Camel Business Partners Tutorial</name> <dependencies> <dependency> <artifactId>camel-core</artifactId> <groupId>org.apache.camel</groupId> <version>1.4.0</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.5</source> <target>1.5</target> </configuration> </plugin> </plugins> </build> </project> |
You can make up your own if you like, but here are the "off the shelf" ones. You can save yourself some time by downloading these to src/test/resources
in your Maven project.
If you look at these files, you'll see that the different input formats use different field names and/or ordering, because of course the sales guys were totally OK with that. Sigh.
Here's the sample of the canonical XML file:
<?xml version="1.0" encoding="UTF-8"?> <invoice xmlns="http://activemq.apache.org/camel/tutorial/partners/invoice"> <partner-id>2</partner-id> <date-received>9/12/2008</date-received> <line-item> <product-id>134</product-id> <description>A widget</description> <quantity>3</quantity> <item-price>10.45</item-price> <order-date>6/5/2008</order-date> </line-item> <!-- // more line-item elements here --> <order-total>218.82</order-total> </invoice> |
If you're ambitions, you can write your own XSD (XML Schema) for files that look like this, and save it to src/main/xsd
.
Solution: If not, you can download mine, and save that to save it to src/main/xsd
.
Down the road we'll want to deal with the XML as Java POJOs. We'll take a moment now to set up those XML binding POJOs. So we'll update the Maven POM to generate JAXB beans from the XSD file.
We need a dependency:
<dependency> <artifactId>camel-jaxb</artifactId> <groupId>org.apache.camel</groupId> <version>1.4.0</version> </dependency> |
And a plugin configured:
<plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>jaxb2-maven-plugin</artifactId> <executions> <execution> <goals> <goal>xjc</goal> </goals> </execution> </executions> </plugin> |
That should do it (it automatically looks for XML Schemas in src/main/xsd
to generate beans for). Run mvn install and it should emit the beans into target/generated-sources/jaxb
. Your IDE should see them there, though you may need to update the project to reflect the new settings in the Maven POM.
To get a start on Customer 1, we'll create an XSLT template to convert the Customer 1 sample file into the canonical XML format, write a small Camel route to test it, and build that into a unit test. If we get through this, we can be pretty sure that the XSLT template is valid and can be run safely in Camel.
Start with the Customer 1 sample input. You want to create an XSLT template to generate XML like the canonical XML sample above – an invoice
element with line-item
elements (one per item in the original XML document). If you're especially clever, you can populate the current date and order total elements too.
Solution: My sample XSLT template isn't that smart, but it'll get you going if you don't want to write one of your own.
Here's where we get to some meaty Camel work. We need to:
The easiest way to do this is to set up a Spring context that defines the Camel stuff, and then use a base unit test class from Spring that knows how to load a Spring context to run tests against. So, the procedure is:
<dependency> <artifactId>camel-spring</artifactId> <groupId>org.apache.camel</groupId> <version>1.4.0</version> </dependency> <dependency> <artifactId>spring-test</artifactId> <groupId>org.springframework</groupId> <version>2.5.5</version> <scope>test</scope> </dependency> |
src/test/java/your-package-here
, perhaps called XMLInputTest.java
src/test/resources
, perhaps called XMLInputTest-context.xml
TestClassName-context.xml
in a subdirectory corresponding to the package of the test class. For instance, if your test class was org.apache.camel.tutorial.XMLInputTest
, it would look for org/apache/camel/tutorial/XMLInputTest-context.xml
src/test/resources
instead of in a package directory under there.setUp
method that instantiates it from the CamelContext. We'll use the ProducerTemplate later to send messages to the route.
protected ProducerTemplate<Exchange> template; protected void setUp() throws Exception { super.setUp(); template = camelContext.createProducerTemplate(); } |
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring-1.4.0.xsd"> <camelContext id="camel" xmlns="http://activemq.apache.org/camel/schema/spring"> </camelContext> </beans> |
Test it by running mvn install and make sure there are no build errors. So far it doesn't test much; just that your project and test and source files are all organized correctly, and the one empty test method completes successfully.
Solution: Your test class might look something like this:
So now we're going to write a Camel route that applies the XSLT to the sample Customer 1 input file, and makes sure that some XML output comes out:
src/test/resources
src/main/resources
src/test/java
somewhere). This route should use the Pipes and Filters integration pattern to:
mock:finish
using code like this:
MockEndpoint finish = MockEndpoint.resolve(camelContext, "mock:finish"); |
InputStream in = XMLInputTest.class.getResourceAsStream("/input-partner1.xml"); assertNotNull(in); |
direct:start
endpoint, using code like this:
template.sendBody("direct:start", in); |
MockEndpoint.assertIsSatisfied(camelContext); |
finish.getExchanges().get(0).getIn().getBody()
.
Solution: Your finished test might look something like this:
Once your test class is working, you might want to extract things like the @Autowired CamelContext, the ProducerTemplate, and the setUp method to a custom base class that you extend with your other tests. |
To get a start on Customer 2, we'll create a POJO to convert the Customer 2 sample CSV data into the JAXB POJOs representing the canonical XML format, write a small Camel route to test it, and build that into a unit test. If we get through this, we can be pretty sure that the CSV conversion and JAXB handling is valid and can be run safely in Camel.
To begin with, CSV is a known data format in Camel. Camel can convert a CSV file to a List (representing rows in the CSV) of Lists (representing cells in the row) of Strings (the data for each cell). That means our POJO can just assume the data coming in is of type List<List<String>>
, and we can declare a method with that as the argument.
Looking at the JAXB code in target/generated-sources/jaxb
, it looks like an Invoice
object represents the whole document, with a nested list of LineItemType objects for the line items. Therefore our POJO method will return an Invoice
(a document in the canonical XML format).
So to implement the CSV-to-JAXB POJO, we need to do something like this:
src/main/java
, perhaps called CSVConverterBean
.List<List<String>>
and the return type Invoice
Invoice
, using the method on the generated ObjectFactory
classList
)LineItemType
(using the ObjectFactory
again)List
) and put them into the correct fields of the LineItemType
DatatypeFactory
to create the XMLGregorianCalendar
values that JAXB uses for the date
fields in the XML – which probably means using a SimpleDateFormat
to parse the date and setting that date on a GregorianCalendar
Invoice
Invoice
Solution: Here's an example of what the CSVConverterBean might look like.
Start with a simple test class and test Spring context like last time, perhaps based on the name CSVInputTest
:
/** * A test class the ensure we can convert Partner 2 CSV input files to the * canonical XML output format, using JAXB POJOs. */ @ContextConfiguration(locations = "/CSVInputTest-context.xml") public class CSVInputTest extends AbstractJUnit38SpringContextTests { @Autowired protected CamelContext camelContext; protected ProducerTemplate<Exchange> template; protected void setUp() throws Exception { super.setUp(); template = camelContext.createProducerTemplate(); } public void testCSVConversion() { // TODO } } |
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring-1.4.0.xsd"> <camelContext id="camel" xmlns="http://activemq.apache.org/camel/schema/spring"> <!-- TODO --> </camelContext> </beans> |
Now the meaty part is to flesh out the test class and write the Camel routes.
<dependency> <artifactId>camel-csv</artifactId> <groupId>org.apache.camel</groupId> <version>1.4.0</version> </dependency> |
List<List<String>>
without a little hint. For that, we need an unmarshal transformation in the route. The unmarshal
method (in the DSL) or element (in the XML) takes a child indicating the format to unmarshal; in this case that should be csv
.<bean>
element in the Spring context XML file (but outside the <camelContext>
element) to define the Spring bean that our route invokes. This Spring bean should have a name
attribute that matches the name used in the bean
endpoint (CSVConverter
in the example above), and a class
attribute that points to the CSV-to-JAXB POJO class you wrote above (such as, org.apache.camel.tutorial.CSVConverterBean
). When Spring is in the picture, any bean
endpoints look up Spring beans with the specified name.Invoice
as the body. You could write a simple line of code to get the Exchange (and Message) from the MockEndpoint to confirm that.Solution: Your finished test might look something like this:
To get a start on Customer 3, we'll create a POJO to convert the Customer 3 sample Excel data into the JAXB POJOs representing the canonical XML format, write a small Camel route to test it, and build that into a unit test. If we get through this, we can be pretty sure that the Excel conversion and JAXB handling is valid and can be run safely in Camel.
Camel does not have a data format handler for Excel by default. We have two options – create an Excel DataFormat (so Camel can convert Excel spreadsheets to something like the CSV List<List<String>>
automatically), or create a POJO that can translate Excel data manually. For now, the second approach is easier (if we go the DataFormat
route, we need code to both read and write Excel files, whereas otherwise read-only will do).
So, we need a POJO with a method that takes something like an InputStream
or byte[]
as an argument, and returns in Invoice
as before. The process should look something like this:
<dependency> <artifactId>poi</artifactId> <groupId>org.apache.poi</groupId> <version>3.1-FINAL</version> </dependency> |
src/main/java
, perhaps called ExcelConverterBean
.InputStream
and the return type Invoice
Invoice
, using the method on the generated ObjectFactory
classInputStream
, and get the first sheet from itLineItemType
(using the ObjectFactory
again)LineItemType
(you'll need some data type conversion logic)
DatatypeFactory
to create the XMLGregorianCalendar
values that JAXB uses for the date
fields in the XML – which probably means setting the date from a date cell on a GregorianCalendar
Invoice
Invoice
Solution: Here's an example of what the ExcelConverterBean might look like.
The unit tests should be pretty familiar now. The test class and context for the Excel bean should be quite similar to the CSV bean.
unmarshal
step since the Excel POJO takes the raw InputStream
from the source endpoint<bean>
and endpoint for the Excel bean prepared above instead of the CSV bean
You may notice that your tests emit a lot less output all of a sudden. The dependency on POI brought in Log4J and configured commons-logging to use it, so now we need a log4j.properties file to configure log output. You can use the attached one (snarfed from ActiveMQ) or write your own; either way save it to |
Solution: Your finished test might look something like this:
With all the data type conversions working, the next step is to write the real routes that listen for HTTP, FTP, or e-mail input, and write the final XML output to an ActiveMQ queue. Along the way these routes will use the data conversions we've developed above.
So we'll create 3 routes to start with, as shown in the diagram back at the beginning:
...