Extract Transform Load (ETL) Example
The ETL (Extract, Transform, Load) example shows how to load data into a database using Camel. In this example we will poll for files, transform them and then store them in the database via the JPA component.
This example lives in the examples/camel-example-etl directory and will poll for XML files in the child src/data directory. When the files are detected, they are converted, using the fallback JAXB Type Converter to a PersonDocument class. This POJO is then transformed using a custom Type Converter into a CustomerEntity bean using the CustomerTransformer class. Then this bean is stored in the database via the JPA component.
The code for this example is as follows
The JPA entity bean (i.e. a POJO with @Entity)
The JAXB2 POJO used to parse the XML
The custom Type Converter used to convert a PersonDocument into a CustomerEntity
Then there is the spring configuration file in src/resources/META-INF/services/camel-context.xml which defines the JPA template and tells Camel to look in the org.apache.camel.example.etl package to find its routes.
So lets start with the route definition in EtlRoutes
The above sets up a route from the src/data directory. Notice we're using the noop mode of the File component so that the files are not moved or deleted when they are processed (so when the tool is restarted they will be processed again). Because the file consumer uses a memory based Idempotent Consumer every file is only processed once (per program run) (see also noop and idempotent* options from file component).
Then we send the message with a PersonDocument body to the JPA endpoint. Notice how this endpoint specifies the expected type. So the Type Converter is gonna try convert the PersonDocument to a CustomerEntity. Here Camel will find the CustomerTransformer class which has an @Converter method
which performs the necessary conversion to an entity bean which is then stored in the database
Running the example
To run the example we use the Camel Maven Plugin. For example from the source or binary distribution the following should work
If you prefer you can just run the Main directly using
Please note that when you run the example for the first time, the converter
CustomerTransformer will not be able to find any entities inside the database, so that along the logs written into the console you should see:
However running the example for a second time, as the entites have been already inserted into the database, the log should now say:
Failing that you can run the Main from inside your IDE if you prefer. Follow the Building instructions to create an Eclipse/IDEA project to import