...
The tokenizer language is a built-in language in camel-core, which is most often used only with the Splitter EIP to split a message using a token-based strategy.
The tokenizer language is intended to tokenize text documents using a specified delimiter pattern. It can also be used to tokenize XML documents with some limited capability. For a truly XML-aware tokenization, the use of the xml tokenizer XMLTokenizer language is recommended as it offers a faster, more efficient tokenization specifically for XML documents. For more details see Splitter.