This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Skip to end of metadata
Go to start of metadata

Metron Parsing Topology (also known as the Normalizing Topology) is designed to take a sensor input (in it's native format) and turn it into a Metron JSON Object.  The topology consists of two components: a standard Storm Kafka Spout (reads data from a Kafka topics and ingests it into a storm topology) and a Metron Parser Kafka Bolt (which parses the message and puts it back onto a Kafka "enrichment" topic.  

 

The Kafka Parser Bolt is an extensible bolt that can take two types of parsers: GROK and purpose-built Java.  It is our strong preference to build up as much coverage as we can using the Grok parsers, as they require no coding.  However, the drawback of the Grok parsers is that they are much slower than Java parsers and sometimes the telemetry is so complex that writing a Grok statement for it is not feasible.  See our list of supported devices for which parser type supports a particular device.

 

Parser TypeTelemetry TypeParser Reference
GrokSimple to parse, low volumeGrok Parsers
JavaComplex to parse, high volumeJava Parsers

 

Each parser outputs a Metron JSON formatted message.  The reason why JSON was chosen as a format for Metron is a very flexible data structure that allows nesting and lists.  We do not dictate what the JSON structure should look like, however, Metron is looking for specific normalized fields in order to do downstream processing.  

 

  • No labels