This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Skip to end of metadata
Go to start of metadata

Enrichments add additional context to the streaming message.  For example, if a given message has an external IP an enrichment would be to tag geo data to that message.  Another example would be if a message contains a domain name then we can tag a whois entry to that message.  There are three primary benefits to adding context via enrichments to a message:

  • Correlation: if you know which user and asset the message is intended to and where it's coming from it's easier to correlated it with other related messages 
  • ML: having full context via streaming allows scoring against ML models via real time as opposed to gathering the context in batch and then applying the model in batch 
  • Accuracy: the underlying enrichment information always changes (users sign on and off, machines change IPs, etc) and you want to enrich as close to the capture time as possible
  • Investigation: having a full context for a given piece of metadata or alert means less consoles to fumble through and gets us closer to the 'single pane of glass' interface 

Metron currently provides an extensible framework to plug in enrichments.  Each enrichment has two components: an enrichment data source and and enrichment bolt.

Prior to enabling an enrichment capability within Metron the enrichment store (which for Metron is primarily Hbase) has to be loaded with enrichment data.  Enrichment data can either be bulk loaded from HDFS or be streamed into enrichment store via pluggable loading framework.  The enrichment loader transforms the enrichment into a JSON format that is understandable to Metron.  The loading framework has additional capabilities for aging data out of the enrichment stores based on time.  Once the stores are loaded an enrichment bolt that can interact with the enrichment store can be incorporated into the enrichment topology.  Each enrichment bolt can enrich a specific field/tag within a Metron message.  When a bolt recognizes that it is able to enrich a field it reaches into the enrichment store, pulls out the enrichment, and tags the message with the enrichment.  The enrichment is then stored within the bolt's in-memory cache.  Metron uses the underlying Storm routing capabilities to make sure that similar enrichment values are sent to the appropriate bolts that already have these values cached in-memory, thereby giving Metron it's superior scale and speed when compared to other big data streaming systems that do not have this capability.  

 

The following list of enrichments is Currently supported in Metron:

 

EnrichmentDescriptionEnrichment StoreEnrichment SourceMetron Message Field Name(s)Loader TypeRefresh RateMetron Enrichment Architecture
GeoIPTags on GeoIP (lat-lon coordinates + City/State/Country) to any external IP address. This can be applied both to alerts as well as metadata telemetries to be able to map them to a geo location.MySQL

Maxmind Geolite

http://dev.maxmind.com/geoip/legacy/geolite/

src_ip, dst_ipBulk from HDFSOnce every 3 monthsGeo Enrichment
AssetGiven an IP, figure out the host name of the asset. Then given the hostname of the asset tell me everything else about that asset that is known from LDAP, AD, or enterprise inventory storesHBaseLDAP, AD, DNS logs, enterprise inventory storessrc_ip, dst_ipNot yet provided. Roadmap item Once every hourAsset Enrichment
UserGiven a session or an alert for a certain ip-application pair, tell me which user this session/alert belongs toHbaseLDAP, AD, proxy logssrc_ip + applicationNot yet provided. Roadmap item Once every 5 minutesUser Enrichment
 More to come....      
  • No labels