Now that we have created a new telemetry we can see how we can add new enrichments to that telemetry. In this exercise we will be looking at adding a whois enrichment to the Squid telemetry we setup in the previous entry. Whois data is expensive so we will not be providing it. Instead I wrote a basic whois scraper (out of context for this exercise) that produces a CSV format for whois data as follows:
google.com, "Google Inc.", "US", "Dns Admin",874306800000
work.net, "", "US", "PERFECT PRIVACY, LLC",788706000000
capitalone.com, "Capital One Services, Inc.", "US", "Domain Manager",795081600000
cisco.com, "Cisco Technology Inc.", "US", "Info Sec",547988400000
cnn.com, "Turner Broadcasting System, Inc.", "US", "Domain Name Manager",748695600000
news.com, "CBS Interactive Inc.", "US", "Domain Admin",833353200000
nba.com, "NBA Media Ventures, LLC", "US", "C/O Domain Administrator",786027600000
espn.com, "ESPN, Inc.", "US", "ESPN, Inc.",781268400000
pravda.com, "Internet Invest, Ltd. dba Imena.ua", "UA", "Whois privacy protection service",806583600000
hortonworks.com, "Hortonworks, Inc.", "US", "Domain Administrator",1303427404000
microsoft.com, "Microsoft Corporation", "US", "Domain Administrator",673156800000
yahoo.com, "Yahoo! Inc.", "US", "Domain Administrator",790416000000
rackspace.com, "Rackspace US, Inc.", "US", "Domain Admin",903092400000
Please cut and paste this data into a file called "whois_ref.csv" on your virtual machine.
The schema of this enrichment is domain|owner|registeredCountry|registeredTimestamp. The first thing we need to do is setup the enrichment source. In order to do this we first need to setup the extractor config as so:
{ "config" : { "columns" : {"domain" : 0
,"owner" : 1
,"home_country" : 2
,"registrar": 3
,"domain_created_timestamp": 4
}
,"indicator_column" : "domain"
,"type" : "whois"
,"separator" : ","
}
,"extractor" : "CSV"
}
Please cut and paste this file into a file called "extractor_config.json" on the virtual machine.
And another config to load the zookeeper enrichment config:
{"zkQuorum" : "node1:2181"
,"sensorToFieldList" : { "squid" : {"type" : "ENRICHMENT"
,"fieldToEnrichmentTypes" : {"url" : [ "whois" ]
}
}
}
}
Please cut and paste this file into a file called "enrichment_config.json" on the virtual machine.
Which means that the system will map the whois enrichment to the field URL. Then execute the following command:
/usr/metron/0.1BETA/bin/flatfile_loader.sh -n enrichment_config.json -i whois_ref.csv -t enrichment -c t -e extractor_config.json
After this your enrichment data will be loaded in Hbase and a Zookeeper mapping will be established. The data will be populated into Hbase table called enrichment. To verify that the logs were properly ingested into Hbase run the following command
hbase shell
scan 'enrichment'
You should see the table bulk loaded with data from the CSV file. Now check if Zookeeper enrichment tag was properly populated:
/usr/metron/0.1BETA/bin/zk_load_configs.sh -z localhost:2181
This spits out all of the configs to standard out, you should find one named "squid."
Now the url field should be enriched with the whois data.
An example document that you can find from the head document would look like:
{
"_index": "squid_index_2016.04.29.15",
"_type": "squid_doc",
"_id": "AVRivF0oROvCuErBOREl",
"_version": 1,
"_score": 1,
"_source": {
"adapter.simplehbaseadapter.end.ts": "1461945392270",
"code": 200,
"UNWATED": "DIRECT",
"enrichmentsplitterbolt.splitter.end.ts": "1461945391650",
"enrichmentsplitterbolt.splitter.begin.ts": "1461945391640",
"index.elasticsearchwriter.ts": "1461945392272",
"elapsed": 161,
"ip_dst_addr": "23.235.40.73",
"original_string": "1461941347.920 161 127.0.0.1 TCP_MISS\/200 102534 GET http:\/\/www.cnn.com\/ - DIRECT\/23.235.40.73 text\/html",
"threatinteljoinbolt.joiner.ts": "1461945392271",
"enrichmentjoinbolt.joiner.ts": "1461945392271",
"action": "TCP_MISS",
"threatintelsplitterbolt.splitter.begin.ts": "1461945392271",
"enrichments.hbaseEnrichment.url.whois.registrar": "Domain Name Manager",
"ip_src_addr": "127.0.0.1",
"timestamp": 1461941347920,
"enrichments.hbaseEnrichment.url.whois.domain": "cnn.com",
"method": "GET",
"enrichments.hbaseEnrichment.url.whois.home_country": "US",
"adapter.simplehbaseadapter.begin.ts": "1461945391651",
"url": "cnn.com",
"source.type": "squid",
"enrichments.hbaseEnrichment.url.whois.domain_created_timestamp": "748695600000",
"enrichments.hbaseEnrichment.url.whois.owner": "Turner Broadcasting System, Inc.",
"threatintelsplitterbolt.splitter.end.ts": "1461945392271",
"bytes": 102534
}
}