...
And another config to load the zookeeper enrichment config. Be sure to replace the $ZOOKEEPER placeholder with your Zookeeper quorum list:
{
"zkQuorum" : "$ZOOKEEPER_HOME:2181"
,"sensorToFieldList" : {
"squid" : {
"type" : "ENRICHMENT"
,"fieldToEnrichmentTypes" : {
"domain_without_subdomains" : [ "whois" ]
}
}
}
}
...
Which means that the system will map the whois enrichment to the field URL. Then execute the following command:
/usr/metron/0.1BETA${METRON_HOME}/bin/flatfile_loader.sh -n enrichment_config.json -i whois_ref.csv -t enrichment -c t -e extractor_config.json
After this your enrichment data will be loaded in Hbase and a Zookeeper mapping will be established. The data will be populated into Hbase HBase table called enrichment. To verify that the logs were properly ingested into Hbase HBase run the following command
echo "scan 'enrichment'" | hbase shell
Note, you should also see a separate HBase table, enrichment_list, automatically populated with a single new enrichment type named "whois."
[root@node1(127.0.0.1 192.168.66.121): ~]
# echo "scan 'enrichment_list'" | hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.1.2.2.6.5.1175-1, r897822d4dd5956ca186974c10382e9094683fa29, Thu Jun 20 17:08:24 UTC 2019
scan 'enrichment_list'
ROW COLUMN+CELL
whois column=t:v, timestamp=1566586822992, value={}
1 row(s) in 0.4950 seconds
You should see the table bulk loaded with data from the CSV file. Now check if Zookeeper enrichment tag was properly populated:
/usr/metron/0.1BETA${METRON_HOME}/bin/zk_load_configs.sh -m DUMP -z localhost:2181$ZOOKEEPER -c ENRICHMENT -n squid
This spits out all of the configs to standard out. We provided a sensor name arg, so you should find see one named "squid."
In order to demonstrate the enrichment capabilities of Metron you need to drop all existing indexes for Squid where the data was ingested prior to enrichments being enabled. To do so go back to the head plugin and deleted the indexes like so:
No need to drop index
If you want to start with a fresh index for squid, you can delete the existing index by doing the following:
curl -XDELETE "http://node1:9200/squid*"
ReMake sure you delete all Squid indexes. Re-ingest the data (see previous blog post for more detail)
cat /var/log/squid/access.log | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic squid
and the new messages should be automatically enriched. The Using the ES Head browser plugin, the new message should look as follows:
...