The instructions outlined below are for an older version of Metron. For the most recent documentation on spinning up a development instance, refer to:
The following provides instructions on how to install Metron on an existing Ambari-managed HDP 2.4 cluster. These are not recommendations for production deployment but rather for dev/testing environments.
For the purposes of this discussion, there will be 7 types of server roles:
- Hadoop Master
- Hadoop Slave
- Ambari Server
- Metron Installer
- Metron Search
- Metron Probe
- Metron Web
Specifications for Existing Hadoop Cluster
The following are the specifications for the Hadoop related nodes:
- All cluster nodes must be running CentOS 6.x.
- The cluster must be running HDP 2.4 managed by Ambari 2.2.x.
The cluster must have a minimum of 2 Hadoop Master nodes and 4 Hadoop slave nodes.
- The minimum spec for each of the Hadoop Slave and Master nodes should be (for Dev/Test/POC envs):
- 8-16 GB RAM
- 4 cores
- 100 GB of storage
- The following services must be installed across the Hadoop Master and Slave nodes:
Each of the following components must be installed on at least 4 Hadoop Slave Nodes:
- No other workloads should be running on the cluster. In other words, this cluster should be dedicated for Metron jobs.
Specifications for Metron Nodes
- At least three nodes must be dedicated for Metron-specific components.
- You should have root access on all Metron nodes.
All Metron nodes (Search, Probes, and Web) must have the following:
- Centos 6.7
- Oracle JDK 1.8
- The minimum spec for each of the Metron nodes should be (for Dev/Test/POC envs):
- 8-16 GB RAM
- 4 cores
- 100 GB of storage
The Metron Installer node must have Docker installed. (Note: If you are using Centos, 7.x is preferred for the Metron Installer node.)
The Metron Installer node requires SSH access to all Hadoop Cluster and Metron nodes. Passwordless is preferred.
Sample Deployment Architecture
Based on the above specs, here is a sample deployment architecture.
Step 1: Setup
For the following instructions, set the following environmental variables. Plug in the necessary values for your deployment.
Step 2: Install or Validate Existing Hadoop Cluster
- Install a new HDP 2.4 using the following instructions or ensure you have an existing HDP cluster that follows the specs outlined in the pre-requisites section.
Step 3: Download and Build Metron
- Log into the Metron Installer node.
- Verify Docker is installed and running.
- Ensure that this node has ssh access to each of the Metron and Hadoop/HDP nodes.
- Download the Metron Binaries from $METRON_0_2_RELEASE_BINARIES
- Untar it: tar -zxvf apache-metron-0.2.0BETA-RC2-incubating.tar.gz
- cd $METRON_HOME
- Run Docker installer image:
docker run -it -v $METRON_HOME:/root/incubator-metron dlyle65535/metron-ansible:126.96.36.199 bash
- Build Metron
mvn clean package -DskipTests
Step 4: Configure Metron for Install
- cd /root/incubator-metron/metron-deployment/inventory
- create a directory called $INSTALL_CONF_DIRECTORY that will hold all of your metron deployment configuration
- Copy the sample configs into the new directory:
- cp - R /root/incubator-metron/metron-deployment/inventory/metron_example/* /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY
- Setup your inventory hosts by editing the file $METRON_HOME/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY/hosts
Configure with your cluster details by defining which hosts the Metron services will be installed on by updating these groups:
[ambari_master] - host running Ambari
[ambari_slaves] - all Ambari-managed hosts
[metron_hbase_tables] - host used to create the HBase tables required by Metron. Requires a HBase client.
[metron_kafka_topics] - host used to create the Kafka topics required by Metron. Requires a Kafka broker.
[enrichment] - submits the topology code to Storm and requires a Storm client
[search] - host(s) where Elasticsearch will be installed
[web] - host where the Metron UI and underlying services will be installed
[sensors] - host where network data will be collected and published to Kafka
- Setup your environment configuration by editing the file $METRON_HOME/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY/group_vars/all
Commonly Used Settings:
- ansible_ssh_private_key_file: /Path/to/private/key/file Point to the private key file for ssh user on the target hosts
- ansible_ssh_user: root The name of the ssh user on the target hosts (requires sudo)
- ambari_port: 8080 Change if your Ambari instance uses a non-default port
- ambari_user: admin Change to user on your Ambari instance
- ambari_password: admin Change to password for your Ambari user above
- num_partitions: 3 Change to your desired number of partitions
- retention_in_gb: 25 Change to your desired retention size
- java_home: /usr/jdk64/jdk1.8.0_40 Location of Java on all hosts
- sensor_test_mode: True Change to false if not running traffic replay
- sniff_interface: eth0 Interface that the Metron sensors will sniff on the [sensors] host
- elasticsearch_network_interface: eth0 Bind interface for the Elasticsearch host(s)
- Ensure that you do't have duplicate entries of sniff_interface
Step 5: Install/Deploy Metron to the Cluster
Note: Adding -u user -k to the below command lines allows deployment in environments where passwordless SSH has not been enabled.
- cd /root/incubator-metron/metron-deployment/playbooks
- Run the following ansible command to deploy Metron with the standard services (Metron Topologies, WebUI, and Monit)
ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr"
- Run the following Ansible command to deploy Metron without installing Sensors
ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr,sensors"