...
We will be installing Metron 0.4.0 with HDP 2.5 on CentOS 6. We will also install MariaDB as a database for Metron REST. Additionally, we'll also install Apache NiFi.
I installed Metron in a test environment with 3 VMs to try it out as well as a single node. I'll try to write this guide so that the necessary steps can easily be adapted for other environments.
Environment
- Single node: 4 CPUs, 16GB RAM.
- Multiple nodes:
- 3 VMs, 2 CPUs per VM and 8 GB RAM per VM.
- Hosts:
10.10.10.1 node1
10.10.10.2 node2
10.10.10.3 node3
Prerequisites:
- CentOS 6
Add the epel repository and install tmux, vim & htop. Installing these utilities is not strictly necessary, but I install these by default for potential troubleshooting & editing of files locally):
Code Block # yum install epel-release -y
...
# yum update -y
...
# yum install vim tmux htop -y
- Set up passwordless SSH between our nodes:
- If passwordless ssh has not yet been set up within the cluster, then in main node generate key:
| Code Block |
|---|
# cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null |
...
# cd ~/.ssh |
...
# cat id_rsa.pub >> authorized_keys |
If you're not installing on a single node, add this newly generated key to all the slave nodes:
Code Block # ssh-copy-id -i ~/.ssh/id_rsa.pub
...
<REPLACE_
...
WITH_
...
NODE_
...
IP>Side note: You might have to adapt your sshd_config file and add "PermitRootLogin yes" amongst other parameters if you want passwordless root access, but that's outside the scope of this document.
- Increase limits for ElasticSearch and Storm on nodes where you will be installing them (if you don't know, increase it everywhere):
# echo -e "elasticsearch - memlock unlimited\nstorm - nproc 257597" >> /etc/security/limits.conf
- Adjust limits to secure level ([link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):
...
- Download and install Maven 3.3.9:
# wget http://apache.volia.net/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
# tar -zxf apache-maven-3.3.9-bin.tar.gz
# mv apache-maven-3.3.9 /opt
# PATH=/opt/apache-maven-3.3.9/bin:$PATH
# echo 'export PATH=/opt/apache-maven-3.3.9/bin:$PATH' > /etc/profile.d/maven.sh
# chmod +x /etc/profile.d/maven.sh
...
supervisor.slots.ports: [6700, 6701, 6702, 6703, 6704, 6705]
- Install Apache NiFi in /root (You can pretty much use any directory you want). Download nifi-1.2.0-bin.tar.gz from https://nifi.apache.org/download.html
...