Contributed by Laurens Vets <laurens@daemon.be>. Version 0.3.3 4 - July 2017.
Introduction
We will be installing Metron 0.4.0 with HDP 2.5 on CentOS 6. We will also install MariaDB as a database for Metron REST. Additionally, we'll also install Apache NiFi.
I installed Metron in a test environment with 3 VMs to try it out as well as a single node. I'll try to write this guide so that the necessary steps can easily be adapted for other environments.
Environment
- Single node: 4 CPUs, 16GB RAM.
- Multiple nodes:
- 3 VMs, 2 CPUs per VM and 8 GB RAM per VM.
- Hosts:
10.10.10.1 node1
10.10.10.2 node2
10.10.10.3 node3
Prerequisites:
- CentOS 6
Add the epel repository and install tmux, vim & htop. Installing these utilities is not strictly necessary, but I install these by default for potential troubleshooting & editing of files locally):
Code Block # yum install epel-release -y
...
# yum update -y
...
# yum install vim tmux htop -y
- Set up passwordless SSH between our nodes:
- If passwordless ssh has not yet been set up within the cluster, then in main node generate key:
| Code Block |
|---|
# cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null |
...
# cd ~/.ssh |
...
# cat id_rsa.pub >> authorized_keys |
If you're not installing on a single node, add this newly generated key to all the slave nodes:
Code Block # ssh-copy-id -i ~/.ssh/id_rsa.pub
...
<REPLACE_
...
WITH_
...
NODE_
...
IP>Side note: You might have to adapt your sshd_config file and add "PermitRootLogin yes" amongst other parameters if you want passwordless root access, but that's outside the scope of this document.
- Increase limits for ElasticSearch and Storm on nodes where you will be installing them (if you don't know, increase it everywhere):
# echo -e "elasticsearch - memlock unlimited\nstorm - nproc 257597" >> /etc/security/limits.conf
- Adjust limits to secure level ([link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):
...
Add "transparent_hugepage=never" in the kernel line after "quiet:
"kernel /vmlinuz-2.6.32-696.3.1.el6.x86_64 ro root=/dev/mapper/vg_centos6-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_centos6/lv_swap rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_centos6/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet"
becomes:
"kernel /vmlinuz-2.6.32-696.3.1.el6.x86_64 ro root=/dev/mapper/vg_centos6-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_centos6/lv_swap rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_centos6/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet transparent_hugepage=never"
Afterwards, run:
# grub-install /dev/sda rhgb quiet transparent_hugepage=never"
Afterwards, run:
# grub-install /dev/sda
- If you do not want to mess with grub/kernel parameters, add the following to /etc/rc.local:
vim /etc/rc.local:
# Disable THP at boot time
if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
fi
After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):
...
# yum install git wget curl rpm tar unzip bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel npm -y
# pip install --upgrade pip
# pip install requests urllib
# pip install --upgrade setuptools
...
- Download and install Maven 3.3.9:
# wget http://apache.volia.net/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
# tar -zxf apache-maven-3.3.9-bin.tar.gz
# mv apache-maven-3.3.9 /opt
# PATH=/opt/apache-maven-3.3.9/bin:$PATH
# echo 'export PATH=/opt/apache-maven-3.3.9/bin:$PATH' > /etc/profile.d/maven.sh
# chmod +x /etc/profile.d/maven.sh
...
- Remove ipv4 'localhost.localdomain' from /etc/hosts
- Remove ipv6 'localhost.localdomain' from /etc/hosts
- Add "127.0.0.1 localhost" to /etc/hosts
- - Install the database we will use for Metron REST:
# yum install mariadbmysql-server mysql-connector-java -y
...
If you're doing a multi node install, also create localrepo on the nodes and copy the packages to the other nodes:
# ssh root@node2 mkdir /localrepo
# scp /localrepo/* <replace_with_node_ip>\.rpm root@node2:/localrepo/.
# ssh root@node2 yum install createrepo -y
# ssh root@node2 createrepo /localrepo
Make sure to run `createrepo /localrepo` on every node!do the above on each node.
- Fetch & create logrotate script for Hadoop Services:
...
- Enable time sync, disable firewall and SElinux on every node:
# yum install ntp -y
# service ntpd start
# /sbin/chkconfig --add ntpd
# /sbin/chkconfig --list ntpd
# /sbin/chkconfig ntpd on
# /sbin/chkconfig --list ntpd
- Disable firewall on every node:
# service iptables save
# service iptables stop
# chkconfig iptables off
- Disable IPv6 firewall on every node:
# service ip6tables save
# service ip6tables stop
# chkconfig ip6tables off
- Disable SElinux on every node:
# setenforce 0 (=> I know, but for the sake of simplicity, quickness & testing, I've disabled selinux.)
...
- Install everything. Metron REST will probably not work as we still need to add a user and the database to MySQL.
At this point, make sure that all the services are up. You might have to manually start a few.
- Configure a user for Metron REST in MySQL. On the node where you installed the Metron REST UI, do:
...
# mysql -u metron -p
> use metronrest;
> insert into users (username, password, enabled) values ('metron','metron',1);
> insert into authorities (username, authority) values ('metron', 'ROLE_USER');
> quit
Bye
#
Make sure that all the services are up.
- Install metron_pcapservice:
...
# yum install centos-release-scl -y
# yum update -y
# yum install python27 -y
# scl enable python27 bash
# cd /opt/rh/python27/root/usr/bin/
# LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install --upgrade pip
# LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install requests
(# /opt/rh/python27/root/usr/bin/virtualenv py27venv
# source py27venv/bin/activate
# pip install --upgrade pip
# pip install ansible==2.0.0.2
# ansible --version
# deactivate)
# .7 install requests
# yum install @Development python-virtualenv libpcap-devel libselinux-python -y
# mkdir /usr/local/pycapa
# cd /usr/local/pycapa
# virtualenv pycapa-venv
# source pycapa-venv/bin/activate
# cp -r /root/metron/metron-sensors/pycapa/. /usr/local/pycapa/.
# pip install --upgrade pip
# /usr/local/pycapa/pycapa-venv/bin/pip install -r requirements.txt
(
# pip install -r requirements.txt)
# /usr/local/pycapa/pycapa-venv/bin/python setup.py install
# ln -s /usr/local/lib/librdkafka.so.1 /opt/rh/python27/root/usr/lib64
# deactivate
...
# wget -O /etc/init.d/pycapa https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/pycapa/templates/pycapa
# sed -i 's/{{ pycapa_log }}/\/var\/log\/pycapa.log/' /etc/init.d/pycapa
# sed -i 's/{{ pycapa_home }}/\/usr\/local\/pycapa/' /etc/init.d/pycapa
# sed -i 's/{{ python27_home }}/\/opt\/rh\/python27\/root/' /etc/init.d/pycapa
# sed -i 's/{{ pycapa_bin }}/\/usr\/local\/pycapa\/pycapa-venv\/bin/' /etc/init.d/pycapa
# sed -i 's/--kafka {{ kafka_broker_url }}/--kafka-broker <IP:6667>/' /etc/init.d/pycapa
# sed -i 's/--topic {{ pycapa_topic }}/--kafka-topic pcap/' /etc/init.d/pycapa
# sed -i 's/{{ pycapa_sniff_interface }}/tap0/' /etc/init.d/pycapa
(# sed -i 's/export LD_LIBRARY_PATH=\/opt\/rh\/python27\/root\/usr\/lib64/export LD_LIBRARY_PATH=\/usr\/local\/lib/' /etc/init.d/pycapa)
# chmod 755 /etc/init.d/pycapa
# yum install @Development libdnet-devel rpm-build libpcap libpcap-devel pcre pcre-devel zlib zlib-devel glib2-devel -y
# yum install kafka -y
...
supervisor.slots.ports: [6700, 6701, 6702, 6703, 6704, 6705]
- Install Apache NiFi in /root (You can pretty much use any directory you want). Download nifi-1.2.0-bin.tar.gz from https://nifi.apache.org/download.html
# cd /root
# wget http://apache.mirror.iweb.ca/nifi/1.2.0/nifi-1.2.0-bin.tar.gz
# tar xf nifi-1.2.0-bin.tar.gz
...