This instruction can be applied both for single and multi-node deployment. It is based on the article: https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html and described how-to install HDP 2.5 with Metron. 

Check it out if anything goes wrong or in troubleshooting section below. There is troubleshooting section in this article as well, make sure to check it first.

IMPORTANT: Some of the values specified in this article need to be substituted with your info, like IP addresses and hostnames. Such values are enclosed with brackers <like that>. Please make sure replace them with your info. 

Prerequisites:

  • Centos 6

  • In order to ease install make sure that your main interface is named eth0. If it is not, you need to adjust some Ambari services configuration accordingly (e.g. ElasticSearch)

  • Single-node: At least 48 GB RAM, 8 cores and 400 GB HDD. Multi-node: At least 32 GB RAM, 4 cores and 200 GB HDD for smooth performance.

  • (optional) Disable PackageKit if it is installed, if it is not, just ignore this point:

    sed -i 's/enabled=1/enabled=0/g' /etc/yum/pluginconf.d/refresh-packagekit.conf

  • Increase limits for ElasticSearch and Storm on nodes where you will be installing them (if you don't know, increase it everywhere):

    echo -e "elasticsearch - memlock unlimited\nstorm - nproc 257597" >> /etc/security/limits.conf

  • Disable IPv6, leaving it enabled may force service to bind to IPv6 addresses only and thus resulting in inability to connect to it (source link):

    sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
echo -e "\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf

  • Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of the kernel line in grub.conf and reboot. (Ambari demands it, do we need to comply?):

    transparent_hugepage=never

    After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):

    # cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]


Metron install pre-preparation

On all nodes Install pre-requisites for Ambari:

yum install epel-release -y
yum update -y
yum install git wget curl rpm scp tar unzip bzip2 wget createrepo reposync yum-utils ntp python-pip -y


On main node where your Ambari will be located install following:

 

pip install --upgrade pip
pip install --upgrade setuptools

 

Install Maven on main node and on Metron node install java 1.8 (if you don't know what it is - install it everywhere):

 

yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel -y

 

Set path to Java 8 if it does not exist:

export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s_/jre/bin/java__")

Save export for future reboots:

echo 'export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s_/jre/bin/java__")' > /etc/profile.d/java_18.sh
chmod +x /etc/profile.d/java_18.sh


Download and install Maven:

wget http://apache.volia.net/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
tar -zxf apache-maven-3.3.9-bin.tar.gz; 
mv apache-maven-3.3.9 /opt/export 
PATH=/opt/apache-maven-3.3.9/bin:$PATH
echo 'export PATH=/opt/apache-maven-3.3.9/bin:$PATH' > /etc/profile.d/maven.sh
chmod +x /etc/profile.d/maven.sh


On Ambari node install and enable docker (we will need it to build Metron mpack for Ambari):

 

yum install docker-io -y
service docker start

 

Also on your build box, install npm (needed to build metron-config, part of the UI):

yum install npm -y

 

Build Metron code:

On main node clone Metron repository:

 

git clone https://github.com/apache/incubator-metron

 

If you install Metron on single node (not multi-node as advised) you need to modify ElasticSearch config templates to use only configuration specified below. Config templates are located in incubator-metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/

cluster.name: metron
network.host: ["_eth0:ipv4_","_local:ipv4_"]
discovery.zen.ping.unicast.hosts: [ <single_node_hostname> ]
path.data: /opt/lmm/es_data
index.number_of_replicas: 0

Fix Kibana install file (it should be fixed in METRON-641:

sed -i 's@{}/kibana@{0}/kibana@g' incubator-metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/kibana_master.py

Build Metron with HDP 2.5 profile:

 

cd incubator-metron
mvn clean install -DskipTests -PHDP-2.5.0.0
cd metron-deployment/packaging/docker/rpm-docker
mvn clean install -DskipTests -PHDP-2.5.0.0

 

On all nodes create localrepo directory and copy RPMs from Ambari node there:

 

mkdir /localrepo
cp -rp /root/incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/* /localrepo/

 

Use scp for remote nodes:

 

scp /localrepo/* <replace_with_node_ip>:/localrepo/

 

If passwordless ssh has not yet been set up within the cluster, then in main node generate key:

 

cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null

 

Add this key to all the slave nodes:

 

ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_node_ip>

 

Ambari 2.4 with HDP 2.5 install

Inspired by: http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-installation/content/ch_Getting_Ready.html

Adjust limits to secure level (inspired by link):

ulimit -n 32768
ulimit -u 65536
echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf


Enable time sync, disable firewall and SElinux:

 

chkconfig ntpd on
service ntpd start
chkconfig iptables off
/etc/init.d/iptables stop
setenforce 0

 

Make sure each node can resolve every other node's hostname or add hostname of each node to /etc/hosts on every node. For example add following lines in /etc/hosts of each node:

10.10.10.1 node1
10.10.10.2 node2
10.10.10.3 node3

Where 10.10.10.1, 10.10.10.2 and 10.10.10.3 are the IPs of your nodes and node1, node2 and node3 are hostnames.

On main node download and setup Ambari repo (you may replace the "2.4.1.0" with a newer Ambari version number):

 

wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.3.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

 

Check that it was added:

 

# yum repolist | grep ambari
Updates-ambari-2.4.1.0 ambari-2.4.1.0 - Updates

 

Install and setup Ambari server:

 

yum install ambari-server -y
ambari-server setup -s

 

Add Metron service to Ambari by running mpack command (make sure to specify correct path to mpack in --mpack=):

 

ambari-server install-mpack --mpack=incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose

 

Start Ambari:

 

ambari-server start

 

Access the Ambari UI by going to the following URL in a web browser (use admin / admin as user / pass):

 

http://<replace_with_master_node_ip>:8080/#/installer/step0

 

Get Started page: Enter any desired cluster name.

Select Version: Make sure "Public Repository" is checked.

Install Options: Specify hostnames of your nodes where Ambari cluster should be installed (all the ones you have specified in /etc/hosts) in Target Hosts. Copy content of the main node private key (/root/.ssh/id_rsa) in "Host Registration Information". If you receive warning like below, ignore it and click OK:

The following hostnames are not valid FQDNs

Choose Services: Select following Services:

 

Metron related

HDFS

YARN + MapReduce2

HBase

Zookeeper

Storm

Flume

Kafka

Elasticsearch

Kibana

Metron

Ambari Metrics

Dependencies

Pig

Tez

Slider

 

Assign Masters: Assign "Kafka Broker" on all nodes. Make sure move following components on one common node:

 

 

Storm UI Server

Metron Indexing

MySQL Server

Kibana Server

Elasticsearch Master

Metron Parsers

Metron Enrichment

Assign Slaves and Clients: select All for:

 

DataNode

NodeManager

RegionServer

Supervisor

Client

 

Customize Services: Following are the list of service that need to be configured:

 

ElasticSearch

Set zen_discovery_ping_unicast_hosts to: <replace_with_elasticsearch_master_hostname> (to the IP of the node where you assigned ElasticSearch Master on the Assign Master tab)

Kibana

Set kibana_es_url to: http://<replace_with_elasticsearch_master_hostname>:9200 (to the IP of the node where you assigned ElasticSearch Master on the Assign Master tab)

Metron

Set Elasticsearch Hosts to: <replace_with_elasticsearch_master_hostname> (to the IP of the node where you assigned ElasticSearch Master on the Assign Master tab)

Change global.json template from (unless it is already fixed as reported in METRON-642):

"es.ip": "{{ es_url }}",

to:

"es.ip": "<replace_with_elasticsearch_master_hostname>",

"es.port": "9300", 

 

Set rest of the configuration values to recommended by Ambari or the one you desire (like DB passwords) and perform install.

After install configuration

Fix ElasticSearch permission (it will crash right after start in Ambari) (unless it is already fixed as reported in METRON-642):

chown -Rh elasticsearch:elasticsearch /etc/elasticsearch

As it will be owned by root by default and will drop error:

Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/scripts

 # ls -la /etc/elasticsearch
...
-rwxr-x---.   1 root          elasticsearch  2571 May 12 09:24 logging.yml
drwxr-x---.   2 root          elasticsearch  4096 May 17 11:49 scripts

Fix path to ES log file in Java parameter (unless it is already fixed as reported in METRON-642)::

sed -i 's@elasticsearchelasticsearch@elasticsearch/elasticsearch@g' /etc/sysconfig/elasticsearch

It is ok if some service will not able to start, check the errors and start them all manually.


TROUBLESHOOTING

Ignore the error Storm UI shown on the screenshot below if you've built your Metron code with HDP-2.5.0.0 profile (in Maven):

It appears because your Kafka Topic was not created or contains no data. Setup streaming and make sure your Kafka topic (from which topology should read data) exists.

 

If you have GUI installed on your server you should run following, before running git clone command:

 

unset SSH_ASKPASS

Or you may receive an error:

(gnome-ssh-askpass:15028): Gtk-WARNING **: cannot open display:

 

If Ambari metrics is not coming up, use this:

 

cd /usr/lib/python2.6/site-packages/resource_monitoring/
python psutil/build.py

 

And re-try.

If you receive an error:

Unsupported major.minor version 52.0

You may need to install Java 1.8 on the node you receive this from:

 

yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel -y

 

 

 

  • No labels