You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 58 Next »



Short summary:

This document details the steps involved in Installing latest version of Apache Incubator Ranger independently on RHEL / Ubuntu / SUSE / Debian OS.

Prerequisites


A. JDK 7 or above needs to be installed. 

B. Install latest version of Database MySQL/ ORACLE/ Postgres/ SQL Server

Instructions

Preparing to install

Install Maven

cd /usr/local 
# Download maven latest distribution tar from apache maven site
tar -­xvf apache-­maven-­<Version>-­bin.tar.gz

export M2_HOME=/usr/local/apache-­maven-­<Version>
export M2=$M2_HOME/bin 
export PATH=$M2:$PATH 
 
#Now to test your install of Maven, enter the following command
mvn -­version 

Install git

yum install git 

Install gcc and export JAVA_HOME

yum search gcc # Select the appropriate gcc and do 'yum install gcc' 
 
export JAVA_HOME=<Java Installation Directory>

Build Ranger Admin from source 

Clone the ranger source code

mkdir ~/dev 
cd ~/dev 
git clone https://github.com/apache/incubator­-ranger.git

Build the source 

cd incubator­-ranger 
export MAVEN_OPTS="­-Xmx512M" 
mvn clean compile package assembly:assembly install
  • Verify all the tar files under target dir

Install Steps for Ranger Policy Admin on  RHEL

a. Lay down the build into appropriate places. Let’s start with the Ranger web admin first.

cd /usr/local 
sudo tar zxvf ~/dev/incubator­-ranger/target/ranger-­0.5.0-­admin.tar.gz
sudo ln ­-s ranger-­0.5.0-­admin ranger-­admin 
cd /usr/local/ranger-­admin 

b. Verify the root password that you had picked while installing mysql. I had chosen root so the relevant section in my install.properties file looks as follows:

db_root_user=root 
db_root_password=root
db_host=localhost

c.The install process would create a couple of users in the database for storing administration and audit information, pick passwords for those too. With my choices here’s how the relevant sections in the install.properties file look now.

  # DB UserId used for the XASecure schema 
  #
  db_name=ranger 
  db_user=rangeradmin 
  db_password=rangeradmin 
  # DB UserId for storing auditlog infromation 
  # 
  audit_db_name=ranger 
  audit_db_user=rangerlogger 
  audit_db_password=rangerlogger

d. Ranger allows you to get fancier with security and authentication mode but for now let’s just leave rest of the things in this file as they are.

e. Once all the required properties are updated run the setup.sh script

./setup.sh

f. This will install the Ranger service.  

g. After this Start Ranger Service by typing.  

./ews/ranger-­admin-­services.sh start 

h. After this Restart the Ranger Admin should work.

You can stop the Ranger service by typing : ./ews/ranger-­admin-­services.sh stop 

To stop and start you can try this ./ews/ranger­-admin­-services.sh restart

i.You can verify by visiting the external URL of the server using browser, for example: 

http://<Host Address>:6080/  


  • Configuring Ranger Admin Authentication Modes :
  •  ACTIVE DIRECTORY

      To enable active directory authentication on Ranger admin, you need to configure following properties of install.properties

PROPERYVALUE
authentication_methodACTIVE_DIRECTORY

xa_ldap_ad_domain

A sample value would be : “example.com

xa_ldap_ad_url

A sample value would be “ldap://127.0.0.1:389
Ldap server URL.

xa_ldap_ad_base_dn

A sample value would be : “DC=example,DC=com”
The Distinguished Name (DN) of the starting point for directory server searches.

xa_ldap_ad_bind_dn

A sample value would be :
“CN=Administrator,CN=Users,DC=example,DC=com”
Full distinguished name (DN), including common name (CN), of an Active Directory user
account that has privileges to search for users. This user account must have at least
domain user privileges.

xa_ldap_ad_bind_password

Password for the account that can search for users.

xa_ldap_ad_referral

Possible values are “ignore ”, “follow ” and “throw ”. default value is “follow”.

       When searching a directory, the server might return several search results, in addition to
       a few continuation references that show where to obtain further results. These results
       and references might be interleaved at the protocol level. When property is set to
       "follow", the AD service provider processes all the normal entries first, before following
       the continuation references. When this property is set to "throw", all of normal entries are
       returned in the enumeration first, before the ReferralException is thrown. By contrast, a
      "referral" error response is processed immediately when property is set to "follow" or
      "throw".

  • LDAP
PROPERTYVALUE
authentication_methodLDAP
xa_ldap_url

A sample value would be : "ldap://127.0.0.1:389" Ldap server URL.

xa_ldap_userDNpatternA sample value would be : "uid={0},ou=users,dc=example,dc=com"
User DN pattern is expanded when a user is being logged in. For example, if the user
‘ldapadmin’ attempted to log in, the LDAP Server would attempt to bind against the DN
‘uid=ldapadmin,ou=users,dc=example,dc=com’ using the password the user provided.
 xa_ldap_groupSearchBase

A sample value would be : "dc=example,dc=com"
Defines the part of the directory tree under which group searches should be performed.

xa_ldap_groupSearchFilter  A sample value would be: "(member=cn={0},ou=users,dc=example,dc=com)" The filter which is used to search for group membership. The default is uniqueMember={0}, corresponding to the groupOfUniqueNames LDAP class. In case of Ranger authentication, the substituted parameter is the full distinguished name of the user. The parameter {0} can be used if you want to filter on the login name.
 xa_ldap_groupRoleAttribute A sample value would be: "cn" The attribute which contains the name of the authority defined by the group entry.
 xa_ldap_base_dn A sample value would be : " dc=example,dc=com" The Distinguished Name (DN) of the starting point for directory server searches.
 xa_ldap_bind_dn A sample value would be : "cn=ldapadmin,ou=users,dc=example,dc=com" Full distinguished name (DN), including common name (CN), of an LDAP user account that has privileges to search for users.
xa_ldap_bind_password  Password for the account that can search for users.
 xa_ldap_referral

default value is “follow” Possible values are “ignore ”, “follow ” and “throw ”. default value is “follow”. When searching a directory, the server might return several search results, in addition to a few continuation references that show where to obtain further results. These results and references might be interleaved at the protocol level. When property is setto "follow", the AD service provider processes all the normal entries first, before following the continuation references. When this property is set to "throw", all of normal entries are returned in the enumeration first, before the ReferralException is thrown. By contrast, a "referral" error response is processed immediately when property is set to "follow" or
"throw".

 
  • UNIX
PROPERTYVALUE
authentication_methodUNIX 
remoteLoginEnabledtrue
authServiceHostName

A sample value would be : localhost An Address of host where unixauth service is running

authentication_method5151
port number on which unixauth service is running. default is 5151
  • Configuring Ranger Admin HA Mode 

               a. Follow the ranger admin install steps above to install it on multiple hosts 

               b. Make sure to use the same configuration and policy DB details 

               c. Configure a load balancer to load balance among ranger admin instances and note down the load balancer URL.

        •  Software (e.g. Apache httpd) or hardware load balancer could be used.
        •  Details outside the scope of this document.                              

                d. Update the policy manager external URL in all the clients of ranger admin (ranger user sync and ranger plugins) to point to the load balancer URL.

                e. Restart all the clients.

Installing the Ranger User-Synch Process

 a. We’ll start by extracting out build at the appropriate place. 

cd /usr/local 
sudo tar ­zxvf ~/dev/ incubator­-ranger/target/ranger-­0.5.0-­usersync.tar.gz
sudo ln ­-s ranger-­0.5.0-­usersync ranger-­usersync
sudo mkdir ­-p /var/log/ranger-­usersync 
sudo chown ranger /var/log/ranger­-usersync 
sudo chgrp ranger /var/log/ranger­-usersync 
cd ranger­usersyncb. 
  • Now let’s edit the install.properties file. Here are the relevant lines that you should edit:

  • POLICY_MGR_URL = http://localhost:6080
  • SYNC_SOURCE = unix  
  • logdir = /var/log/ranger/usersync 

c. Now install the usersync by running the setup command 

export JAVA_HOME=/usr/lib/jvm/java­-1.7.0­-openjdk-­amd64 
./setup.sh 

After installing ranger­usersync, follow the same steps to start/stop services of usersync work.
./ranger­-usersync­-services.sh start 

Configuring Ranger User-Sync process to use LDAP/AD server:

      To Sync LDAP or AD users following properties should be configured in install.properties file before executing the setup.sh.

SYNC_SOURCE

sync source, “ldap” should be used for both LDAP or AD user
syncing.

SYNC_LDAP_URL


URL of source ldap server. Must specify a value if SYNC_SOURCE is ldap

SYNC_LDAP_BIND_DN

Ldap bind dn used to connect to ldap and query for users and groups. Must specify a value if SYNC_SOURCE is ldap.

SYNC_LDAP_BIND_PASSWORD

Ldap bind password for the bind dn specified above. Please ensure read access to this file is limited to root, to protect the password
Must specify a value if SYNC_SOURCE is ldap unless anonymous search is allowed by the directory on users and group.

SYNC_LDAP_SEARCH_BASEsearch base for users and groups 
SYNC_LDAP_USER_SEARCH_BASE

search base for users,overrides value specified in SYNC_LDAP_SEARCH_BASE. Must specify a value if SYNC_SOURCE is ldap and
SYNC_LDAP_SEARCH_BASE is empty.

SYNC_LDAP_USER_SEARCH_SCOPE

search scope for the users, only base, one and sub are supported values.

NEED TO BE CUSTOMIZEPROPERTYDESCRIPTION
#Please customize the value to suit your deployment. SYNC_LDAP_USER_OBJECT_CLASSobject class to identify user entries.  
#Please customize the value to suit your deployment. SYNC_LDAP_USER_SEARCH_FILTER

optional additional filter constraining the users selected for syncing.

# please customize the value to suit your deployment. SYNC_LDAP_USER_NAME_ATTRIBUTE

attribute from user entry that would be treated as user name

# please customize the value to suit your deployment. SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE

attribute from user entry whose values would be treated as group values to be pushed into Policy Manager database. You could provide multiple attribute names separated by comma.

# possible values:  none, lower, upper SYNC_LDAP_USERNAME_CASE_CONVERSION

UserSync ­ Case Conversion Flags.

# possible values:  none, lower, upper SYNC_LDAP_GROUPNAME_CASE_CONVERSION

UserSync ­ Case Conversion
Flags.

# any value other than true would be treated as falseSYNC_GROUP_SEARCH_ENABLED

do we want to do ldapsearch to find groups
instead of relying on user entry attributes. valid values: true, false

 SYNC_GROUP_USER_MAP_SYNC_ENABLED

do we want to do ldapsearch to find groups instead of relying on user entry attributes and sync memberships of those groups valid values: true, false. any value other than true would be treated as false.

 SYNC_GROUP_SEARCH_BASE

search base for groups. overrides value specified in SYNC_LDAP_SEARCH_BASE,SYNC_LDAP_USER_SEARCH_BASE.
if a value is not specified, takes the value of SYNC_LDAP_SEARCH_BASE
if SYNC_LDAP_SEARCH_BASE is also not specified, takes the value of
SYNC_LDAP_USER_SEARCH_BASE

# any value other than true would be treated as falseSYNC_GROUP_SEARCH_SCOPE

search scope for the groups, only base, one and sub are supported values

# any value other than true would be treated as falseSYNC_GROUP_OBJECT_CLASS

object class to identify group entries. default value: groupofnames

please customize the value to suit your deployment. SYNC_LDAP_GROUP_SEARCH_FILTER

optional additional filter constraining the groups selected for syncing. default value is empty.

# please customize the value to suit your deployment.  SYNC_GROUP_NAME_ATTRIBUTE 

attribute from group entry that would be treated as group name.

 SYNC_GROUP_MEMBER_ATTRIBUTE_NAME

attribute from group entry that is list of members. default value: member.

 SYNC_PAGED_RESULTS_SIZE

page size for paged results control.search results would be returned page by page with the specified number of entries per page default value: 500

# please customize the value to suit your deployment. SYNC_PAGED_RESULTS_ENABLED

do we want to use paged results control during
ldapsearch for user entries. valid values: true, false. default value: true
any value other than true would be treated as false. if the value is false, typical AD would not return more than 1000 entries.

 

  Table: LDAP/AD Properties with sample values 

PROPERTIES NAMESAMPLE VALUES FOR LDAP USER SYNCHSAMPLE VALUES FOR AD USERSYNCH
SYNC_LDAP_URL ldap://127.0.0.1:389 ldap://127.0.0.1:389 
SYNC_LDAP_BIND_DN 

cn=ldapadmin,ou=users,dc=example,dc=com

cn=adadmin,cn=Users,dc=exa
mple,dc=com

SYNC_LDAP_BIND_PASSWORD secret secret 
SYNC_LDAP_SEARCH_BASE dc=example,dc=com dc=example,dc=com 
SYNC_LDAP_USER_SEARCH_BASEou=users,dc=example,dc=com dc=example,dc=com 
SYNC_LDAP_USER_SEARCH_SCOPE sub sub 
SYNC_LDAP_USER_OBJECT_CLASSperson person 
SYNC_LDAP_USER_SEARCH_FILTER (objectcategory=person) 
SYNC_LDAP_USER_NAME_ATTRIBUTEuid or cn sAMAccountName 

SYNC_LDAP_USER_GROUP_NAME_AT
TRIBUTE

memberof,ismemberof memberof,ismemberof 

SYNC_LDAP_USERNAME_CASE_CONV
ERSION

lower lower 

SYNC_LDAP_GROUPNAME_CASE_CON
VERSION

lower lower 
SYNC_GROUP_SEARCH_ENABLED false false 

SYNC_GROUP_USER_MAP_SYNC_ENA
BLED

false false 
SYNC_GROUP_SEARCH_BASE ou=groups,dc=example,dc=com dc=example,dc=com 
SYNC_GROUP_SEARCH_SCOPE subsub
SYNC_GROUP_OBJECT_CLASS groupofnames groupofnames 
SYNC_LDAP_GROUP_SEARCH_FILTER  
SYNC_GROUP_NAME_ATTRIBUTE cncn

SYNC_GROUP_MEMBER_ATTRIBUTE_
NAME

member member 
SYNC_PAGED_RESULTS_ENABLED true true 
SYNC_PAGED_RESULTS_SIZE 500500

Installing Apache  Hadoop

    • Now let’s download and install hadoop. Following the excellent instructions available on the hadoop site itself. Follow steps given in pseudo distributed mode.

    • These instructions were written for version 2.7.0. So grab that tar (hadoop-­2.7.0.tar.gz) and checksum file (hadoop-­2.7.0.tar.gz.mds).

    • Instructions on this page ask that java be installed. If java is not there, install JDK first.
sudo yum install java-­1.7.0-­openjdk­-devel
    • Make note of the location where you installed hadoop. Here I assume that you have installed it in
/usr/local/hadoop
    • Create a user under which we could install and ultimately run the various hadoop processes. And login as that user.
sudo useradd ­­--home-­dir /var/hadoop --­­create-­home --­­shell /bin/bash ­­--user-­group hadoop
    •  if you get below given message then try next command  
sudo useradd --­­home-­dir /var/hadoop ­­--create-­home --­­shell /bin/bash hadoop -­g hadoop
sudo tar ­zxvf ~/dev/hadoop-­2.7.0.tar.gz -­C /usr/local
cd /usr/local 
sudo ln ­-s hadoop­-2.7.0 hadoop 
sudo chown hadoop -­R hadoop hadoop-­2.7.0
sudo chgrp hadoop ­-R hadoop hadoop­-2.7.0 
TO ADD HDFS USER
useradd hdfs 
to check whether user hadoop login works, try: ­ -sudo su ­-hadoop


Enabling Ranger HDFS Plugins

a. We’ll start by extracting our build at the appropriate place (/usr/local).

cd /usr/local
sudo tar zxvf ~/dev/incubator-­ranger/target/ranger-­0.5.0-­hdfs­-plugin.tar.gz 
sudo ln -­s ranger-­0.5.0-­hdfs-­plugin ranger-­hdfs-­plugin
cd ranger-­hdfs-­plugin 

b. Now let’s edit the install.properties file. Here are the relevant lines that you should edit:

    • Change the install.properties file 
PROPERTYVALUE
POLICY_MGR_URL


http://localhost:6080

REPOSITORY_NAMEhadoopdev 
XAAUDIT.DB.IS_ENABLEDtrue
XAAUDIT.DB.FLAVOURMYSQL
XAAUDIT.DB.HOSTNAMElocalhost 
XAAUDIT.DB.DATABASE_NAMEranger_audit 
XAAUDIT.DB.USER_NAMErangerlogger 
XAAUDIT.DB.PASSWORDrangerlogger 

c. Now enable the hdfs­-plugin by running the enable-­hdfs-­plugin.sh command (Remember to set JAVA_HOME)

Note: Hadoop conf and hadoop lib folder are not found at expected locations as per the script because of which Ranger hdfs plugin installation fails. To resolve this issue create a symlink as conf dir of hadoop linking to hadoop conf dir
cd /usr/local/hadoop
ln -­s /usr/local/hadoop/etc/hadoop/ /usr/local/hadoop/conf 
  • Export HADOOP_HOME to bashrc
echo "export HADOOP_HOME=/usr/local/hadoop" >> /etc/bashrc 
cd /usr/local/ranger-­hdfs-­plugin 
./ enable-­hdfs-­plugin.sh
  • One more change that we need to do is copy all the jar files from ${hadoop_home}/lib
cp /usr/local/hadoop/lib/*.jar /usr/local/hadoop/share/hadoop/hdfs/lib/
  • Provide required permission to logs directory 
   chown root:hadoop /usr/local/hadoop/logs 
   chmod g+w /usr/local/hadoop/logs 
  • Provide required permission to users in OS file system and hdfs file system according to your environment and requirement.

d.  Once these changes are done Restart hadoop.

  • Stop NameNode, SecondaryNameNode and DataNode daemon: 
  su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop­-daemon.sh stop namenode"
  su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop-­daemon.sh stop secondarynamenode"
  su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop-­daemon.sh stop datanode"
  • Start NameNode, SecondaryNameNode and DataNode daemon: 
  su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop­-daemon.sh start namenode"
  su -­l hdfs -­c "/usr/local/hadoop/sbin/hadoop­-daemon.sh start secondarynamenode"
  su ­-l hdfs ­-c "/usr/local/hadoop/sbin/hadoop­-daemon.sh start secondarynamenode"

e. This should start the association of ranger­hdfs­plugin with hadoop. 

  • You can verify by logging into the Ranger Admin Web interface ­> Audit > Agents.

 

Installing Apache Hive(1.2.0)

sudo tar xzvf ~/dev/apache-­hive-­1.2.0-­bin.tar.gz ­-C /usr/local
cd /usr/local 
sudo ln ­-s apache-­hive-­1.2.0-­bin hive
useradd hive
cd hive 
 
Export HIVE_HOME to bashrc 
echo "export HIVE_HOME=/usr/local/hive" >> /etc/bashrc  
Note:HiveServer2 doesn’t start unless HADOOP_VERSION is exported to bashrc

Enabling Ranger Hive Plugin

  • We’ll start by extracting our build at the appropriate place. 
cd /usr/local 
sudo tar zxvf ~/dev/incubator­-ranger/target/ranger-­0.5.0-­hive­-plugin.tar.gz 
sudo ln -­s ranger-­0.5.0-­hive-­plugin ranger-­hive-­plugin
cd ranger­-hive-­plugin  
  • Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
  • Change the insall.properties file 
PROPERTYVALUE
POLICY_MGR_URL


http://localhost:6080

REPOSITORY_NAMEhivedev 
XAAUDIT.DB.IS_ENABLED true
XAAUDIT.DB.FLAVOUR=MYSQL MYSQL
XAAUDIT.DB.HOSTNAMElocalhost
XAAUDIT.DB.DATABASE_NAMEranger_audit
XAAUDIT.DB.USER_NAME rangerlogger
XAAUDIT.DB.PASSWORDrangerlogger
  •  Now enable the hive­-plugin by running the enable-­hive­-plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger­-hive-­plugin
./enable­-hive-­plugin.sh 
  • Once these changes are done Restart hive. This should start the association of ranger-­hive-­plugin with hive.
    1. You can verify by logging into the Ranger Admin Web interface ­> Audit Tab ­> Agents
  • Provide required permission to users in OS file system and hdfs file system according to your environment and requirement..

NOTES: If /var/log/hive directory does not exist then create one and assign to user hive. 

mkdir /var/log/hive
chown -­R hive: hive /var/log/hive

 

  • Change properties file permission for hive user.
chown -­R hive:hadoop /usr/local/apache-­hive-­1.2.0-­bin/conf/hiveserver2-­site.xml 
chown -­R hive:hadoop /usr/local/apache-­hive­-1.2.0­-bin/conf/hive-­log4j.properties
chown -­R hive:hadoop /usr/local/apache­-hive­-1.2.0­-bin/conf/hive­-site.xml
  • To start hive metastore :
su ­-l hive -­c "env HADOOP_HOME=/usr/local/hadoop JAVA_HOME=/usr/lib/jvm/java-­1.7.0-­openjdk.x86_64 nohup hive --­­service metastore > /var/log/hive/hive.out 2> /var/log/hive/hive.log &”
  • To start Hive server2 :
su -­l hive -­c "env HADOOP_HOME=/usr/local/hadoop JAVA_HOME=/usr/lib/jvm/java­-1.7.0-­openjdk.x86_64 nohup /usr/local/hive/bin/hiveserver2 ­hiveconf hive.metastore.uris=\" \" > /var/log/hive/hiveServer2.out 2>/var/log/hive/hiveServer2.log &”

  • To Stop: 
ps aux | awk '{print $1,$2}' | grep hive | awk '{print $2}' | xargs kill >/dev/null 2>&1
  • To Login in Hive shell: 
/usr/local/hive/bin/beeline ­-u "jdbc:hive2://localhost:10000" -­n rituser ­-p rituser
  • If hive metastore and hiveserver2 do not start then update below given key-­values according to your environment in following files.

hiveserver2­site.xml  

<configuration>
<property>
<name>hive.security.authorization.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.security.authorization.manager</name>

<value>org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer
Factory</value>
</property>
<property>
<name>hive.security.authenticator.manager</name>

<value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</v
alue>

</property>
<property>
<name>hive.conf.restricted.list</name>

<value>hive.security.authorization.enabled,hive.security.authorization.manage
r,hive.security.authenticator.manager</value>
</property>
</configuration>


hive­site.xml 

<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive</value>
</property>
<name>hive.exec.local.scratchdir</name>
<value>/tmp/hive</value>
<property>
</property>
<name>hive.downloaded.resources.dir</name>
<value>/tmp/hive_resources</value>
<property>
</property>
<name>hive.scratch.dir.permission</name>

<value>733</value>
<property>
</property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
<property>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
</property>
<property>
<name>hive.hwi.listen.host</name>
<value>localhost</value>
</property>
<property>

 

Installing Apache Hbase (1.1.0.1)

sudo tar xzvf ~/dev/hbase­-1.1.0.1-­bin.tar.gz -­C /usr/local
cd /usr/local
sudo ln ­-s hbase-­1.1.0.1 hbase
useradd hbase 
cd hbase 
 
Export HBASE_HOME to bashrc 
echo "export HBASE_HOME=/usr/local/hbase" >> /etc/bashrc 
  • For HBase 0.98.5 and later, you are required to set the JAVA_HOME environment variable before starting HBase

Enabling Ranger Hbase Plugins

  • We’ll start by extracting our build at the appropriate place.
cd /usr/local 
sudo tar zxvf ~/dev/incubator-­ranger/target/ranger­-0.5.0-­hbase­-plugin.tar.gz 
sudo ln -­s ranger-­0.5.0-­hbase-­plugin ranger-­hbase-­plugin
cd ranger­-hbase­-plugin 
  • Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
  • Change the insall.properties file 
PROPERTYVALUE
POLICY_MGR_URL

http://localhost:6080

REPOSITORY_NAMEhbasedev 
XAAUDIT.DB.IS_ENABLEDtrue 
XAAUDIT.DB.FLAVOURMYSQL
XAAUDIT.DB.HOSTNAMElocalhost
XAAUDIT.DB.DATABASE_NAMEranger_audit
XAAUDIT.DB.USER_NAMErangerlogger
XAAUDIT.DB.PASSWORDrangerlogger
  • Now enable the hbase­-plugin by running the enable­-hbase-­plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-­hbase-­plugin 
./enable-­hbase-­plugin.sh
  • Once these changes are done Restart hbase. This should start the association of ranger-­hbase-­plugin with hbase.
    1. You can verify by logging into the Ranger Admin Web interface ­> Audit Tab ­> Agents
  •  To Stop master and regionserver try: 
/usr/local/hbase/bin/hbase­-daemon.sh stop master 
/usr/local/hbase/bin/hbase­-daemon.sh stop regionserver
  •  Provide required permission to users in OS file system and hdfs file system according to your environment and requirement.

Installing Apache Knox Gateway

sudo tar ­-zxvf ~/dev/knox-­0.6.0.tar.gz -C /usr/local
cd /usr/local
sudo ln -­s knox-­0.6.0 knox
cd knox
Knox Master Secret : knox 

Enabling Ranger Knox Plugins

  • We’ll start by extracting our build at the appropriate place. 
cd /usr/local
tar ­-zxvf ~/dev/incubator-­ranger/target/ranger-­0.5.0-­knox­-plugin.tar.gz
sudo ln -­s ranger-­0.5.0-­knox­-plugin ranger-­knox-­plugin
cd ranger­-knox-­plugin 
  • Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
  • Change the insall.properties file 
PROPERYVALUE
POLICY_MGR_URL

http://localhost:6080

REPOSITORY_NAMEknoxdev
KNOX_HOME/usr/local/knox
XAAUDIT.DB.IS_ENABLEDtrue
XAAUDIT.DB.HOSTNAMElocalhost 
XAAUDIT.DB.DATABASE_NAMEranger 
XAAUDIT.DB.USER_NAMErangerlogger 
XAAUDIT.DB.PASSWORDrangerlogger 
  • Now enable the knox­-plugin by running the enable-­knox-­plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-­knox-­plugin
./enable-­knox-­plugin.sh
  • Once these changes are done Restart Knox ( Gateway / LDAP ) 
  • if you get permission denied error during knox start please provide required privileges to knox user. for example : 
chown -­R knox:knox /usr/local/knox/data 
chown ­-R knox:knox /usr/local/knox/logs
chown ­-R knox:knox /usr/local/knox/pids 
chown ­-R knox:hadoop /usr/local/knox/pids/*
  • You can verify by logging into the Ranger Admin Web interface ­> Audit > Agents

Trusting Self Signed Knox Certificate

       When Knox is listening on its SSL port with self signed certificate, you have to import SSL certificate of Knox into truststore used by XA PolicyManager. Here are steps for importing Knox SSL certificate in truststore used by XA PolicyManager.

  • Log in the machine running Knox
  • Export knox certificate
    1. cd $GATEWAY_HOME/data/security/keystores 
    2. This is typically /usr/local/knox/data/security/keystores on Linux machine. 
    3. keytool -­exportcert ­-alias gateway-­identity -­keystore gateway.jks -­file knox.crt
  • Copy knox.crt file onto machine running Ranger Admin/PolicyManager to a working directory, for example /usr/local/ranger-­admin
  • Replicate cacerts
cd /usr/local/ranger-­admin
cp $JAVA_HOME/jre/lib/security/cacerts cacertswithknox
  • Import Knox certificate into the replicated new keystore
keytool -­import -­trustcacerts -­file <knox.crt created above> -­alias knox -keystore cacertswithknox
password: changeit
  • Edit /usr/local/ranger-­admin/ews/ranger-­admin-­services.sh
    1. Add parameter -­Djavax.net.ssl.trustStore=<path to the cacertswithknox> to the java call in the script.
  • Restart Ranger Admin/PolicyManager. 

Installing Apache Storm (0.10.0)

sudo tar ­-zxvf ~/dev/apache­-storm-­0.10.0-­beta1.tar.gz -C /usr/local
cd /usr/local
sudo ln ­-s apache-­storm­-0.10.0­beta1 storm 
cd storm 

Enabling Ranger Storm Plugins

  • We’ll start by extracting our build at the appropriate place. 
cd /usr/local 
tar -­zxvf ~/dev/incubator-­ranger/target/ranger-­0.5.0-­storm-­plugin.tar.gz
sudo ln -­s ranger-­0.5.0-­storm-­plugin ranger-­storm-­plugin 
cd ranger-storm-plugin	
  • Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
  • Change the insall.properties file 
PROPERTYVALUE
POLICY_MGR_URL

http://localhost:6080

REPOSITORY_NAMEstormdev 
XAAUDIT.DB.IS_ENABLEDtrue
XAAUDIT.DB.HOSTNAMElocalhost 
XAAUDIT.DB.DATABASE_NAMEranger 
XAAUDIT.DB.USER_NAMErangerlogger 
XAAUDIT.DB.PASSWORDXAAUDIT.DB.PASSWORD=rangerlogger
  • Now enable the storm-plugin by running the enable-­storm-plugin.sh command (Remember to set JAVA_HOME)
cd /usr/local/ranger-­storm-­plugin
./enable­-storm-­plugin.sh  
  • Once these changes are done Restart Storm
  • You can verify by logging into the Ranger Admin Web interface ­> Audit > Agents

Installing Apache Yarn

You can run a MapReduce job on YARN in a pseudo­distributed mode by setting a few parameters and running ResourceManager daemon and NodeManager daemon in addition

The following instructions assume that hadoop installations steps mentioned in Installing Apache Hadoop are already executed.

Enabling Ranger Yarn Plugin

  • We’ll start by extracting our build at the appropriate place (/usr/local). 
cd /usr/local  
sudo tar zxvf ~/dev/incubator-­ranger/target/ranger-­0.5.0-­yarn-­plugin.tar.gz  
sudo ln -­s ranger-­0.5.0-­yarn-­plugin ranger-­yarn-­plugin
cd ranger­-yarn-­plugin
  • Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
  • Change the install.properties file 
PROPERTYVALUE
POLICY_MGR_URL

http://localhost:6080

REPOSITORY_NAMEyarndev
XAAUDIT.DB.IS_ENABLEDtrue
XAAUDIT.DB.FLAVOURMYSQL
XAAUDIT.DB.HOSTNAMElocalhost
XAAUDIT.DB.DATABASE_NAMEranger_audit 
XAAUDIT.DB.USER_NAMErangerlogger 
XAAUDIT.DB.PASSWORDrangerlogger 
  • Now enable the yarn­-plugin by running the enable-­yarn-­plugin.sh command.
cd /usr/local/ranger-­yarn-­plugin
./ enable-­yarn-­plugin.sh
  • One more change that we need to do is copy all the jar files from  ${hadoop_home}/lib
cp /usr/local/ranger­-yarn-­plugin/lib/*.jar /usr/local/hadoop/share/hadoop/yarn/lib/
  • if you get permission denied error during yarn start please provide required privileges to yarn user in local and hdfs file system. for example :
mkdir /var/log/yarn
chown ­-R yarn:yarn /var/log/yarn
  • Once these changes are done Start ResourceManager daemon and NodeManager daemon.
    • Start the ResourceManager on ResourceManager hosts. 
su yarn ­-c "/usr/local/hadoop/sbin/yarn­-daemon.sh start resourcemanager"
ps ­-ef | grep -­i resourcemanager 
    • Start the NodeManager on NodeManager hosts. 
su yarn ­-c "/usr/local/hadoop/sbin/yarn-­daemon.sh start nodemanager"
ps -­ef | grep -­i nodemanager
    • Stop the ResourceManager on ResourceManager hosts. 
su yarn ­-c "/usr/local/hadoop/sbin/yarn-­daemon.sh stop resourcemanager"
ps ­-ef | grep -­i resourcemanager
    • Stop the NodeManager on NodeManager hosts. 
su yarn -­c "/usr/local/hadoop/sbin/yarn-­daemon.sh stop nodemanager"
ps ­-ef | grep -­i nodemanager 
  • This should start the association of ranger-­yarn-­plugin with hadoop. 
    1. You can verify by logging into the Ranger Admin Web interface ­> Audit > Agents

Installing Ranger KMS (0.5.0)

Prerequisites: (Need to done for all host on which Ranger KMS needs to be installed)

  • Download “Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files” zip using below link depending upon the Java version used
    1. http://www.oracle.com/technetwork/java/javase/downloads/jce­-7­download-­432124.html 
    2. http://www.oracle.com/technetwork/java/javase/downloads/jce8­download­2133166.html
  • unzip the above downloaded zip file to java’s security folder (Depending upon the java version used)
unzip UnlimitedJCEPolicyJDK7.zip into $JDK_HOME/jre/lib/security 
unzip jce_policy-­8.zip into $JDK_HOME/jre/lib/security 

 STEPS FOR RANGER KMS:

  • We’ll start by extracting our build at the appropriate place(/usr/local). 
cd /usr/local 
sudo tar ­-zxvf ~/dev/incubator­-ranger/target/ranger-­0.5.0-­kms.tar.gz  
sudo ln ­-s ranger-­0.5.0-­kms ranger-­kms 
cd ranger-­kms 
  • Please note that Ranger KMS plugin is integrated with Ranger KMS and will be installed automatically when KMS is installed.
  • Now let’s edit the install.properties file. Here are the relevant lines that you should edit:
  • Change the install.properties file 
    1. DB_FLAVOR 
    2. SQL_CONNECTOR_JAR 
    3. db_root_user 
    4. db_root_password
    5. db_host 
    6. db_name
    7. db_user 
    8. db_password 
PROPERTYVALUE
POLICY_MGR_URL


http://localhost:6080

REPOSITORY_NAMEkmsdev
KMS_MASTER_KEY_PASSWD  enter master key password
XAAUDIT.DB.IS_ENABLEDtrue
XAAUDIT.DB.FLAVOURMYSQL 
XAAUDIT.DB.HOSTNAMElocalhost 
XAAUDIT.DB.DATABASE_NAMEranger_audit 
XAAUDIT.DB.USER_NAMErangerlogger
XAAUDIT.DB.PASSWORDrangerlogger
  • Edit “hdfs-site.xml”( Need to give provider else it will not support hadoop commands)Replace localhost with <internal host name> 
    1. Go to path cd /usr/local/hadoop/conf/ 
    2. vim hdfs­site.xml 
    3. For property “dfs.encryption.key.provider.uri” ,enter the value “kms://http@<internal host name>:9292/kms”
    4. save and quit 
  • Edit “core­site.xml”( Need to give provider else it will not support hadoop commands)
  • Replace localhost with <internal host name> 
    1. Go to path cd /usr/local/hadoop/conf/ 
    2. vim core­site.xml 
    3. For property “hadoop.security.key.provider.path” ,enter the value “kms://http@<internal host name>:9292/kms”
  • Once these changes are done Restart hadoop.
    1. Stop NameNode, SecondaryNameNode and DataNode daemon: 
su -­l hdfs -­c "/usr/local/hadoop/sbin/hadoop­daemon.sh stopnamenode"
su ­-l hdfs -­c "/usr/local/hadoop/sbin/hadoop­daemon.sh startnamenode"
  • Run setup  
./setup.sh
  • Start the kms server
ranger­-kms start
  • You can verify the plugin is communicating to Ranger admin in Audit-­>plugins tab.
  • If kmsdev service is not created in Ranger Admin then kms­-plugin will not able to connect to Ranger admin.
  • To Create the Kms service 
PROPERTYVALUE
REPOSITORY_NAME

name specified in installed.properties (e.g
kmsdev)

KMS URLkms://http@<internal host name>:9292/kms 
Username<username> (for e.g keyadmin)
Password<password> 
Check Test Connection

 

   ENABLING AUDIT LOGGING TO HDFS:

  • To enable Audit to HDFS for a plugin do the below:
    1. set XAAUDIT.HDFS.ENABLE = true for respective component plugin in the install.properties file which may be found in /usr/local/ranger­<component>­plugin/ directory. 
    2. configure NameNode host in the XAAUDIT.HDFS.HDFS_DIR.
    3. create a policy in HDFS service from Ranger Admin for individual component users (hive/hbase/knox/storm/yarn/kafka/kms) to give READ+ WRITE permission for the particular audit folder. i.e for enabling Hive component to log Audits to HDFS , we need to create a policy for hiveuser with READ+ WRITE permissions to respective audit directory
    4. Audit to HDFS caches logs in local directory, which can be specified in XAAUDIT.HDFS.LOCAL_BUFFER_DIRECTORY ( this can be like ‘/var/log/<component>/**), which is the path where audit is stored temporarily, likewise for archived logs we need to update XAAUDIT.HDFS.LOCAL_ARCHIVE_DIRECTORY value ( this can be like ‘/var/log/<component>/**), before enabling the plugin for the component.

Note that, HDFS audit logging is for archive purposes. For seeing audit report in the Ranger Admin UI, recommended option is Solr.

 

    ENABLING AUDIT LOGGING TO SOLR:

  1. To enable Audit report from Solr, do the below change in Ranger admin: 
  • Set following properties in install.properties of ranger service to work audit to solr in Ranger
PROPERTIESVALUE
audit_storesolr
audit_solr_urls 

http://solr_host:6083/solr/ranger_audits

audit_solr_user ranger_solr
audit_solr_password NONE 

Restart Ranger.

   2. To enable Audit to Solr for a plugin do the below:

  • Set following properties in install.properties of plugin to start logging audit to Solr : for eg Hbase
PROPERTYVALUE
XAAUDIT.SOLR.IS_ENABLEDtrue 
XAAUDIT.SOLR.ENABLEtrue 
XAAUDIT.SOLR.URL

http://solr_host:6083/solr/ranger_audits

XAAUDIT.SOLR.USERranger_solr  
XAAUDIT.SOLR.PASSWORDNONE 
XAAUDIT.SOLR.FILE_SPOOL_DIRvar/log/hadoop/hdfs/audit/solr/spool 
  • Enable ranger plugin for Hbase. 
  • Restart Hbase component. 

 

 

 

 

 

 


  • No labels