This page covers configuration changes and steps to enable HA for Ranger KMS using load balancer. Steps given are of Apache HTTPD.

Load Balancer without SSL

Install httpd and dependent packages

ssh to in your cluster where you want to setup the load-balancer.
Go to /usr/local
1. cd /usr/local
Download latest httpd from https://httpd.apache.org/download.cgi and dependencies i.e apr and apr-util from http://archive.apache.org/dist/apr/, since I have downloaded latest httpd so I shall be using the latest apr and apr-util
1. wget https://archive.apache.org/dist/httpd/httpd-2.4.25.tar.gz
2. wget http://archive.apache.org/dist/apr/apr-1.5.2.tar.gz
3. wget http://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz
Untar them in the same location
1. tar -xvf httpd-2.4.25.tar.gz
2. tar -xvf apr-1.5.2.tar.gz
3. tar -xvf apr-util-1.5.4.tar.gz
Now we will move apr and apr-util to srclib directory of httpd.
1. Rename apr-1.5.2 to apr:
  1. mv apr-1.5.2/ apr
  2. Move to httpd/srclib directory:
  3. mv apr httpd-2.4.25/srclib/
2. Rename apr-util-1.5.4 to apr-util
  1. mv apr-util-1.5.4/ apr-util
3. Move apr-util to httpd/srclib directory:
  1. mv apr-util httpd-2.4.25/srclib/
Install PCRE (Perl-Compatible Regular Expressions Library)
1. yum install pcre pcre-devel
  (here I am installing using yum however you can install latest bit from http://www.pcre.org/
Install gcc (ANSI-C Compiler and Build System)
1. yum install gcc
Configuring the source tree
1. cd /usr/local/httpd-2.4.25
2. ./configure
Make Build
1. make
Install
1. make install
Now let's test the work done till now
1. cd /usr/local/apache2/bin
2. ./apachectl start
3. curl localhost
  1. output: <html><body><h1>It works!</h1></body></html>

Load balancer Configurations :

Take the backup of conf file.
- cd /usr/local/apache2/conf
- cp httpd.conf ~/httpd.conf.backup

Edit httpd.conf file
- vi /usr/local/apache2/conf/httpd.conf
If you are not running your load balancer on default port 80 then change default listening port in line Listen 80
Find below given module entries and enable them (remove #)

LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so

LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so

LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so

Update server Admin email address or disable it
- #ServerAdmin you@example.com
Now at the end of httpd.conf file add below given entry to read custom config file Include conf/ranger-kms-cluster.conf

Ranger KMS config Changes

To create custom conf file
- vi ranger-kms-cluster.conf
- Add below given lines of the table then change the port. <VirtualHost *:88> mentioned there is having port 88 which should be same which you set in previous step.(12)
- URL mentioned in bold i.e BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1 and BalancerMember http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2 shall be Ranger-KMS hosts addresses.

# This is the Apache server configuration file providing SSL support.

# It contains the configuration directives to instruct the server how to

# serve pages over an https connection. For detailing information about these

# directives see <URL:http://httpd.apache.org/docs/2.2/mod/mod_ssl.html>

#

# Do NOT simply read the instructions in here without understanding

# what they do. They're here only as hints or reminders. If you are unsure

# consult the online docs. You have been warned.

#Listen 80

ProxyRequests off

ProxyPreserveHost on

Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1

BalancerMember http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2

Order Deny,Allow

Deny from none

Allow from all

ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off

</Proxy>

# balancer-manager

# This tool is built into the mod_proxy_balancer

# module and will allow you to do some simple

# modifications to the balanced group via a gui

# web interface.

SetHandler balancer-manager

Order deny,allow

Allow from all

</Location>

ProxyPass /balancer-manager !

ProxyPass / balancer://rangerkmscluster/

ProxyPassReverse / balancer://rangerkmscluster/

</VirtualHost>

Restart the apache httpd server
- cd /usr/local/apache2/bin
- ./apachectl restart
Update the configs in HDFS service,
- hadoop.security.key.provider.path (core-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
- dfs.encryption.key.provider.uri (hdfs-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
Also make sure, KMS URL for Ranger KMS service created on Ranger Admin is updated to kms://http@<load-balancer-node>:<load-balancer-port>/kms
Restart Ranger KMS servers.

Load Balancer in SSL Env

Install httpd and dependent packages

Follow steps 1 to 5 same as mentioned in earlier section to install httpd. Then follow below given steps.

Install other required packages
1. yum groupinstall "Development Tools"
2. yum install openssl-devel
3. yum install pcre-devel
Configuring the source tree
1. cd /usr/local/httpd-2.4.25
2. ./configure --enable-so --enable-ssl --with-mpm=prefork --with-included-apr
Make Build
1. make
Install
1. make install
Now let's test the work done till now
1. cd /usr/local/apache2/bin
2. ./apachectl start
3. curl localhost
  1. output: <html><body><h1>It works!</h1></body></html>

HTTPS/SSL Configuration changes

Take the backup of conf file.
- cd /usr/local/apache2/conf
- cp httpd.conf ~/httpd.conf.backup
Edit httpd.conf file
- vi /usr/local/apache2/conf/httpd.conf

If you are not running your apache server on default port 80 then change default listening port in line Listen 80
Find below given module entries and enable them (remove #)

LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so

LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so

LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so

LoadModule ssl_module modules/mod_ssl.so

If LoadModule ssl_module modules/mod_ssl.so is not available in your conf file then you should make sure that ssl module is installed properly otherwise you won’t able to use load balancer with ssl.
Update server Admin email address or disable it.
- #ServerAdmin you@example.com
Now let's test the work done till now.
- cd /usr/local/apache2/bin
- ./apachectl restart
Update the configs in HDFS service,
- hadoop.security.key.provider.path (core-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
- dfs.encryption.key.provider.uri (hdfs-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
Also make sure, KMS URL for Ranger KMS service created on Ranger Admin is updated to kms://http@<load-balancer-node>:<load-balancer-port>/kms

Certificate Generation

Listing down steps to generate self signed certificate.
1. cd ~/
2. Generate private key
3. openssl genrsa -out server.key 2048
4. Generate CSR
5. openssl req -new -key server.key -out server.csr
6. Generate Self Signed Key
7. openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
8. Generate keystore in PEM format
9. openssl pkcs12 -export -passout pass:<passwordValue> -in server.crt -inkey server.key -out lbkeystore.p12 -name httpd.lb.server.alias
10. Use keytool to convert PEM format keystore to JKS format
11. keytool -importkeystore -deststorepass <passwordValue> -destkeypass <destPasswordValue> -destkeystore httpd_lb_keystore.jks -srckeystore lbkeystore.p12 -srcstoretype PKCS12 -srcstorepass <passwordValue> -alias httpd.lb.server.alias
12. Create truststore of load balancer self signed keystore
  1. keytool -export -keystore httpd_lb_keystore.jks -alias httpd.lb.server.alias -file httpd-lb-trust.cer
13. Copy generated key and certificate at appropriate location :
  1. cp server.crt /usr/local/apache2/conf/
  2. cp server.key /usr/local/apache2/conf/

Ranger KMS HA Configuration changes

Now at the end of /usr/local/apache2/conf/httpd.conf file add below given entry to read custom config file.
- Include /usr/local/apache2/conf/ranger-kms-lb-ssl.conf
Create custom conf file for SSL configuration and load balancer.
- vi /usr/local/apache2/conf/ranger-kms-lb-ssl.conf
Let's configure Ranger KMS URLs on Load Balancer.
- URL mentioned in Bold i.e BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1 and http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2 shall be Ranger-KMS hosts address.
- While adding below given lines of the table then change the port. <VirtualHost *:8443> mentioned there is having port 8443 which should be same which you set in earlier step.
Sample ranger-kms-lb-ssl.conf

SSLEngine On

SSLProxyEngine On

SSLCertificateFile /usr/local/apache2/conf/server.crt

SSLCertificateKeyFile /usr/local/apache2/conf/server.key

SSLVerifyClient optional

SSLOptions +ExportCertData

SSLProxyVerify none

SSLProxyCheckPeerCN off

SSLProxyCheckPeerName off

SSLProxyCheckPeerExpire off

ProxyRequests off

Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1

BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=2

Order Deny,Allow

Deny from none

Allow from all

ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off

</Proxy>

# balancer-manager

# This tool is built into the mod_proxy_balancer

# module and will allow you to do some simple

# modifications to the balanced group via a gui

# web interface.

SetHandler balancer-manager

Order deny,allow

Allow from all

</Location>

ProxyPass /balancer-manager !

ProxyPass /balancer://rangerkmscluster/

ProxyPassReverse /balancer://rangerkmscluster/

</VirtualHost>

Restart the apache httpd server
- /usr/local/apache2/bin/apachectl restart
Done

Ranger KMS Plugin with HA

To Enable Ranger KMS Plugin to communicate to Ranger KMS via load balancer copy the truststore file generated in step 12 of Certificate Creation, in KMS node. i.e in location /tmp and import the certificate in KMS truststore with below given command.
- keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/security/serverKeys//ranger-plugin-truststore.jks -storepass changeit
Restart Ranger KMS.
Now in the Audit --> Plugin tab, you should see an entry for Ranger KMs service name with Http Response Code as 200.