This page covers configuration changes and steps to enable HA for Ranger KMS using load balancer. Steps given are of Apache HTTPD.
Load Balancer without SSL
Install httpd and dependent packages
- ssh to in your cluster where you want to setup the load-balancer.
- Go to /usr/local
- cd /usr/local
- Download latest httpd from https://httpd.apache.org/download.cgi and dependencies i.e apr and apr-util from http://archive.apache.org/dist/apr/, since I have downloaded latest httpd so I shall be using the latest apr and apr-util
- Untar them in the same location
- tar -xvf httpd-2.4.25.tar.gz
- tar -xvf apr-1.5.2.tar.gz
- tar -xvf apr-util-1.5.4.tar.gz
- Now we will move apr and apr-util to srclib directory of httpd.
- Rename apr-1.5.2 to apr:
- mv apr-1.5.2/ apr
- Move to httpd/srclib directory:
- mv apr httpd-2.4.25/srclib/
- Rename apr-util-1.5.4 to apr-util
- mv apr-util-1.5.4/ apr-util
- Move apr-util to httpd/srclib directory:
- mv apr-util httpd-2.4.25/srclib/
- Rename apr-1.5.2 to apr:
- Install PCRE (Perl-Compatible Regular Expressions Library)
- yum install pcre pcre-devel
(here I am installing using yum however you can install latest bit from http://www.pcre.org/
- yum install pcre pcre-devel
- Install gcc (ANSI-C Compiler and Build System)
- yum install gcc
- Configuring the source tree
- cd /usr/local/httpd-2.4.25
- ./configure
- Make Build
- make
- Install
- make install
- Now let's test the work done till now
- cd /usr/local/apache2/bin
- ./apachectl start
- curl localhost
- output: <html><body><h1>It works!</h1></body></html>
Load balancer Configurations :
- Take the backup of conf file.
- cd /usr/local/apache2/conf
- cp httpd.conf ~/httpd.conf.backup
- Edit httpd.conf file
- vi /usr/local/apache2/conf/httpd.conf
- If you are not running your load balancer on default port 80 then change default listening port in line Listen 80
- Find below given module entries and enable them (remove #)
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
- Update server Admin email address or disable it
- #ServerAdmin you@example.com
- Now at the end of httpd.conf file add below given entry to read custom config file Include conf/ranger-kms-cluster.conf
Ranger KMS config Changes
- To create custom conf file
- vi ranger-kms-cluster.conf
- Add below given lines of the table then change the port. <VirtualHost *:88> mentioned there is having port 88 which should be same which you set in previous step.(12)
- URL mentioned in bold i.e BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1 and BalancerMember http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2 shall be Ranger-KMS hosts addresses.
# This is the Apache server configuration file providing SSL support.
# It contains the configuration directives to instruct the server how to
# serve pages over an https connection. For detailing information about these
# directives see <URL:http://httpd.apache.org/docs/2.2/mod/mod_ssl.html>
#
# Do NOT simply read the instructions in here without understanding
# what they do. They're here only as hints or reminders. If you are unsure
# consult the online docs. You have been warned.
#Listen 80
<VirtualHost *:88>
ProxyRequests off
ProxyPreserveHost on
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://rangercluster>
BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1
BalancerMember http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2
Order Deny,Allow
Deny from none
Allow from all
ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off
</Proxy>
# balancer-manager
# This tool is built into the mod_proxy_balancer
# module and will allow you to do some simple
# modifications to the balanced group via a gui
# web interface.
<Location /balancer-manager>
SetHandler balancer-manager
Order deny,allow
Allow from all
</Location>
ProxyPass /balancer-manager !
ProxyPass / balancer://rangerkmscluster/
ProxyPassReverse / balancer://rangerkmscluster/
</VirtualHost>
- Restart the apache httpd server
- cd /usr/local/apache2/bin
- ./apachectl restart
- Update the configs in HDFS service,
- hadoop.security.key.provider.path (core-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
- dfs.encryption.key.provider.uri (hdfs-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
- Also make sure, KMS URL for Ranger KMS service created on Ranger Admin is updated to kms://http@<load-balancer-node>:<load-balancer-port>/kms
- Restart Ranger KMS servers.
Load Balancer in SSL Env
Install httpd and dependent packages
Follow steps 1 to 5 same as mentioned in earlier section to install httpd. Then follow below given steps.
- Install other required packages
yum groupinstall "Development Tools"
yum install openssl-devel
yum install pcre-devel
- Configuring the source tree
- cd /usr/local/httpd-2.4.25
- ./configure --enable-so --enable-ssl --with-mpm=prefork --with-included-apr
- Make Build
- make
- Install
- make install
- Now let's test the work done till now
- cd /usr/local/apache2/bin
- ./apachectl start
- curl localhost
- output: <html><body><h1>It works!</h1></body></html>
HTTPS/SSL Configuration changes
- Take the backup of conf file.
- cd /usr/local/apache2/conf
- cp httpd.conf ~/httpd.conf.backup
- Edit httpd.conf file
- vi /usr/local/apache2/conf/httpd.conf
- If you are not running your apache server on default port 80 then change default listening port in line Listen 80
- Find below given module entries and enable them (remove #)
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
LoadModule ssl_module modules/mod_ssl.so
- If LoadModule ssl_module modules/mod_ssl.so is not available in your conf file then you should make sure that ssl module is installed properly otherwise you won’t able to use load balancer with ssl.
- Update server Admin email address or disable it.
- #ServerAdmin you@example.com
- Now let's test the work done till now.
- cd /usr/local/apache2/bin
- ./apachectl restart
- Update the configs in HDFS service,
- hadoop.security.key.provider.path (core-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
- dfs.encryption.key.provider.uri (hdfs-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
- Also make sure, KMS URL for Ranger KMS service created on Ranger Admin is updated to kms://http@<load-balancer-node>:<load-balancer-port>/kms
Certificate Generation
- Listing down steps to generate self signed certificate.
- cd ~/
Generate private key
- openssl genrsa -out server.key 2048
Generate CSR
- openssl req -new -key server.key -out server.csr
Generate Self Signed Key
- openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Generate keystore in PEM format
- openssl pkcs12 -export -passout pass:<passwordValue> -in server.crt -inkey server.key -out lbkeystore.p12 -name httpd.lb.server.alias
Use keytool to convert PEM format keystore to JKS format
- keytool -importkeystore -deststorepass <passwordValue> -destkeypass <destPasswordValue> -destkeystore httpd_lb_keystore.jks -srckeystore lbkeystore.p12 -srcstoretype PKCS12 -srcstorepass <passwordValue> -alias httpd.lb.server.alias
Create truststore of load balancer self signed keystore
keytool -export -keystore httpd_lb_keystore.jks -alias httpd.lb.server.alias -file httpd-lb-trust.cer
Copy generated key and certificate at appropriate location :
- cp server.crt /usr/local/apache2/conf/
- cp server.key /usr/local/apache2/conf/
Ranger KMS HA Configuration changes
- Now at the end of /usr/local/apache2/conf/httpd.conf file add below given entry to read custom config file.
- Include /usr/local/apache2/conf/ranger-kms-lb-ssl.conf
- Create custom conf file for SSL configuration and load balancer.
vi /usr/local/apache2/conf/ranger-kms-lb-ssl.conf
Let's configure Ranger KMS URLs on Load Balancer.
URL mentioned in Bold i.e BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1 and http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2 shall be Ranger-KMS hosts address.
While adding below given lines of the table then change the port. <VirtualHost *:8443> mentioned there is having port 8443 which should be same which you set in earlier step.
- Sample ranger-kms-lb-ssl.conf
<VirtualHost *:8443>
SSLEngine On
SSLProxyEngine On
SSLCertificateFile /usr/local/apache2/conf/server.crt
SSLCertificateKeyFile /usr/local/apache2/conf/server.key
SSLVerifyClient optional
SSLOptions +ExportCertData
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
ProxyRequests off
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://rangercluster>
BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1
BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=2
Order Deny,Allow
Deny from none
Allow from all
ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off
</Proxy>
# balancer-manager
# This tool is built into the mod_proxy_balancer
# module and will allow you to do some simple
# modifications to the balanced group via a gui
# web interface.
<Location /balancer-manager>
SetHandler balancer-manager
Order deny,allow
Allow from all
</Location>
ProxyPass /balancer-manager !
ProxyPass /balancer://rangerkmscluster/
ProxyPassReverse /balancer://rangerkmscluster/
</VirtualHost>
- Restart the apache httpd server
- /usr/local/apache2/bin/apachectl restart
- Done
Ranger KMS Plugin with HA
- To Enable Ranger KMS Plugin to communicate to Ranger KMS via load balancer copy the truststore file generated in step 12 of Certificate Creation, in KMS node. i.e in location /tmp and import the certificate in KMS truststore with below given command.
- keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/security/serverKeys//ranger-plugin-truststore.jks -storepass changeit
- Restart Ranger KMS.
- Now in the Audit --> Plugin tab, you should see an entry for Ranger KMs service name with Http Response Code as 200.