This page covers configuration changes and steps to enable HA for Ranger KMS using load balancer. Steps given are of Apache HTTPD.

Load Balancer without SSL

Install httpd and dependent packages

  1. ssh to in your cluster where you want to setup the load-balancer.
  2. Go to /usr/local
    1. cd /usr/local
  3. Download latest httpd from https://httpd.apache.org/download.cgi and dependencies i.e apr and apr-util from http://archive.apache.org/dist/apr/, since I have downloaded latest httpd so I shall be using the latest apr and apr-util
    1. wget https://archive.apache.org/dist/httpd/httpd-2.4.25.tar.gz
    2. wget http://archive.apache.org/dist/apr/apr-1.5.2.tar.gz
    3. wget http://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz
  4. Untar them in the same location
    1. tar -xvf httpd-2.4.25.tar.gz
    2. tar -xvf apr-1.5.2.tar.gz
    3. tar -xvf apr-util-1.5.4.tar.gz
  5. Now we will move apr and apr-util to srclib directory of httpd.
    1. Rename apr-1.5.2 to apr:
      1. mv apr-1.5.2/ apr
      2. Move to httpd/srclib directory:
      3. mv apr httpd-2.4.25/srclib/
    2. Rename apr-util-1.5.4 to apr-util
      1. mv apr-util-1.5.4/ apr-util
    3. Move apr-util to httpd/srclib directory:
      1. mv apr-util httpd-2.4.25/srclib/
  6. Install PCRE (Perl-Compatible Regular Expressions Library)
    1. yum install pcre pcre-devel
      (here I am installing using yum however you can install latest bit from http://www.pcre.org/
  7. Install gcc (ANSI-C Compiler and Build System)
    1. yum install gcc
  8. Configuring the source tree
    1. cd /usr/local/httpd-2.4.25
    2. ./configure
  9. Make Build  
    1. make
  10. Install
    1. make install
  11. Now let's test the work done till now
    1. cd /usr/local/apache2/bin
    2. ./apachectl start
    3. curl localhost
      1. output: <html><body><h1>It works!</h1></body></html>

Load balancer Configurations :

  • Take the backup of conf file.
    • cd /usr/local/apache2/conf
    • cp httpd.conf ~/httpd.conf.backup
  • Edit httpd.conf file
    • vi /usr/local/apache2/conf/httpd.conf
  • If you are not running your load balancer on default port 80 then change default listening port in line Listen 80
  • Find below given module entries and enable them (remove #)

LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so

LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so

LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so


  • Update server Admin email address or disable it
  • Now at the end of httpd.conf file add below given entry to read custom config file Include conf/ranger-kms-cluster.conf

Ranger KMS config Changes

  • To create custom conf file
    • vi ranger-kms-cluster.conf
    • Add below given lines of the table then change the port. <VirtualHost *:88> mentioned there is having port 88 which should be same which you set in previous step.(12)
    • URL mentioned in bold i.e BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1 and BalancerMember http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2 shall be Ranger-KMS hosts addresses. 


# This is the Apache server configuration file providing SSL support.

# It contains the configuration directives to instruct the server how to

# serve pages over an https connection. For detailing information about these

# directives see <URL:http://httpd.apache.org/docs/2.2/mod/mod_ssl.html>

#

# Do NOT simply read the instructions in here without understanding

# what they do.  They're here only as hints or reminders.  If you are unsure

# consult the online docs. You have been warned.

#Listen 80

<VirtualHost *:88>

       ProxyRequests off

       ProxyPreserveHost on

       Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

       <Proxy balancer://rangercluster>

               BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1

               BalancerMember http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2

               Order Deny,Allow

               Deny from none

               Allow from all

               ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off

       </Proxy>

       # balancer-manager

       # This tool is built into the mod_proxy_balancer

       # module and will allow you to do some simple

       # modifications to the balanced group via a gui

       # web interface.

       <Location /balancer-manager>

               SetHandler balancer-manager

               Order deny,allow

               Allow from all

       </Location>

      ProxyPass /balancer-manager !

      ProxyPass / balancer://rangerkmscluster/

      ProxyPassReverse / balancer://rangerkmscluster/

</VirtualHost>


  • Restart the apache httpd server
    • cd /usr/local/apache2/bin
    • ./apachectl restart
  • Update the configs in HDFS service,
    • hadoop.security.key.provider.path (core-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
    • dfs.encryption.key.provider.uri (hdfs-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
  • Also make sure, KMS URL for Ranger KMS service created on Ranger Admin is updated to kms://http@<load-balancer-node>:<load-balancer-port>/kms
  • Restart Ranger KMS servers.

Load Balancer in SSL Env 

Install httpd and dependent packages

Follow steps 1 to 5 same as mentioned in earlier section to install httpd. Then follow below given steps. 

  1. Install other required packages

    1. yum groupinstall "Development Tools"

    2. yum install openssl-devel

    3. yum install pcre-devel

  2. Configuring the source tree
    1. cd /usr/local/httpd-2.4.25
    2. ./configure --enable-so --enable-ssl --with-mpm=prefork --with-included-apr
  3. Make Build  
    1. make
  4.  Install
    1. make install
  5. Now let's test the work done till now
    1. cd /usr/local/apache2/bin
    2. ./apachectl start
    3. curl localhost
      1. output: <html><body><h1>It works!</h1></body></html>

HTTPS/SSL Configuration changes

  • Take the backup of conf file.
    • cd /usr/local/apache2/conf
    • cp httpd.conf ~/httpd.conf.backup
  • Edit httpd.conf file
    • vi /usr/local/apache2/conf/httpd.conf
  • If you are not running your apache server on default port 80 then change default listening port in line Listen 80
  • Find below given module entries and enable them (remove #)

LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so            

LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so

LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so

LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so

LoadModule ssl_module modules/mod_ssl.so

  • If LoadModule ssl_module modules/mod_ssl.so is not available in your conf file then you should make sure that ssl module is installed properly otherwise you won’t able to use load balancer with ssl.
  • Update server Admin email address or disable it.
  • Now let's test the work done till now.
    • cd /usr/local/apache2/bin
    • ./apachectl restart
  • Update the configs in HDFS service,
    • hadoop.security.key.provider.path (core-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
    • dfs.encryption.key.provider.uri (hdfs-site) - kms://http@<load-balancer-node>:<load-balancer-port>/kms
  • Also make sure, KMS URL for Ranger KMS service created on Ranger Admin is updated to kms://http@<load-balancer-node>:<load-balancer-port>/kms

Certificate Generation

  • Listing down steps to generate self signed certificate.
    1. cd ~/

    2. Generate private key

    3. openssl genrsa -out server.key 2048

    4. Generate CSR

    5. openssl req -new -key server.key -out server.csr

    6. Generate Self Signed Key

    7. openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

    8. Generate keystore in PEM format

    9. openssl pkcs12 -export -passout pass:<passwordValue> -in server.crt -inkey server.key -out lbkeystore.p12 -name httpd.lb.server.alias

    10. Use keytool to convert PEM format keystore to JKS format

    11. keytool -importkeystore -deststorepass <passwordValue> -destkeypass <destPasswordValue> -destkeystore httpd_lb_keystore.jks -srckeystore lbkeystore.p12 -srcstoretype PKCS12 -srcstorepass <passwordValue> -alias httpd.lb.server.alias

    12. Create truststore of load balancer self signed keystore

      1. keytool -export -keystore httpd_lb_keystore.jks -alias httpd.lb.server.alias -file httpd-lb-trust.cer

    13. Copy generated key and certificate at appropriate location : 

      1. cp server.crt /usr/local/apache2/conf/
      2. cp server.key /usr/local/apache2/conf/

Ranger KMS HA Configuration changes

  • Now at the end of /usr/local/apache2/conf/httpd.conf file add below given entry to read custom config file.
    • Include /usr/local/apache2/conf/ranger-kms-lb-ssl.conf
  • Create custom conf file for SSL configuration and load balancer.

    • vi /usr/local/apache2/conf/ranger-kms-lb-ssl.conf

  • Let's configure Ranger KMS URLs on Load Balancer.

    • URL mentioned in Bold i.e BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1 and http://ranger-kms-host2-fqdn:9292 loadfactor=1 route=2 shall be Ranger-KMS hosts address.

    • While adding below given lines of the table then change the port. <VirtualHost *:8443> mentioned there is having port 8443 which should be same which you set in earlier step.

  • Sample ranger-kms-lb-ssl.conf

<VirtualHost *:8443>

       SSLEngine On

       SSLProxyEngine On

       SSLCertificateFile /usr/local/apache2/conf/server.crt

       SSLCertificateKeyFile /usr/local/apache2/conf/server.key

       SSLVerifyClient optional

       SSLOptions +ExportCertData

       SSLProxyVerify none

       SSLProxyCheckPeerCN off

       SSLProxyCheckPeerName off

       SSLProxyCheckPeerExpire off

       ProxyRequests off

       Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

       <Proxy balancer://rangercluster>

           BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=1

BalancerMember http://ranger-kms-host1-fqdn:9292 loadfactor=1 route=2

       

               Order Deny,Allow

               Deny from none

               Allow from all

               ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off

       </Proxy>

       # balancer-manager

       # This tool is built into the mod_proxy_balancer

       # module and will allow you to do some simple

       # modifications to the balanced group via a gui

       # web interface.

       <Location /balancer-manager>

               SetHandler balancer-manager

               Order deny,allow

               Allow from all

       </Location>

      ProxyPass /balancer-manager !

      ProxyPass /balancer://rangerkmscluster/

      ProxyPassReverse /balancer://rangerkmscluster/

</VirtualHost>

  • Restart the apache httpd server
    • /usr/local/apache2/bin/apachectl restart
  • Done

Ranger KMS Plugin with HA

  • To Enable Ranger KMS Plugin to communicate to Ranger KMS via load balancer copy the truststore file generated in step 12 of Certificate Creation,  in KMS node. i.e in location /tmp and import the certificate in KMS truststore with below given command.
    • keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/security/serverKeys//ranger-plugin-truststore.jks -storepass changeit
  • Restart Ranger KMS.
  • Now in the Audit --> Plugin tab, you should see an entry for Ranger KMs service name with Http Response Code as 200.


  • No labels