Apache Ranger uses Apache Solr to store audit logs and provides UI searching through the audit logs. It is required that Solr is installed and configured before installing RangerAdmin or any of the Ranger component plugins. There are options for Solr installation:
Configuring Apache Solr for high scale environment could be challenging. Please review the following links if you are expecting a very high volume of Audit Logs.
Please note, if you are using Apache Ambari, then Ambari maintains it's own solr-config template. So make sure to update the template also, otherwise, your manual changes might be overwritten. |
cd security-admin/contrib/solr_for_audit_setup (https://github.com/apache/incubator-ranger/tree/master/security-admin/contrib/solr_for_audit_setup)
cd $HOME git clone https://github.com/apache/incubator-ranger.git cd incubator-ranger/security-admin/contrib/solr_for_audit_setup |
You can download the Solr package from Apache Solr Downloads. Make sure Solr version is 5.2 or above. You might also let the Ranger script setup.sh to automatically download, install and configure Solr for you. If you want setup.sh to install Solr, then set the following properties in install.properties and then pick one of the configuration options from the next section.
| Property Name | Sample values | Description |
|---|---|---|
| SOLR_INSTALL | true | If this is set to true, then the setup.sh will download the Solr package and install it. |
| SOLR_DOWNLOAD_URL | http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz | It is recommended to use one for Apache mirrors to download the Solr package. Please pick the mirror site from http://lucene.apache.org/solr/mirrors-solr-latest-redir.html |
| SOLR_INSTALL_FOLDER | /opt/solr | The location where you want to install Solr. |
You can configure Solr to run as standalone or SolrCloud. If you want setup.sh to configure for standalone mode, then follow this section Standalone Configuration. If you are configuring for SolrCloud, then follow this section SolrCloud Configuration. If you want to configure your own Solr, then refer to this section Self Configuration
Modify the install.properties for the following properties:
| Property Name | Sample values | Description |
|---|---|---|
| JAVA_HOME | Provide the path to where you have installed JDK. If it is Hadoop, then you can check /etc/hadoop/conf/hadoop-env.sh for the value of JAVA_HOME. Please note, Solr only support JDK 1.7 and above. | |
| SOLR_USER | solr | The Linux user used to run Solr |
| SOLR_INSTALL_FOLDER | /opt/solr | Location where the Solr is installed. This is the same property used if you want setup.sh to install Solr |
| SOLR_RANGER_HOME | /opt/solr/ranger_audit_server | This is the location where Ranger related configuration and schema files will be copied |
| SOLR_RANGER_PORT | 6083 | The port you want Solr to listen on. |
| SOLR_DEPLOYMENT | standalone | The value standalone will configure solr to run as standalone. |
| SOLR_RANGER_DATA_FOLDER | /opt/solr/ranger_audit_server/data | This is the folder where you want the index data to be stored. It is important that the volume for this folder has enough disk space. It is recommended to have at least 1 TB free space for index data. Please take regular backup of this folder. |
| SOLR_LOG_FOLDER | /var/log/solr/ranger_audits | The folder where where want Solr logs to go. Make sure the volume for this folder has enough disk space. Please delete old log files on regular basis. |
| SOLR_MAX_MEM | 2g | This is the memory assigned for Solr. Make sure you provide adequate memory to the Solr process |
After updating the install.properties with the above values, run the following command:
./setup.sh <logs ...> ########## Done ################### Created file /opt/solr/ranger_audit_server/install_notes.txt with instructions to start and stop ################################### |
After setup.sh returns successfully, open the file $SOLR_RANGER_HOME/install_notes.txt for instructions to start and stop Solr.
After starting Solr for RangerAudit, Solr will listen at ${SOLR_PORT}. E.g Check Solr by accessing http://${SOLR_HOST}:6083 from your browser.
Installing and configuring SolrCloud needs few additional steps. We need to the following:
First, modify the install.properties for the following properties:
| Property Name | Sample values | Description |
|---|---|---|
| JAVA_HOME | Provide the path to where you have installed JDK. If it is Hadoop, then you can check /etc/hadoop/conf/hadoop-env.sh for the value of JAVA_HOME. Please note, Solr only support JDK 1.7 and above. | |
| SOLR_USER | solr | The Linux user used to run Solr process |
| SOLR_INSTALL_FOLDER | /opt/solr | Location where the Solr is installed. This is the same property used if you want setup.sh to install Solr |
| SOLR_RANGER_HOME | /opt/solr/ranger_audit_server | This is the location where the scripts and index data will be stored. Please note, in SolrCloud, there is no publicly configurable option to provide the location for storing the index data. So make sure you set the value to the folder where the volume as enough disk space. |
| SOLR_RANGER_PORT | 6083 | The port you want Solr to listen on. |
| SOLR_DEPLOYMENT | solrcloud | The value solrclould will configure solr to run as SolrCloud. |
| SOLR_ZK | ${zk_host}:2181/ranger_audits | It is recommended to give sub-folder to create the Ranger Audit related configurations. In this way, you can use ZooKeeper for other installations of Solr also. You have to give the zookeeper node only for the last node. E.g. zk1:2181,zk2:2182,zk3:2181/ranger_audits |
| SOLR_SHARDS | 1 | If you wish to distribute your audit logs, then you can use multiple shards. Make sure the number of shards is equal or less than the number of Solr nodes you will be running. |
| SOLR_REPLICATION | 1 | It is highly recommended to set up at least 2 nodes and replicate the indexes. This gives redundancy to index data and also load balancing of Solr queries. Please note, Solr recommends that you should have SOLR_SHARD * SOLR_REPLICATION Solr instances. E.g. if you have 3 shards and 2 replications, then you have 6 Solr instances. |
| SOLR_LOG_FOLDER | /var/log/solr/ranger_audits | The folder where where want Solr logs to go. Make sure the volume for this folder has enough disk space. Please delete old log files on regular basis. |
| SOLR_MAX_MEM | 2g | This is the memory assigned for Solr. Make sure you provide adequate memory to the Solr process. If you are using very high transaction/request Hadoop environment, then it might better to assign up to 32GB memory for Solr. |
After updating the install.properties with the above values, run the following command:
./setup.sh <logs ...> ########## Done ################### Created file /opt/solr/ranger_audit_server/install_notes.txt with instructions to start and stop ################################### |
After setup.sh returns successfully, open the file $SOLR_RANGER_HOME/install_notes.txt for additional steps:
For configuring SolrCloud, you need to do the following:
1. Using ./setup.sh script install and configure Solr for Ranger Audits on all other nodes also (don't start it yet)
2. Execute /opt/solr/ranger_audit_server/scripts/add_ranger_audits_conf_to_zk.sh (only once from any node where solr is installed)
3. Start Solr on all nodes: /opt/solr/ranger_audit_server/scripts/start_solr.sh
4. Create Ranger Audit collection: /opt/solr/ranger_audit_server/scripts/create_ranger_audits_collection.sh (only once \
from any node where solr is installed)
Make sure you have enough disk space for index. It is recommended to have at least 1TB free.
After starting Solr for RangerAudit, Solr will listen at ${SOLR_PORT}. E.g Check Solr by accessing http://${SOLR_HOST}:6083 from your browser.
If you are brave enough and you know what you are doing, then you can custom install Solr and configure it. The conf folder in the package contains the reference solrconflig.xml and schema.xml.
cd incubator-ranger/security-admin/contrib/solr_for_audit_setup $SOLR_INSTALL_HOME/bin/solr create_collection -c ranger_audits -d conf -shards 1 -replicationFactor 1 |
Ranger Admin and Ranger Plugins need the URL to Solr collection. Check the install_notes.txt for the appropriate value. The sample URL is:
http://${SOLR_HOST}:6083/solr/ranger_audits (Replace ${SOLR_HOST} with the server were Solr is installed.
For Ranger Admin, configure the following properties in install.properties:
#Source for Audit DB
# * audit_db is solr or db
audit_store=solr
# * audit_solr_url URL to Solr. E.g. http://<solr_host>:6083/solr/ranger_audits
audit_solr_urls=http://localhost:6083/solr/ranger_audits
For all plugins, configure the following properties in install.properties
XAAUDIT.SOLR.ENABLE=true
XAAUDIT.SOLR.URL=http://localhost:6083/solr/ranger_audits
(replace localhost with the Solr host)
