Apache Solr Documentation

6.5 Ref Guide (PDF Download)
Solr Tutorial
Solr Community Wiki

Older Versions of this Guide (PDF)

Ref Guide Topics

Meta-Documentation

*** As of June 2017, the latest Solr Ref Guide is located at https://lucene.apache.org/solr/guide ***

Please note comments on these pages have now been disabled for all users.

Skip to end of metadata
Go to start of metadata
65%

Index Replication distributes complete copies of a master index to one or more slave servers. The master server continues to manage updates to the index. All querying is handled by the slaves. This division of labor enables Solr to scale to provide adequate responsiveness to queries against large search volumes.

The figure below shows a Solr configuration using index replication. The master server's index is replicated on the slaves.


A Solr index can be replicated across multiple slave servers, which then process requests.

Topics covered in this section:

2

 

Index Replication in Solr

Solr includes a Java implementation of index replication that works over HTTP:

  • The configuration affecting replication is controlled by a single file, solrconfig.xml
  • Supports the replication of configuration files as well as index files
  • Works across platforms with same configuration
  • No reliance on OS-dependent file system features (eg: hard links)
  • Tightly integrated with Solr; an admin page offers fine-grained control of each aspect of replication
  • The Java-based replication feature is implemented as a request handler. Configuring replication is therefore similar to any normal request handler.

 

Replication In SolrCloud

Although there is no explicit concept of "master/slave" nodes in a SolrCloud cluster, the ReplicationHandler discussed on this page is still used by SolrCloud as needed to support "shard recovery" – but this is done in a peer to peer manner.  When using SolrCloud, the ReplicationHandler must be available via the /replication path. Solr does this implicitly unless overridden explicitly in your solrconfig.xml, but If you wish to override the default behavior, make certain that you do not explicitly set any of the "master" or "slave" configuration options mentioned below, or they will interfere with normal SolrCloud operation.

Replication Terminology

The table below defines the key terms associated with Solr replication.

Term

Definition

Index

A Lucene index is a directory of files. These files make up the searchable and returnable data of a Solr Core.

Distribution

The copying of an index from the master server to all slaves. The distribution process takes advantage of Lucene's index file structure.

Inserts and Deletes

As inserts and deletes occur in the index, the directory remains unchanged. Documents are always inserted into newly created files. Documents that are deleted are not removed from the files. They are flagged in the file, deletable, and are not removed from the files until the index is optimized.

Master and Slave

A Solr replication master is a single node which receives all updates initially and keeps everything organized. Solr replication slave nodes receive no updates directly, instead all changes (such as inserts, updates, deletes, etc.) are made against the single master node. Changes made on the master are distributed to all the slave nodes which service all query requests from the clients.

Update

An update is a single change request against a single Solr instance. It may be a request to delete a document, add a new document, change a document, delete all documents matching a query, etc. Updates are handled synchronously within an individual Solr instance.

Optimization

A process that compacts the index and merges segments in order to improve query performance. Optimization should only be run on the master nodes. An optimized index may give query performance gains compared to an index that has become fragmented over a period of time with many updates. Distributing an optimized index requires a much longer time than the distribution of new segments to an un-optimized index.

Segments

A self contained subset of an index consisting of some documents and data structures related to the inverted index of terms in those documents.

mergeFactor

A parameter that controls the number of  segments in an index. For example, when mergeFactor is set to 3, Solr will fill one segment with documents until the limit maxBufferedDocs is met, then it will start a new segment. When the number of segments specified by mergeFactor is reached (in this example, 3) then Solr will merge all the segments into a single index file, then begin writing new documents to a new segment.

Snapshot

A directory containing hard links to the data files of an index. Snapshots are distributed from the master nodes when the slaves pull them, "smart copying" any segments the slave node does not have in snapshot directory that contains the hard links to the most recent index data files.

Configuring the ReplicationHandler

In addition to ReplicationHandler configuration options specific to the master/slave roles, there are a few special configuration options that are generally supported (even when using SolrCloud).

  • maxNumberOfBackups an integer value dictating the maximum number of backups this node will keep on disk as it receives backup commands.
  • Similar to most other request handlers in Solr you may configure a set of "defaults, invariants, and/or appends" parameters corresponding with any request parameters supported by the ReplicationHandler when processing commands.

Configuring the Replication RequestHandler on a Master Server

Before running a replication, you should set the following parameters on initialization of the handler:

Name

Description

replicateAfter

String specifying action after which replication should occur. Valid values are commit, optimize, or startup. There can be multiple values for this parameter. If you use "startup", you need to have a "commit" and/or "optimize" entry also if you want to trigger replication on future commits or optimizes.

backupAfter

String specifying action after which a backup should occur. Valid values are commit, optimize, or startup. There can be multiple values for this parameter. It is not required for replication, it just makes a backup.

maxNumberOfBackups

Integer specifying how many backups to keep. This can be used to delete all but the most recent N backups.

confFiles

The configuration files to replicate, separated by a comma.

commitReserveDuration

If your commits are very frequent and your network is slow, you can tweak this parameter to increase the amount of time taken to download 5Mb from the master to a slave. The default is 10 seconds.

The example below shows a possible 'master' configuration for the ReplicationHandler, including a fixed number of backups and an invariant setting for the maxWriteMBPerSec request parameter to prevent slaves from saturating its network interface

#666666solid optimize optimize schema.xml,stopwords.txt,elevate.xml 00:00:10 2 16 ]]>

Replicating solrconfig.xml

In the configuration file on the master server, include a line like the following:

#666666solidsolrconfig_slave.xml:solrconfig.xml,x.xml,y.xml ]]>

This ensures that the local configuration solrconfig_slave.xml will be saved as solrconfig.xml on the slave. All other files will be saved with their original names.

On the master server, the file name of the slave configuration file can be anything, as long as the name is correctly identified in the confFiles string; then it will be saved as whatever file name appears after the colon ':'.

Configuring the Replication RequestHandler on a Slave Server

The code below shows how to configure a ReplicationHandler on a slave.

#666666solid http://remote_host:port/solr/core_name/replication 00:00:20 internal 5000 10000 username password ]]>

Setting Up a Repeater with the ReplicationHandler

A master may be able to serve only so many slaves without affecting performance. Some organizations have deployed slave servers across multiple data centers. If each slave downloads the index from a remote data center, the resulting download may consume too much network bandwidth. To avoid performance degradation in cases like this, you can configure one or more slaves as repeaters. A repeater is simply a node that acts as both a master and a slave.

  • To configure a server as a repeater, the definition of the Replication requestHandler in the solrconfig.xml file must include file lists of use for both masters and slaves.
  • Be sure to set the replicateAfter parameter to commit, even if replicateAfter is set to optimize on the main master. This is because on a repeater (or any slave), a commit is called only after the index is downloaded. The optimize command is never called on slaves.
  • Optionally, one can configure the repeater to fetch compressed files from the master through the compression parameter to reduce the index download time.

Here is an example of a ReplicationHandler configuration for a repeater:

#666666solid commit schema.xml,stopwords.txt,synonyms.txt http://master.solr.company.com:8983/solr/core_name/replication 00:00:60 ]]>

Commit and Optimize Operations

When a commit or optimize operation is performed on the master, the RequestHandler reads the list of file names which are associated with each commit point. This relies on the replicateAfter parameter in the configuration to decide which types of events should trigger replication.

Setting on the Master

Description

commit

Triggers replication whenever a commit is performed on the master index.

optimize

Triggers replication whenever the master index is optimized.

startup

Triggers replication whenever the master index starts up.

The replicateAfter parameter can accept multiple arguments. For example:

#666666solidstartup commit optimize ]]>

Slave Replication

The master is totally unaware of the slaves. The slave continuously keeps polling the master (depending on the pollInterval parameter) to check the current index version of the master. If the slave finds out that the master has a newer version of the index it initiates a replication process. The steps are as follows:

  • The slave issues a filelist command to get the list of the files. This command returns the names of the files as well as some metadata (for example, size, a lastmodified timestamp, an alias if any).
  • The slave checks with its own index if it has any of those files in the local index. It then runs the filecontent command to download the missing files. This uses a custom format (akin to the HTTP chunked encoding) to download the full content or a part of each file. If the connection breaks in between, the download resumes from the point it failed. At any point, the slave tries 5 times before giving up a replication altogether.
  • The files are downloaded into a temp directory, so that if either the slave or the master crashes during the download process, no files will be corrupted. Instead, the current replication will simply abort.
  • After the download completes, all the new files are moved to the live index directory and the file's timestamp is same as its counterpart on the master.
  • A commit command is issued on the slave by the Slave's ReplicationHandler and the new index is loaded.

Replicating Configuration Files

To replicate configuration files, list them using using the confFiles parameter. Only files found in the conf directory of the master's Solr instance will be replicated.

Solr replicates configuration files only when the index itself is replicated. That means even if a configuration file is changed on the master, that file will be replicated only after there is a new commit/optimize on master's index.

Unlike the index files, where the timestamp is good enough to figure out if they are identical, configuration files are compared against their checksum. The schema.xml files (on master and slave) are judged to be identical if their checksums are identical.

As a precaution when replicating configuration files, Solr copies configuration files to a temporary directory before moving them into their ultimate location in the conf directory. The old configuration files are then renamed and kept in the same conf/ directory. The ReplicationHandler does not automatically clean up these old files.

If a replication involved downloading of at least one configuration file, the ReplicationHandler issues a core-reload command instead of a commit command.

Resolving Corruption Issues on Slave Servers

If documents are added to the slave, then the slave is no longer in sync with its master. However, the slave will not undertake any action to put itself in sync, until the master has new index data. When a commit operation takes place on the master, the index version of the master becomes different from that of the slave. The slave then fetches the list of files and finds that some of the files present on the master are also present in the local index but with different sizes and timestamps. This means that the master and slave have incompatible indexes. To correct this problem, the slave then copies all the index files from master to a new index directory and asks the core to load the fresh index from the new directory.

HTTP API Commands for the ReplicationHandler

You can use the HTTP commands below to control the ReplicationHandler's operations.

Command

Description

http://master_host:port/solr/core_name/replication?command=enablereplication

Enables replication on the master for all its slaves.

http://master_host:port/solr/core_name/replication?command=disablereplication

Disables replication on the master for all its slaves.

http://host:port/solr/core_name/replication?command=indexversion

Returns the version of the latest replicatable index on the specified master or slave.

http://slave_host:port/solr/core_name/replication?command=fetchindex

Forces the specified slave to fetch a copy of the index from its master.

If you like, you can pass an extra attribute such as masterUrl or compression (or any other parameter which is specified in the <lst name="slave"> tag) to do a one time replication from a master. This obviates the need for hard-coding the master in the slave.

http://slave_host:port/solr/core_name/replication?command=abortfetch

Aborts copying an index from a master to the specified slave.

http://slave_host:port/solr/core_name/replication?command=enablepoll

Enables the specified slave to poll for changes on the master.

http://slave_host:port/solr/core_name/replication?command=disablepoll

Disables the specified slave from polling for changes on the master.

http://slave_host:port/solr/core_name/replication?command=details

Retrieves configuration details and current status.

http://host:port/solr/core_name/replication?command=filelist&generation=<generation-number>

Retrieves a list of Lucene files present in the specified host's index. You can discover the generation number of the index by running the indexversion command.

http://master_host:port/solr/core_name/replication?command=backup

Creates a backup on master if there are committed index data in the server; otherwise, does nothing. This command is useful for making periodic backups.

supported request parameters:

  • numberToKeep: request parameter can be used with the backup command unless the maxNumberOfBackups initialization parameter has been specified on the handler – in which case maxNumberOfBackups is always used and attempts to use the numberToKeep request parameter will cause an error.
  • name : (optional) Backup name . The snapshot will be created in a directory called snapshot.<name> within the data directory of the core . By default the name is generated using date in yyyyMMddHHmmssSSS format. If location parameter is passed , that would be used instead of the data directory
  • location: Backup location
http:// master_host:port /solr/core_name/replication?command=deletebackup

Delete any backup created using the backup command .

request parameters:

  • name: The name of the snapshot . A snapshot with the name snapshot.<name> must exist .If not, an error is thrown
  • location: Location where the snapshot is created

Distribution and Optimization

Optimizing an index is not something most users should generally worry about - but in particular users should be aware of the impacts of optimizing an index when using the ReplicationHandler.

The time required to optimize a master index can vary dramatically. A small index may be optimized in minutes. A very large index may take hours. The variables include the size of the index and the speed of the hardware.

Distributing a newly optimized index may take only a few minutes or up to an hour or more, again depending on the size of the index and the performance capabilities of network connections and disks. During optimization the machine is under load and does not process queries very well. Given a schedule of updates being driven a few times an hour to the slaves, we cannot run an optimize with every committed snapshot.

Copying an optimized index means that the entire index will need to be transferred during the next snappull. This is a large expense, but not nearly as huge as running the optimize everywhere. Consider this example: on a three-slave one-master configuration, distributing a newly-optimized index takes approximately 80 seconds total. Rolling the change across a tier would require approximately ten minutes per machine (or machine group). If this optimize were rolled across the query tier, and if each slave node being optimized were disabled and not receiving queries, a rollout would take at least twenty minutes and potentially as long as an hour and a half. Additionally, the files would need to be synchronized so that the following the optimize, snappull would not think that the independently optimized files were different in any way. This would also leave the door open to independent corruption of indexes instead of each being a perfect copy of the master.

Optimizing on the master allows for a straight-forward optimization operation. No query slaves need to be taken out of service. The optimized index can be distributed in the background as queries are being normally serviced. The optimization can occur at any time convenient to the application providing index updates.

While optimizing may have some benefits in some situations, a rapidly changing index will not retain those benefits for long, and since optimization is an intensive process, it may be better to consider other options, such as lowering the merge factor (discussed in the section on Index Configuration).

 

  • No labels

14 Comments

  1. A typo: "If you use "startup", you need to have a "commit" and/or "optimize" entry also if you want to trigger replication on futures commits or optimizes. "

    should be:

    "If you use "startup", you need to have a "commit" and/or "optimize" entry also if you want to trigger replication on future commits or optimizes. "

  2. A typo: " The slave continuously keeps polling the master (depending on the pollInterval parameter) to check the current index version the master."

    should be:

    " The slave continuously keeps polling the master (depending on the pollInterval parameter) to check the current index version of the master."

    1. I fixed both of them. Thanks Dmitry!

  3. Several issues: "After the download completes, all the new files are 'mov'ed to the live index directory and the file's timestamp is same as its counterpart in on the master master."

    1. why 'mov'ed and not just "moved"?
    2. "..as its counterpart in on" => "..as its counterpart on"
    3. "master master" twice

  4. Period is missing in: "Only files found in the conf directory of the master's Solr instance will be replicated"

  5. "To correct this problem, the slave then copies all the index files from master to a new index directory and and asks the core to load the fresh index from the new directory."

    twice "and and"

  6. (table) Period is missing: "Returns the version of the latest replicatable index on the specified master or slave "

  7. Closing parenthesis is forgotten: "While optimizing has many benefits, a rapidly changing index will not retain those benefits for long, and since optimization is an intensive process, it may be better to consider other options, such as lowering the merge factor (discussed in this Guide in the section on Configuring the Lucene Index Writers."

  8. SOLR 5.2.1 seems not to be happy with the client-example. It complains "'masterUrl' must be specified without the /replication suffix".

    The minimal Configuration for a server seems to be '<requestHandler name="/replication" class="solr.ReplicationHandler" >', the parameter "replicateAfter" defaults to "commit"

    Thanks for your work.

     

    1. you are right .The message is confusing and leads you to believe that it has failed. We should change the examples and omit the /replication" suffix

  9. I maintain a old system, with replication beside a Master01 with 6 Slaves. When data of Master01 up to 10G, the replication in code or in Admin Solr web ok in 5 slave, but 1 slave failed. I tried to increase Ram à HDD but this error still occurs. I think that may be due to a certain configuration of Master01 has limited the amount of data read in one moment? I began working with Solr, This error has been pending for several days. Hope everyone help or offer suggestions. Thanks!

    1. These comments are not the correct place to ask for support.  We have a mailing list and an IRC channel.  Please use one of those resources.

      http://lucene.apache.org/solr/community.html