Index Handlers are Request Handlers designed to add, delete and update documents to the index. In addition to having plugins for importing rich documents using Tika or from structured data sources using the Data Import Handler, Solr natively supports indexing structured documents in XML, CSV and JSON.
The recommended way to configure and use request handlers is with path based names that map to paths in the request url. However, request handlers can also be specified with the
qt (query type) parameter if the
is appropriately configured. It is possible to access the same handler using more than one name, which can be useful if you wish to specify different sets of default options.
A single unified update request handler supports XML, CSV, JSON, and javabin update requests, delegating to the appropriate
ContentStreamLoader based on the
Content-Type of the ContentStream.
The default configuration file has the update request handler configured by default.
XML Formatted Index Updates
Index update commands can be sent as XML message to the update handler using
Content-type: application/xml or
The XML schema recognized by the update handler for adding documents is very straightforward:
<add>element introduces one more documents to be added.
<doc>element introduces the fields making up a document.
<field>element presents the content for a specific field.
The add command supports some optional attributes which may be specified.
Add the document within the specified number of milliseconds
Default is true. Indicates if the unique key constraints should be checked to overwrite previous versions of the same document (see below)
If the document schema defines a unique key, then by default an
/update operation to add a document will overwrite (ie: replace) any document in the index with the same unique key. If no unique key has been defined, indexing performance is somewhat faster, as no check has to be made for an existing documents to replace.
If you have a unique key field, but you feel confident that you can safely bypass the uniqueness check (eg: you build your indexes in batch, and your indexing code guarantees it never adds the same document more then once) you can specify the
overwrite="false" option when adding your documents.
XML Update Commands
Commit and Optimize Operations
<commit> operation writes all documents loaded since the last commit to one or more segment files on the disk. Before a commit has been issued, newly indexed content is not visible to searches. The commit operation opens a new searcher, and triggers any event listeners that have been configured.
Commits may be issued explicitly with a
<commit/> message, and can also be triggered from
<autocommit> parameters in
<optimize> operation requests Solr to merge internal data structures in order to improve search performance. For a large index, optimization will take some time to complete, but by merging many small segment files into a larger one, search performance will improve. If you are using Solr's replication mechanism to distribute searches across many systems, be aware that after an optimize, a complete index will need to be transferred. In contrast, post-commit transfers are usually much smaller.
<optimize> elements accept these optional attributes:
Default is true. Blocks until a new searcher is opened and registered as the main query searcher, making the changes visible.
(commit only) Default is false. Merges segments that have more than 10% deleted docs, expunging them in the process.
|maxSegments||(optimize only) Default is 1. Merges the segments down to no more than this number of segments.|
Here are examples of <commit> and <optimize> using optional attributes:
Documents can be deleted from the index in two ways. "Delete by ID" deletes the document with the specified ID, and can be used only if a UniqueID field has been defined in the schema. "Delete by Query" deletes all documents matching a specified query, although
commitWithin is ignored for a Delete by Query. A single delete message can contain multiple delete operations.
scoreparameter with a value of "
none" to avoid a
ClassCastException. See the section on the Join Query Parser for more details on the
The rollback command rolls back all add and deletes made to the index since the last commit. It neither calls any event listeners nor creates a new searcher. Its syntax is simple:
curl to Perform Updates
You can use the
curl utility to perform any of the above commands, using its
--data-binary option to append the XML message to the
curl command, and generating a HTTP POST request. For example:
For posting XML messages contained in a file, you can use the alternative form:
Short requests can also be sent using a HTTP GET command, URL-encoding the request, as in the following. Note the escaping of "<" and ">":
Responses from Solr take the form shown here:
The status field will be non-zero in case of failure.
Using XSLT to Transform XML Index Updates
The UpdateRequestHandler allows you to index any arbitrary XML using the
<tr> parameter to apply an XSL transformation. You must have an XSLT stylesheet in the
conf/xslt directory of your config set that can transform the incoming data to the expected
<add><doc/></add> format, and use the
tr parameter to specify the name of that stylesheet.
Here is an example XSLT stylesheet:
This stylesheet transforms Solr's XML search result format into Solr's Update XML syntax. One example usage would be to copy a Solr 1.3 index (which does not have CSV response writer) into a format which can be indexed into another Solr file (provided that all fields are stored):
You can also use the stylesheet in
XsltUpdateRequestHandler to transform an index when updating:
JSON Formatted Index Updates
Solr can accept JSON that conforms to a defined structure, or can accept arbitrary JSON-formatted documents. If sending arbitrarily formatted JSON, there are some additional parameters that need to be sent with the update request, described below in the section Transforming and Indexing Custom JSON.
JSON formatted update requests may be sent to Solr's
/update handler using
Content-Type: application/json or
JSON formatted updates can take 3 basic forms, described in depth below:
- A single document to add, expressed as a top level JSON Object. To differentiate this from a set of commands, the
json.command=falserequest parameter is required.
- A list of documents to add, expressed as a top level JSON Array containing a JSON Object per document.
- A sequence of update commands, expressed as a top level JSON Object (aka: Map).
Adding a Single JSON Document
The simplest way to add Documents via JSON is to send each document individually as a JSON Object, using the
Adding Multiple JSON Documents
Adding multiple documents at one time via JSON can be done via a JSON Array of JSON Objects, where each object represents a document:
A sample JSON file is provided at
example/exampledocs/books.json and contains an array of objects that you can add to the Solr
Sending JSON Update Commands
In general, the JSON update syntax supports all of the update commands that the XML update handler supports, through a straightforward mapping. Multiple commands, adding and deleting documents, may be contained in one message:
Comments are not allowed in JSON, but duplicate names are.
The comments in the above example are for illustrative purposes only, and can not be included in actual commands sent to Solr.
As with other update handlers, parameters such as
overwrite may be specified in the URL instead of in the body of the message.
The JSON update format allows for a simple delete-by-id. The value of a
delete can be an array which contains a list of zero or more specific document id's (not a range) to be deleted. For example, a single document:
Or a list of document IDs:
The value of a "delete" can be an array which contains a list of zero or more id's to be deleted. It is not a range (start and end).
You can also specify
_version_ with each "delete":
You can specify the version of deletes in the body of the update request as well.
JSON Update Convenience Paths
In addition to the
/update handler, there are a few additional JSON specific request handler paths available by default in Solr, that implicitly override the behavior of some request parameters:
/update/json path may be useful for clients sending in JSON formatted update commands from applications where setting the Content-Type proves difficult, while the
/update/json/docs path can be particularly convenient for clients that always want to send in documents – either individually or as a list – without needing to worry about the full JSON command syntax.
Custom JSON Documents
Solr can support custom JSON. This is covered in the section Transforming and Indexing Custom JSON.
CSV Formatted Index Updates
CSV formatted update requests may be sent to Solr's
/update handler using
Content-Type: application/csv or
A sample CSV file is provided at
example/exampledocs/books.csv that you can use to add some documents to the Solr
CSV Update Parameters
The CSV handler allows the specification of many parameters in the URL in the form:
The table below describes the parameters for the update handler.
Global (g) or Per Field (f)
Character used as field separator; default is ","
g,(f: see split)
If true, remove leading and trailing whitespace from values. Default=false.
Set to true if first line of input contains field names. These will be used if the fieldnames parameter is absent.
Comma separated list of field names to use when adding documents.
A literal value for a specified field name.
Comma separated list of field names to skip.
Number of lines to discard in the input stream before the CSV data starts, including the header, if present. Default=0.
The character optionally used to surround values to preserve characters such as the CSV separator or whitespace. This standard CSV format handles the encapsulator itself appearing in an encapsulated value by doubling the encapsulator.
g,(f: see split)
The character used for escaping CSV separators or other reserved characters. If an escape is specified, the encapsulator is not used unless also explicitly specified since most formats use either encapsulation or escaping, not both
Keep and index zero length (empty) fields. Default=false.
Map one value to another. Format is value:replacement (which can be empty.)
If true, split a field into multiple values by a separate parser.
If true (the default), check for and overwrite duplicate documents, based on the uniqueKey field declared in the Solr schema. If you know the documents you are indexing do not contain any duplicates then you may see a considerable speed up setting this to false.
Issues a commit after the data has been ingested.
Add the document within the specified number of milliseconds.
Map the rowid (line number) to a field specified by the value of the parameter, for instance if your CSV doesn't have a unique key and you want to use the row id as such.
Add the given offset (as an int) to the rowid before adding it to the document. Default is 0
Indexing Tab-Delimited files
The same feature used to index CSV documents can also be easily used to index tab-delimited files (TSV files) and even handle backslash escaping rather than CSV encapsulation.
For example, one can dump a MySQL table to a tab delimited file with:
This file could then be imported into Solr by setting the
separator to tab (%09) and the
escape to backslash (%5c).
CSV Update Convenience Paths
In addition to the
/update handler, there is an additional CSV specific request handler path available by default in Solr, that implicitly override the behavior of some request parameters:
/update/csv path may be useful for clients sending in CSV formatted update commands from applications where setting the Content-Type proves difficult.
Nested Child Documents
Solr indexes nested documents in blocks as a way to model documents containing other documents, such as a blog post parent document and comments as child documents -- or products as parent documents and sizes, colors, or other variations as child documents. At query time, the Block Join Query Parsers can search these relationships. In terms of performance, indexing the relationships between documents may be more efficient than attempting to do joins only at query time, since the relationships are already stored in the index and do not need to be computed.
Nested documents may be indexed via either the XML or JSON data syntax (or using SolrJ) - but regardless of syntax, you must include a field that identifies the parent document as a parent; it can be any field that suits this purpose, and it will be used as input for the block join query parsers.
To support nested documents, the schema must include an indexed/non-stored field _root _ . The value of that field is populated automatically and is the same for all documents in the block, regardless of the inheritance depth.
For example, here are two documents and their child documents:
In this example, we have indexed the parent documents with the field
content_type, which has the value "parentDocument". We could have also used a boolean field, such as
isParent, with a value of "true", or any other similar approach.
This example is equivalent to the XML example above, note the special
_childDocuments_ key need to indicate the nested documents in JSON.
One limitation of indexing nested documents is that the whole block of parent-children documents must be updated together whenever any changes are required. In other words, even if a single child document or the parent document is changed, the whole block of parent-child documents must be indexed together.