Apache Solr Documentation

6.5 Ref Guide (PDF Download)
Solr Tutorial
Solr Community Wiki

Older Versions of this Guide (PDF)

6.6 Draft Ref Guide Topics

Meta-Documentation

This Unreleased Guide Covers Apache Solr 6.6
As of 5 May 2017, committer editing of this Ref Guide is LOCKED for migration to the new Ref Guide. See SOLR-10290 for details.

Skip to end of metadata
Go to start of metadata

Function queries enable you to generate a relevancy score using the actual value of one or more numeric fields. Function queries are supported by the DisMax, Extended DisMax, and standard query parsers.

Function queries use functions. The functions can be a constant (numeric or string literal), a field, another function or a parameter substitution argument. You can use these functions to modify the ranking of results for users. These could be used to change the ranking of results based on a user's location, or some other calculation.

Function query topics covered in this section:

Using Function Query

Functions must be expressed as function calls (for example, sum(a,b) instead of simply a+b).

There are several ways of using function queries in a Solr query:

  • Via an explicit QParser that expects function arguments, such  func or frange . For example:

  • In a Sort expression. For example:

  • Add the results of functions as pseudo-fields to documents in query results. For instance, for:

    the output would be:

  • Use in a parameter that is explicitly for specifying functions, such as the EDisMax query parser's boost param, or DisMax query parser's bf (boost function) parameter. (Note that the bf parameter actually takes a list of function queries separated by white space and each with an optional boost. Make sure you eliminate any internal white space in single function queries when using bf). For example:

  • Introduce a function query inline in the lucene QParser with the _val_ keyword. For example:

Only functions with fast random access are recommended.

Available Functions

The table below summarizes the functions available for function queries.

Function

Description

Syntax Examples

abs

Returns the absolute value of the specified value or function.

abs(x)
abs(-5)

"constant"

Specifies a floating point constant.

1.5

def

def is short for default. Returns the value of field "field", or if the field does not exist, returns the default value specified. and yields the first value where exists()==true.)

def(rating,5): This def() function returns the rating, or if no rating specified in the doc, returns 5
def(myfield, 1.0): equivalent to if(exists(myfield),myfield,1.0)

div

Divides one value or function by another. div(x,y) divides x by y.

div(1,y)
div(sum(x,100),max(y,1))

dist

Return the distance between two vectors (points) in an n-dimensional space. Takes in the power, plus two or more ValueSource instances and calculates the distances between the two vectors. Each ValueSource must be a number. There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector.

dist(2, x, y, 0, 0): calculates the Euclidean distance between (0,0) and (x,y) for each document

dist(1, x, y, 0, 0): calculates the Manhattan (taxicab) distance between (0,0) and (x,y) for each document

dist(2, x,y,z,0,0,0): Euclidean distance between (0,0,0) and (x,y,z) for each document.

dist(1,x,y,z,e,f,g): Manhattan distance between (x,y,z) and (e,f,g) where each letter is a field name

docfreq(field,val)

Returns the number of documents that contain the term in the field. This is a constant (the same value for all documents in the index).

You can quote the term if it's more complex, or do parameter substitution for the term value.

docfreq(text,'solr')

...&defType=func
&q=docfreq(text,$myterm)
&myterm=solr

field

Returns the numeric docValues or indexed value of the field with the specified name.  In its simplest (single argument) form, this function can only be used on single valued fields, and can be called using the name of the field as a string, or for most conventional field names simply use the field name by itself with out using the field(...) syntax.

When using docValues, an optional 2nd argument can be specified to select the "min" or "max" value of multivalued fields.

0 is returned for documents without a value in the field.

These 3 examples are all equivalent:

  • myFloatFieldName
  • field(myFloatFieldName)
  • field("myFloatFieldName")

The last form is convinient when your field name is atypical:

  • field("my complex float fieldName")

For multivalued docValues fields:

  • field(myMultiValuedFloatField,min)
  • field(myMultiValuedFloatField,max)

hsin

The Haversine distance calculates the distance between two points on a sphere when traveling along the sphere. The values must be in radians. hsin also take a Boolean argument to specify whether the function should convert its output to radians.

hsin(2, true, x, y, 0, 0)

idf

Inverse document frequency; a measure of whether the term is common or rare across all documents. Obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient. See also tf.

idf(fieldName,'solr'): measures the inverse of the frequency of the occurrence of the term 'solr' in fieldName.

if

Enables conditional function queries.  In if(test,value1,value2):

  • test is or refers to a logical value or expression that returns a logical value (TRUE or FALSE).
  • value1 is the value that is returned by the function if test yields TRUE.
  • value2 is the value that is returned by the function if test yields FALSE.

An expression can be any function which outputs boolean values, or even functions returning numeric values, in which case value 0 will be interpreted as false, or strings, in which case empty string is interpreted as false.

if(termfreq (cat,'electronics'), popularity,42) :
This function checks each document for the to see if it contains the term "electronics" in the cat field. If it does, then the value of the popularity field is returned, otherwise the value of 42 is returned.

linear

Implements m*x+c where m and c are constants and x is an arbitrary function. This is equivalent to sum(product(m,x),c), but slightly more efficient as it is implemented as a single function.

linear(x,m,c)
linear(x,2,4) returns 2*x+4

log

Returns the log base 10 of the specified function.

log(x)

log(sum(x,100))

map

Maps any values of an input function x that fall within min and max inclusive to the specified target. The arguments min and max must be constants. The arguments target and default can be constants or functions. If the value of x does not fall between min and max, then either the value of x is returned, or a default value is returned if specified as a 5th argument.

map(x,min,max,target)
map(x,0,0,1) - changes any values of 0 to 1. This can be useful in handling default 0 values.

map(x,min,max,target,default)
map(x,0,100,1,-1) - changes any values between 0 and 100 to 1, and all other values to -1.

map(x,0,100, sum(x,599), docfreq(text,solr)) - changes any values between 0 and 100 to x+599, and all other values to frequency of the term 'solr' in the field text.

max

Returns the maximum numeric value of multiple nested functions or constants, which are specified as arguments: max(x,y,...). The max function can also be useful for "bottoming out" another function or field at some specified constant.

(Use the field(myfield,max) syntax for selecting the maximum value of a single multivalued field)

max(myfield,myotherfield,0)

maxdoc

Returns the number of documents in the index, including those that are marked as deleted but have not yet been purged. This is a constant (the same value for all documents in the index).

maxdoc()

min

Returns the minimum numeric value of multiple nested functions of constants, which are specified as arguments: min(x,y,...). The min function can also be useful for providing an "upper bound" on a function using a constant.

(Use the field(myfield,min) syntax for selecting the minimum value of a single multivalued field)

min(myfield,myotherfield,0)

ms

Returns milliseconds of difference between its arguments. Dates are relative to the Unix or POSIX time epoch, midnight, January 1, 1970 UTC. Arguments may be the name of an indexed TrieDateField, or date math based on a constant date or NOW .

  • ms(): Equivalent to ms(NOW), number of milliseconds since the epoch.
  • ms(a): Returns the number of milliseconds since the epoch that the argument represents.
  • ms(a,b) : Returns the number of milliseconds that b occurs before a (that is, a - b)

ms(NOW/DAY)
ms(2000-01-01T00:00:00Z)
ms(mydatefield)
ms(NOW,mydatefield)
ms(mydatefield, 2000-01-01T00:00:00Z)
ms(datefield1, datefield2)

norm(field)

Returns the "norm" stored in the index for the specified field.  This is the product of the index time boost and the length normalization factor, according to the Similarity for the field.

norm(fieldName)

numdocs

Returns the number of documents in the index, not including those that are marked as deleted but have not yet been purged. This is a constant (the same value for all documents in the index).

numdocs()

ord

Returns the ordinal of the indexed field value within the indexed list of terms for that field in Lucene index order (lexicographically ordered by unicode value), starting at 1. In other words, for a given field, all values are ordered lexicographically; this function then returns the offset of a particular value in that ordering. The field must have a maximum of one value per document (not multi-valued). 0 is returned for documents without a value in the field.

ord() depends on the position in an index and can change when other documents are inserted or deleted.

See also rord below.

ord(myIndexedField)

Example: If there were only three values ("apple","banana","pear") for a particular field X, then: ord(X) would be 1 for documents containing "apple", 2 for documnts containing "banana", etc...

payload

Returns the float value computed from the decoded payloads of the term specified. The return value is computed using the min, max, or average of the decoded payloads. A special first function can be used instead of the others, to short-circuit term enumeration and return only the decoded payload of the first term. The field specified must have float or integer payload encoding capability (via DelimitedPayloadTokenFilter or NumericPayloadTokenFilter). If no payload is found for the term, the default value is returned.

  • payload(field_name,term): default value is 0.0, `average` function is used.
  • payload(field_name,term,default_value): default value can be a constant, field name, or another float returning function. `average` function used.
  • payload(field_name,term,default_value,function): function values can be `min`, `max`, `average`, or `first`.

 

payload(payloaded_field_dpf,term,0.0,first)
pow

Raises the specified base to the specified power. pow(x,y) raises x to the power of y.

pow(x,y)
pow(x,log(y))
pow(x,0.5): the same as sqrt

product

Returns the product of multiple values or functions, which are specified in a comma-separated list.  mul(...) may also be used as an alias for this function.

product(x,y,...)
product(x,2)
product(x,y)
mul(x,y)

query

Returns the score for the given subquery, or the default value for documents not matching the query. Any type of subquery is supported through either parameter de-referencing $otherparam or direct specification of the query string in the Local Parameters through the v key.

query(subquery, default)

q=product (popularity, query({!dismax v='solr rocks'}): returns the product of the popularity and the score of the DisMax query.

q=product (popularity, query($qq))&qq={!dismax}solr rocks: equivalent to the previous query, using parameter de-referencing.

q=product (popularity, query($qq,0.1)) &qq={!dismax} solr rocks: specifies a default score of 0.1 for documents that don't match the DisMax query.

recip

Performs a reciprocal function with recip(x,m,a,b) implementing a/(m*x+b) where m,a,b are constants, and x is any arbitrarily complex function.

When a and b are equal, and x>=0, this function has a maximum value of 1 that drops as x increases. Increasing the value of a and b together results in a movement of the entire function to a flatter part of the curve. These properties can make this an ideal function for boosting more recent documents when x is rord(datefield).

recip(myfield,m,a,b)
recip(rord (creationDate), 1,1000,1000)

rord

Returns the reverse ordering of that returned by ord.

rord(myDateField)

scale

Scales values of the function x such that they fall between the specified minTarget and maxTarget inclusive. The current implementation traverses all of the function values to obtain the min and max, so it can pick the correct scale.

The current implementation cannot distinguish when documents have been deleted or documents that have no value. It uses 0.0 values for these cases. This means that if values are normally all greater than 0.0, one can still end up with 0.0 as the min value to map from. In these cases, an appropriate map() function could be used as a workaround to change 0.0 to a value in the real range, as shown here:
scale(map(x,0,0,5),1,2)

scale(x, minTarget, maxTarget)
scale(x,1,2): scales the values of x such that all values will be between 1 and 2 inclusive.

sqedist

The Square Euclidean distance calculates the 2-norm (Euclidean distance) but does not take the square root, thus saving a fairly expensive operation. It is often the case that applications that care about Euclidean distance do not need the actual distance, but instead can use the square of the distance. There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector.

sqedist(x_td, y_td, 0, 0)

sqrt

Returns the square root of the specified value or function.

sqrt(x)sqrt(100)sqrt(sum(x,100))

strdist

Calculate the distance between two strings. Uses the Lucene spell checker StringDistance interface and supports all of the implementations available in that package, plus allows applications to plug in their own via Solr's resource loading capabilities. strdist takes (string1, string2, distance measure). Possible values for distance measure are:

jw: Jaro-Winkler

edit: Levenstein or Edit distance

ngram: The NGramDistance, if specified, can optionally pass in the ngram size too. Default is 2.

FQN: Fully Qualified class Name for an implementation of the StringDistance interface. Must have a no-arg constructor.

strdist("SOLR",id,edit)

sub

Returns x-y from sub(x,y).

sub(myfield,myfield2)
sub(100, sqrt(myfield))

sum

Returns the sum of multiple values or functions, which are specified in a comma-separated list. add(...) may be used as an alias for this function

sum(x,y,...) sum(x,1)
sum(x,y)
sum(sqrt(x),log(y),z,0.5)
add(x,y)

sumtotaltermfreq

Returns the sum of totaltermfreq values for all terms in the field in the entire index (i.e., the number of indexed tokens for that field). (Aliases sumtotaltermfreq to sttf.)

If doc1:(fieldX:A B C) and doc2:(fieldX:A A A A):
docFreq(fieldX:A) = 2 (A appears in 2 docs)
freq(doc1, fieldX:A) = 4  (A appears 4 times in doc 2)
totalTermFreq(fieldX:A) = 5  (A appears 5 times across all docs)
sumTotalTermFreq(fieldX) = 7  in fieldX, there are 5 As, 1 B, 1 C

termfreq

Returns the number of times the term appears in the field for that document.

termfreq(text,'memory')

tf

Term frequency; returns the term frequency factor for the given term, using the Similarity for the field. The tf-idf value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the document, which helps to control for the fact that some words are generally more common than others.  See also idf.

tf(text,'solr')

top

Causes the function query argument to derive its values from the top-level IndexReader containing all parts of an index. For example, the ordinal of a value in a single segment will be different from the ordinal of that same value in the complete index.

The ord() and rord() functions implicitly use top(), and hence ord(foo) is equivalent to top(ord(foo)).

 

totaltermfreq

Returns the number of times the term appears in the field in the entire index. (Aliases totaltermfreq to ttf.)

ttf(text,'memory')

The following functions are boolean – they return true or false.  They are mostly useful as the first argument of the if function, and some of these can be combined. If used somewhere else, it will yield a '1' or '0'.

FunctionDescriptionSyntax Examples

and

Returns a value of true if and only if all of its operands evaluate to true.

and(not (exists (popularity)), exists (price)): returns true for any document which has a value in the price field, but does not have a value in the popularity field

or

A logical disjunction.

or(value1,value2): TRUE if either value1 or value2 is true.

xor

Logical exclusive disjunction, or one or the other but not both.

xor(field1,field2) returns TRUE if either field1 or field2 is true; FALSE if both are true.

not

The logically negated value of the wrapped function.

not(exists(author)): TRUE only when exists(author) is false.

exists

Returns TRUE if any member of the field exists.

exists(author) returns TRUE for any document has a value in the "author" field.

exists(query(price:5.00)) returns TRUE if "price" matches "5.00".

gt, gte, lt, lte, eq5 comparison functions: Greater Than, Greater Than or Equal, Less Than, Less Than or Equal, Equalif(lt(ms(mydatefield),315569259747),0.8,1) translates to this pseudocode: if mydatefield < 315569259747 then 0.8 else 1

 

Example Function Queries

To give you a better understanding of how function queries can be used in Solr, suppose an index stores the dimensions in meters x,y,z of some hypothetical boxes with arbitrary names stored in field boxname. Suppose we want to search for box matching name findbox but ranked according to volumes of boxes. The query parameters would be:

q=boxname:findbox _val_:"product(x,y,z)"

This query will rank the results based on volumes. In order to get the computed volume, you will need to request the score, which will contain the resultant volume:

&fl=*, score

Suppose that you also have a field storing the weight of the box as weight. To sort by the density of the box and return the value of the density in score, you would submit the following query:

Sort By Function

You can sort your query results by the output of a function. For example, to sort results by distance, you could enter:

Sort by function also supports pseudo-fields: fields can be generated dynamically and return results as though it was normal field in the index. For example,

&fl=id,sum(x, y),score

would return:

Related Topics

  • No labels

12 Comments

  1. Typo (unbalanced quotes and missing space) in example function queries section. 

    q=boxname:findbox_val_:"product(product(x,y),z) => q=boxname:findbox _val_:"product(product(x,y),z)"

    http://localhost:8983/solr/select/?q=boxname:findbox_val_div(weight,product(product(x,y),z))" => http://localhost:8983/solr/select/?q=boxname:findbox _val_:"div(weight,product(product(x,y),z))"

     

  2. The syntax given IF statement is incorrect. The '==' operator is not valid here. Discussed with hoss on IRC. 

    "==" is't valid syntax in the if ... you need to use a nested function that returns a boolean (or an integer where 0 means false)

  3. Hmmm... "vector()" doesn't seem to be documented here ... need to add that, and while at it: audit the full list of documented functions compared to the list of default ValueSourceParsers.

  4. Is this page missing the 'mod' function?

    1. Yes, mod is missing.  Also absent:

      literal (though it is mentioned in the context of the first paragraph as string literals being a type of constant) ,

      currency (may be premature since its implementation is Solr 4.2)

      min (though 'max' is present)

  5. I think all of the Math.* ones are missing, no?  Also, in addition to hsin, there is ghhsin (geohash based haversine).  Also missing is the geohash method.

  6. Should the 4th distance function example be Manhattan?

    dist(1,x,y,z,e,f,g): Euclidean distance between (x,y,z) and (e,f,g) where each letter is a field name

  7. Missing parenthesis in the syntax example of the boolean functions gt, gte, lt, lte, eq

    if(lt(ms(mydatefield,315569259747),0.8,1)  >>>  if(lt(ms(mydatefield),315569259747),0.8,1)
    1. Fixed; thanks for reporting.