This page is dedicated to Scoring implementations within Apache Nutch. The language used within this document is intended to reflect that used within the Nutch community and vocabulary may vary from time to time, words may be used interchangeably to refer to the same thing, etc. If you feel there is a discrepancy with this document then please let us know by contacting us.
Amongst other things Apache Nutch is pluggable and modular with extensible interfaces. Parse, Index and ScoringFilter can all use custom implementations. This document explains the basics of scoring in Apache Nutch, including information on:
A scoring filter will manipulate scoring variables in CrawlDatum and in resulting search indexes. Filters can be chained in a specific order, to provide multi-stage scoring adjustments.
Scoring occurs in numerous places throughout the Nutch codebase and consequently within the crawl cycle. This section describes the point of occurence and functional purpose Scoring serves at each step. You will see that the list of elements has been structured to represent the logical and typical progression of a Nutch crawl cycle.
db.score.injected
in this case as this assigns a default of 1.0f
against the score of new pages added by the injector. This default score can however be overridden by associating the nutch.score
metadata flag against any URL in a seed list. This allows to set a custom score for a specific URL. If this is the case we assign this score to the CrawlDatum object, if not then we use the default score as described above.db.score.injected
configuration property present within nutch-site.xml
as populated in the initial sortJob execution as described above. This value represents the score of new pages added by the Injector. This is relevant for us as we must know if a record already exists and we wish to update but not overwrite the value.Generator.Selector.generatorSortValue
method which prepares a sort value for sorting and selecting the top N scoring pages during fetchlist generation. The arguments for Hadoop are: Text key url
representing the url of the page we are trying to score, CrawlDatum datum
which represents the page's datum which should not be modified in this task, and an initial sort value initSort
of 1.0f
. It should be noted that the final value doesn't always need to be set to 1.0f
as it can be linked to a value from previous filters in the chain of Scoring implementations. The result of executing the Generator.Selector.generatorSortValue
function is subsequently used to consider only entries with a score superior to the threshold. Only URL entries above the threshold will then be fetched.FixingOpicScoring - In planning.