Page History

...

Bootstrapping from DMOZ

...

The injector adds URLs to the crawldb. Let's inject URLs from the DMOZ Open Directory. First we must download and uncompress the file listing all of the DMOZ pages. (This is a 200+ MB file, so this will take a few minutes.)

...

Next we select a random subset of these pages. (We use a random subset so that everyone who runs this tutorial doesn't hammer the same sites.) DMOZ contains around three million URLs. We select one out of every 5,000, so that we end up with around 1,000 URLs:

...

The parser also takes a few minutes, as it must parse the full file. Finally, we initialize the crawldb with the selected URLs.

...

Now we have a Web database with around 1,000 as-yet unfetched URLs in it.

...

• download binary file from here
• unzip to $HOME/apache-solr, we will now refer to this as ${APACHE_SOLR_HOME}

• create resources for a new "nutch" Solr core

  No Format
  mkdir -p ${APACHE_SOLR_HOME}/server/solr/configsets/nutch/ cp -r ${APACHE_SOLR_HOME}/server/solr/configsets/_default/* ${APACHE_SOLR_HOME}/server/solr/configsets/nutch/

  
  

• copy the Nutch's schema.xml into the Solr conf directory

  • (Nutch 1.15 or prior) copy the schema.xml from the conf/ directory:

    No Format
    cp ${NUTCH_RUNTIME_HOME}/conf/schema.xml ${APACHE_SOLR_HOME}/server/solr/configsets/nutch/conf/

    
    

  • (Nutch 1.16) copy the schema.xml from the indexer-solr source folder (source package):

    No Format
    cp .../src/plugin/indexer-solr/schema.xml ${APACHE_SOLR_HOME}/server/solr/configsets/nutch/conf/

    Note: due to NUTCH-2745 the schema.xml is not contained in the binary package. Please download the schema.xml from the source repository.

  • You may also try to use the most recent schema.xml in case of issues launching Solr with this schema.

...