Overview

Apache Nutch can store its crawl data (CrawlDb, LinkDb, segments, WebGraph, HostDb) in Apache Ozone using the ofs:// Hadoop-compatible filesystem scheme. Because Nutch uses only the generic Hadoop FileSystem API throughout its codebase, no source code changes are required -- the integration is entirely configuration and classpath management.

This page covers prerequisites, configuration, running Nutch against Ozone, and important considerations.

Compatibility

Nutch interacts with storage exclusively through org.apache.hadoop.fs.FileSystem and org.apache.hadoop.fs.Path. It does not use any HDFS-specific classes such as DistributedFileSystem. Every FileSystem operation Nutch uses is supported by Ozone's ofs implementation:

Nutch does not use any of the unsupported ofs operations (append, setPermission, setOwner, setReplication, truncate, concat, createSymlink, xattrs, or ACLs).

For the full API compatibility matrix, see the Ozone ofs migration guide.

Why ofs:// Instead of o3fs://

Ozone offers two Hadoop-compatible schemes:

• o3fs:// -- operates on a single bucket only; the bucket and volume are encoded in the URI authority.
• ofs:// -- provides a full rooted view across all volumes and buckets; the URI authority is the Ozone Manager host or HA service ID.

ofs:// is recommended because it allows Nutch to access data across volumes and buckets without reconfiguring fs.defaultFS, and it simplifies path management.

Prerequisites

• A running Apache Ozone cluster (version 1.4.x or later recommended; 2.x preferred).
• The ozone-filesystem-hadoop3 JAR matching your Ozone version.
• Apache Nutch 1.x (uses Hadoop 3.4.2).
• Ozone Manager hostname or HA service ID.

Setup

1. Create the Ozone Volume and Bucket

An Ozone administrator must create a volume and bucket before Nutch can write data:

ozone sh volume create nutch
ozone sh bucket create /nutch/crawl --layout FILE_SYSTEM_OPTIMIZED

The FILE_SYSTEM_OPTIMIZED (FSO) layout is strongly recommended because it supports atomic directory renames, which Nutch relies on for safe database updates (see Considerations below).

2. Add the Ozone Filesystem JAR to the Classpath

The Ozone filesystem adapter JAR must be available at runtime. Choose one of these methods:

Option A -- Environment variable (recommended for cluster deployments):

export HADOOP_CLASSPATH=/opt/ozone/share/ozone/lib/ozone-filesystem-hadoop3-*.jar:$HADOOP_CLASSPATH

Option B -- Copy into Nutch lib directory (standalone deployments):

cp /opt/ozone/share/ozone/lib/ozone-filesystem-hadoop3-*.jar $NUTCH_HOME/lib/

Option C -- Add as an Ivy dependency (build-time inclusion):

Add to ivy/ivy.xml inside the <dependencies> block, after the Hadoop dependencies:

<!-- Optional: Apache Ozone filesystem support -->
<dependency org="org.apache.ozone" name="ozone-filesystem-hadoop3"
            rev="2.0.0" conf="*->default">
    <exclude org="org.slf4j" name="*"/>
</dependency>

Then rebuild: ant clean runtime.

3. Configure Hadoop for the ofs:// Scheme

Create or edit $NUTCH_HOME/conf/core-site.xml:

<?xml version="1.0"?>
<configuration>

  <!-- Register the ofs:// filesystem implementation -->
  <property>
    <name>fs.ofs.impl</name>
    <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>
  </property>

  <!-- Point all Nutch I/O at the Ozone cluster -->
  <property>
    <name>fs.defaultFS</name>
    <value>ofs://om-host.example.com/</value>
  </property>

</configuration>

For Ozone Manager HA, use the service ID:

<property>
  <name>fs.defaultFS</name>
  <value>ofs://omservice/</value>
</property>

When using an HA service ID, you must also provide the OM address configuration (typically via ozone-site.xml on the classpath).

Running Nutch with Ozone

Option A -- Ozone as the Default Filesystem

With fs.defaultFS set to ofs://, all Nutch commands work as normal. Paths are relative to the Ozone root:

# Inject seed URLs
nutch inject /nutch/crawl/crawldb /nutch/crawl/seed

# Generate a fetch list
nutch generate /nutch/crawl/crawldb /nutch/crawl/segments

# Fetch
nutch fetch /nutch/crawl/segments/<timestamp>

# Parse
nutch parse /nutch/crawl/segments/<timestamp>

# Update CrawlDb
nutch updatedb /nutch/crawl/crawldb /nutch/crawl/segments/<timestamp>

All data lands under ofs://omservice/nutch/crawl/:

ofs://omservice/nutch/crawl/
  crawldb/
    current/        (MapFiles -- the active CrawlDb)
    .locked         (lock file during updates)
  linkdb/
    current/        (MapFiles -- the active LinkDb)
  segments/
    20260217120000/ (one directory per fetch round)
      crawl_generate/
      crawl_fetch/
      content/
      parse_data/
      parse_text/
  webgraph/
    nodedb/
    inlinkdb/
    outlinkdb/

Option B -- Explicit ofs:// Paths (Dual HDFS/Ozone Operation)

To keep HDFS as the default and selectively store data in Ozone, omit the fs.defaultFS change and pass ofs:// paths directly:

nutch inject ofs://omservice/nutch/crawl/crawldb ofs://omservice/nutch/crawl/seed
nutch generate ofs://omservice/nutch/crawl/crawldb ofs://omservice/nutch/crawl/segments

This allows a single Nutch installation to read/write both HDFS and Ozone.

Option C -- Nutch crawl Script

The bin/crawl convenience script accepts a base directory. Point it at the Ozone path:

bin/crawl -i -s /nutch/crawl/seed /nutch/crawl 3

With fs.defaultFS set to ofs://, this stores everything in Ozone.

Considerations

Atomic Renames and FSO Buckets

Nutch uses FileSystem.rename() to atomically swap database versions (e.g., replacing crawldb/current with new output). In Ozone:

• FSO (File System Optimized) buckets perform renames as atomic metadata operations -- matching HDFS behavior. This is the recommended bucket layout for Nutch.
• Legacy (non-FSO) buckets perform renames by copying each key individually, which is non-atomic and slower.

Always create buckets with --layout FILE_SYSTEM_OPTIMIZED.

Cross-Bucket Rename Limitation

Ozone does not support renaming files across different buckets. Nutch's rename operations (in FSUtils.replace(), LinkDb, CrawlDb, etc.) rename paths within the same parent directory, so they naturally stay within a single bucket. This is not an issue as long as all Nutch data resides in one bucket.

MapReduce Execution

When running Nutch MapReduce jobs on a YARN cluster, every node must have the Ozone filesystem JAR on its classpath. Options:

• Set HADOOP_CLASSPATH cluster-wide.
• Place the JAR in Hadoop's share/hadoop/common/lib/ on all nodes.
• Use the -libjars option when submitting jobs (Nutch's NutchJob class supports this via Hadoop's ToolRunner).

Compression

Nutch's MapFile and SequenceFile writers use the cluster's default compression settings. Ozone supports the same compression codecs as HDFS (Snappy, LZ4, Zstandard, etc.) as long as the native libraries are available.

Trash

If Hadoop trash is enabled (fs.trash.interval > 0), Ozone moves deleted keys to a .Trash directory within the same bucket. Configure the Ozone-specific trash policy:

<property>
  <name>fs.trash.classname</name>
  <value>org.apache.hadoop.fs.ozone.OzoneTrashPolicy</value>
</property>

No Erasure Coding via FileSystem API

Unlike HDFS, Ozone's erasure coding is configured at the bucket level, not through FileSystem API calls. Nutch does not call erasure coding APIs, so this is not an issue -- but be aware that the replication strategy is controlled by the Ozone bucket configuration rather than per-file settings.

Verifying the Setup

After configuration, verify connectivity:

# List the Ozone root (should show volumes)
hdfs dfs -ls ofs://omservice/

# List the Nutch bucket
hdfs dfs -ls ofs://omservice/nutch/crawl/

# Write a test file
hdfs dfs -put /etc/hosts ofs://omservice/nutch/crawl/test-file
hdfs dfs -cat ofs://omservice/nutch/crawl/test-file
hdfs dfs -rm ofs://omservice/nutch/crawl/test-file

Then run a small crawl to validate end-to-end:

# Create a seed file
mkdir -p /tmp/nutch-seed
echo "https://nutch.apache.org/" > /tmp/nutch-seed/urls.txt
hdfs dfs -mkdir -p ofs://omservice/nutch/crawl/seed
hdfs dfs -put /tmp/nutch-seed/urls.txt ofs://omservice/nutch/crawl/seed/

# Run inject + generate to verify writes
nutch inject ofs://omservice/nutch/crawl/crawldb ofs://omservice/nutch/crawl/seed
nutch generate ofs://omservice/nutch/crawl/crawldb ofs://omservice/nutch/crawl/segments

# Confirm data was written
hdfs dfs -ls -R ofs://omservice/nutch/crawl/crawldb/

References

• Apache Ozone ofs Documentation
• Ozone ofs Migration from HDFS
• Ozone ofs Supported FileSystem APIs
• Apache Ozone Downloads
• Apache Nutch Wiki