Introduction

This article provides guidance on setting up Nutch to work with a HTTP/HTTPS proxy for use cases where Nutch is operated from behind a firewall.

For convenience this article features Tinyproxy; a light-weight HTTP/HTTPS proxy daemon for POSIX operating systems. 

Learning Outcomes

  1. Understand the base use case and architecture associated with integrating Nutch with a HTTP/HTTPS proxy.
  2. Use Tinyproxy with Nutch to satisfy the above use case.

Use Case

This use case assumes you are a Nutch administrator seeking to perform a fairly vanilla data acquisition task. By default, Nutch sends requests directly from the server's IP address to target websites. Over time, this may lead to several challenges: some sites may detect the high volume of requests as suspicious activity and block the server IP, halting the crawl prematurely. Additionally, certain resources are geo-restricted, meaning they're only accessible from specific regions, which limits the scope of data collection if the server is located elsewhere. Rate limiting imposed by websites further complicates matters, as exceeding request thresholds can result in temporary bans or distorted data due to incomplete fetches.

To address these issues, you decide to integrate a lightweight proxy e.g. Tinyproxy, into the Nutch deployment. By configuring Nutch to route all HTTP/HTTPS requests through the proxy (via settings in nutch-site.xml), you can achieve greater control and flexibility. The proxy acts as an intermediary, masking the origin IP and allowing for potential rotation across multiple proxies if needed, which reduces the risk of detection and blocking. This enables uninterrupted crawling even on sites with strict anti-bot measures.

Moreover, the proxy facilitates access to region-specific content by routing traffic through servers in different locations, expanding the breadth of data you can collect. It also supports basic throttling mechanisms, helping to spread requests and comply with website robots.txt policies, thereby promoting ethical crawling practices. With enhanced logging capabilities from the proxy, you can monitor traffic patterns, troubleshoot issues, and optimize the crawl configuration for better performance.

Deploying Nutch with a proxy empowers you to conduct more robust, scalable, and sustainable web crawling operations, ensuring reliable data acquisition while minimizing disruptions and respecting online ecosystems. This setup may save time and resources and enhances the overall quality and completeness of the indexed content.

Architecture

The following C4 Container diagram provides adequate system abstraction to contextualize the above use case.

In essence the Tinyproxy service becomes part of the larger crawling system.

Note, additional components such as indexing subsystem, etc. are not included in this particular diagram.


Install Tinyproxy

See the Downloads section of the Tinyproxy documentation for official guidance regarding installation on your taregt operation system. For convenience a couple of installation options are below.

# linux
apt-get install tinyproxy

# macos
brew install tinyproxy

Using Docker

Using Docker

The remainder of this documentation uses the ajoergensen/tinyproxy container image


docker pull ajoergensen/tinyproxy

Configure TinyProxy

Tinyproxy is configured via a file named tinyproxy.conf


touch tinyproxy.conf && vim tinyproxy.conf


Set values for the Port and Allow (here, using my localhost) options. N.B. Most of these configuration settings are default and can be easily altered to suit. The tinyproxy.conf configuration file permits a LogLevel Info configuration setting to ensure verbose information is available to assist debugging.


Port 8888
Allow 127.0.0.1

User nobody
Group nogroup
ViaProxyName "tinyproxy"
ConnectPort 443
ConnectPort 563
Timeout 600
DefaultErrorFile "/usr/share/tinyproxy/default.html"
StatFile "/usr/share/tinyproxy/stats.html"
Logfile "/var/log/tinyproxy/tinyproxy.log"
LogLevel Info
PidFile "/var/run/tinyproxy/tinyproxy.pid"
MaxClients 100
MinSpareServers 5
MaxSpareServers 20
StartServers 10
MaxRequestsPerChild 0

Start & Test Tinyproxy

Starting

# If you are NOT using the Docker container image then the process lifecycle can be managed as follows
/etc/init.d/tinyproxy stop
/etc/init.d/tinyproxy start
/etc/init.d/tinyproxy restart

# Using Docker
docker run --rm --name tinyproxy -p 8888:8888 -v "./tinyproxy.conf:/etc/tinyproxy.conf:ro" ajoergensen/tinyproxy

Test the proxy with your browser

Proxy testing will vary by OS and Web Browser. Consult the resources below for guidance.

More thorough testing can be conducted by exploring the Tinyproxy Filtering Policy

Configure Nutch

Copy the proxy configuration (see below) from conf/nutch-default.xml to conf/nutch-site.xml and fill up with the values of your proxy

<property>
  <name>http.proxy.host</name>
  <value>127.0.0.1</value>
  <description>The proxy hostname.  If empty, no proxy is used.</description>
</property>

<property>
  <name>http.proxy.port</name>
  <value>8888</value>
  <description>The proxy port.</description>
</property>

<property>
  <name>http.proxy.username</name>
  <value></value>
  <description>Username for proxy. This will be used by
  'protocol-httpclient', if the proxy server requests basic, digest
  and/or NTLM authentication. To use this, 'protocol-httpclient' must
  be present in the value of 'plugin.includes' property.
  NOTE: For NTLM authentication, do not prefix the username with the
  domain, i.e. 'susam' is correct whereas 'DOMAIN\susam' is incorrect.
  </description>
</property>

<property>
  <name>http.proxy.password</name>
  <value></value>
  <description>Password for proxy. This will be used by
  'protocol-httpclient', if the proxy server requests basic, digest
  and/or NTLM authentication. To use this, 'protocol-httpclient' must
  be present in the value of 'plugin.includes' property.
  </description>
</property>

<property>
  <name>http.proxy.realm</name>
  <value></value>
  <description>Authentication realm for proxy. Do not define a value
  if realm is not required or authentication should take place for any
  realm. NTLM does not use the notion of realms. Specify the domain name
  of NTLM authentication as the value for this property. To use this,
  'protocol-httpclient' must be present in the value of
  'plugin.includes' property.
  </description>
</property>

<property>
  <name>http.proxy.type</name>
  <value>HTTP</value>
  <description>
    Proxy type: HTTP or SOCKS (cf. java.net.Proxy.Type).
    Note: supported by protocol-okhttp.
  </description>
</property>

<property>
  <name>http.proxy.exception.list</name>
  <value></value>
  <description>A comma separated list of hosts that don't use the proxy
  (e.g. intranets). Example: www.apache.org</description>
</property>


Now if you crawl sites, Nutch will use your proxy. You can monitor it by looking at the logs of Tinyproxy during a crawl.


tail -f /var/log/tinyproxy.log

Stretch Use Case

TInyproxy Filtering Policy

An extension to the base use case may be to establish a filtering policy which can be configured via Tinyproxy's FilterDefaultDeny configuration setting. The default filtering policy is to allow everything that is not matched by a filtering rule e.g. a deny list.

By experimenting with this configuration property we can easily change the default policy of the filtering system. If this directive is commented out, or is set to "No" then the default policy is to allow everything which is not specifically denied by the filter file. However, by setting this directive to "Yes" the default policy becomes to deny everything which is not specifically allowed by the filter file e.g. the inverse.

Tinyproxy supports filtering of web sites based on URLs or domains. We need to specify the location of a text file containing the filter rules, one rule per line. This can be done as follows

vi /etc/filter

and add site urls to be blocked. The list should comprise of single URLs, one per line, just like the seed list for performing crawls.

google.com
apache.org

You can then augment the Tinyproxy configuration to accommodate this additional use case.

Port 8888
Allow 127.0.0.1
Filter "/etc/filter"
FilterURLs On
FilterDefaultDeny No #filters will act as a blacklist

User nobody
Group nogroup
ViaProxyName "tinyproxy"
ConnectPort 443
ConnectPort 563
Timeout 600
DefaultErrorFile "/usr/share/tinyproxy/default.html"
StatFile "/usr/share/tinyproxy/stats.html"
Logfile "/var/log/tinyproxy/tinyproxy.log"
LogLevel Info
PidFile "/var/run/tinyproxy/tinyproxy.pid"
MaxClients 100
MinSpareServers 5
MaxSpareServers 20
StartServers 10
MaxRequestsPerChild 0
  • No labels