This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • Open Relevance Viewer

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0

Crowd Sourcing

Wiki MarkupCrowdsourcing is the act of outsourcing tasks, traditionally performed by an employee or contractor, to a large group of people or community (a crowd), through an open call \[[Reference\]|].


To develop and maintain a reusable software application that can be used for crowdsourcing. The software will be compromized of several key sub systems:


Importing data will allow new material such as corpora and the associated annotation sets to be used on the corpora. Possible types of corpora can be text, image, or video based.

Import Schemas

The following section describe the type of default formats for importing data into the OPV. Click here

Exporting Data


Judging Modules

At this time the Judging module is required to present the following opptionsjudment options:

  • Relevant
  • Not Relevant
  • Skip This

This set of choices is available for each result returned from a query.

Data Archive

The data archive should contain the following information:

  • Unique Query Identifier
  • Result set based on query
    • Result set should contain a URI for the result
  • Judgement set based on result set that was judged
  • Metrics
    • precision
    • recall
  • User UUID that ties the usage to a registered user or an anonymous user.

Data Visualization

Nothing planned at this time.

Server System Requirements

The software should be run on most operating systems.

Client System Requirements

The software should run inside an end user's web browser. Later versions of the software should support an API for submitting queries, downloading results, submitting judgements and exporting past query/result/judgement sets.

Non-Functional Requirements

Related Materials

TREC Format