This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0


  • Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions
    • The correct answers for each search are known ahead of time
    • Humans judges often decide these correct answers, stored as Relevancy Assertions
    • Can be labor intensive to setup
    • ... continued on the Relevancy Assertion Testing page
  • AB Testing / User Preference
    • Tracks explicit or implicit preferences between engines A/B
    • Often dispenses with the notion of the "correct" answer
    • Can be easier to setup, but some fear the best answers will be missed by both engines
    • continued on the AB Testing page

Beyond Precision and Recall: How Engines are Judged


  • Comparing TREC to ORP
    • TREC Code
      • Written in C
      • License Restrictions
    • ORP Code
      • Java
      • Apache License (still some restrictions)
  • Interface between ORP and Search Engines
    • Search Engines generally don't support native TREC/ORP formats
      • Each engine would need adapters
      • OR ORP could interface with native Search Engines' APIs
      • OR Use some intermediate format like OpenSearch or other Federated Search technique
    • maybe Lucene Benchmark Package
  • Implementing Judgments
    • Grid Methods, borrow from TREC?
      • Classic Precision and Recall
      • MRR, MAP, BPref, NDGCGNDCG
    • AB Methods
      • Predictive? Constant mixing, TODO: needs expansion