Child pages
  • TestPlan1.5.1
Skip to end of metadata
Go to start of metadata

Test Plan for Apache OpenNLP 1.5.1

This page contains the test plan for the 1.5.1 release.

The 1.5.1 release does not introduce any changes to the feature
generation (expect OPENNLP-138) and should produce exactly the
same output as the 1.5.0 release.

Compatibility Test with OpenNLP 1.5.0 SourceForge Models

The 1.5.0 SourceForge models must be fully compatible with the 1.5.1
release. In this test all the English models are tested for compatibilty
on the English 300k sentences Leipzig Corpus. It is tested that
the output produced with the same model by both versions has the same md5 hash.

Component

Model

Perf 1.5.0

Perf 1.5.1

Tester

Passed

Comment

Sentence Detector

en-sent.bin

42565.4 sent/s

42186.7 sent/s

joern

yes

 

Tokenizer

en-token.bin

3059.5 sent/s

3091.8 sent/s

joern

yes

 

Name Finder

en-ner-person.bin

290.7 sent/s

487.1 sent/s

joern

no

OPENNLP-138, feature-gen fix

POS Tagger

en-pos-maxent.bin

721.3 sent/s

732.1 sent/s

joern

yes

 

POS Tagger

en-pos-perceptron.bin

1097.7 sent/s

1110.6 sent/s

joern

 

OPENNLP-155 might improve accuracy a little

Chunker

en-chunker.bin

169,5 sent/s

167,3 sent/s

colen

yes

computerB, tested with CONLL2000 (2012 sentences)

Parser

en-parser-chunking.bin

4.3 sent/s

11.6 sent/s

joern

yes

Macbook was sleeping a little while doing 1.5.0

Note: Test was done on MacBook Pro 13" 7.1, 2.66 GHz Core 2 Duo, 8GB Ram, 256GB SSD running OS X 10.6.6
and Java 1.6.0_22 64-Bit Server.The performance varies because light weight tasks have been performed in the background while testing.

Note: computerB is a DualCore T8100, 4GB Ram, 250GB HD running Ubuntu 10.10 64-Bit and Java 1.6.0_20

Note: "Concurrent" in the comment means that both tests where started at the same time.

Regression Test Training with (private) English data

The training of both versions with the same data must produce
a model with identical output. The model output is tested with
the procedure from the previous test.

To pass the test the event hash and the model output must be identical.

Component

Model

Training Time 1.5.0

Training Time 1.5.1

Tester

Passed

Comment

Sentence Detector

en-sent.bin

0m12.847s

0m11.255s

joern

yes

 

Tokenizer

en-token.bin

2m16.694s

2m30.115s

joern

yes

Re-test tagging was very slow, only 250 sent/s

Name Finder

en-ner-date.bin

 

 

joern

no

OPENNLP-138

Name Finder

en-ner-location.bin

 

 

joern

no

OPENNLP-138

Name Finder

en-ner-money.bin

 

 

joern

no

OPENNLP-138

Name Finder

en-ner-organization.bin

 

 

joern

no

OPENNLP-138

Name Finder

en-ner-percentage.bin

 

 

joern

no

OPENNLP-138

Name Finder

en-ner-person.bin

 

 

joern

no

OPENNLP-138

POS Tagger

en-pos-maxent.bin

 

 

joern

 

 

POS Tagger

en-pos-perceptron.bin

 

 

joern

 

 

Chunker

en-chunker.bin

 

 

joern

 

 

Parser

en-parser-chunking.bin

110m8.712s

138m9.045s

joern

yes

 

Note: Time was measured with the time command, the value is the "real" time value.

Performance test with public data

Test the tagging performance with all the publicly available training
and test data for various languages.

It is assumed that the training will be done with a cutoff of 5 and 100 iterations,
if different values are used please write them into the comment.

Component

Data

Tester

Tagging Perf 1.5.0

Tagging Perf 1.5.1

Comment

Sentence Detector

 

 

 

 

Will not be done in this release.

Tokenizer

 

 

 

 

We need a de-tokenizer dictionary first, will be done in next release.

Name Finder

CONLL 2002 Dutch Person ned.testa

joern

 

Precision: 0.7906976744186046
Recall: 0.48364153627311524
F-Measure: 0.6001765225066196

 

Name Finder

CONLL 2002 Dutch Person ned.testb

joern

 

Precision: 0.8527980535279805
Recall: 0.6384335154826958
F-Measure: 0.7302083333333333

 

Name Finder

CONLL 2002 Dutch Organization ned.testa

joern

 

Precision: 0.8386075949367089
Recall: 0.38629737609329445
F-Measure: 0.5289421157684631

 

Name Finder

CONLL 2002 Dutch Organization ned.testb

joern

 

Precision: 0.7784200385356455
Recall: 0.4580498866213152
F-Measure: 0.5767309064953604

 

Name Finder

CONLL 2002 Dutch Location ned.testa

joern

 

Precision: 0.8362831858407079
Recall: 0.3945720250521921
F-Measure: 0.5361702127659574

 

Name Finder

CONLL 2002 Dutch Location ned.testb

joern

 

Precision: 0.854251012145749
Recall: 0.5452196382428941
F-Measure: 0.665615141955836

 

Name Finder

CONLL 2002 Dutch Misc ned.testa

joern

 

Precision: 0.8300492610837439
Recall: 0.4505347593582888
F-Measure: 0.5840554592720971

 

Name Finder

CONLL 2002 Dutch Misc ned.testb

joern

 

Precision: 0.8373205741626795
Recall: 0.44229149115417016
F-Measure: 0.5788313120176405

 

Name Finder

CONLL 2002 Combined ned.testa

joern

 

Precision: 0.7906976744186046
Recall: 0.48364153627311524
F-Measure: 0.6001765225066196

 

Name Finder

CONLL 2002 Dutch Combined ned.testb

joern

 

Precision: 0.8527980535279805
Recall: 0.6384335154826958
F-Measure: 0.7302083333333333

 

Name Finder

CONLL 2002 Spanish Person esp.testa

joern

 

Precision: 0.8982630272952854
Recall: 0.5924713584288053
F-Measure: 0.7140039447731755

 

Name Finder

CONLL 2002 Spanish Person esp.testb

joern

 

Precision: 0.9008
Recall: 0.7659863945578231
F-Measure: 0.8279411764705882

 

Name Finder

CONLL 2002 Spanish Organization esp.testa

joern

 

Precision: 0.8216258879242304
Recall: 0.6123529411764705
F-Measure: 0.7017189079878665

 

Name Finder

CONLL 2002 Spanish Organization esp.testb

joern

 

Precision: 0.8009331259720062
Recall: 0.7357142857142858 
F-Measure: 0.7669396872673119

 

Name Finder

CONLL 2002 Spanish Location esp.testa

joern

 

Precision: 0.7481789802289281
Recall: 0.7306910569105691
F-Measure: 0.739331619537275

 

Name Finder

CONLL 2002 Spanish Location esp.testb

joern

 

Precision: 0.8226221079691517
Recall: 0.5904059040590406
F-Measure: 0.6874328678839956

 

Name Finder

CONLL 2002 Spanish Misc esp.testa

joern

 

Precision: 0.6446886446886447
Recall: 0.3955056179775281
F-Measure: 0.49025069637883006

 

Name Finder

CONLL 2002 Spanish Misc esp.testb

joern

 

Precision: 0.6595744680851063
Recall: 0.36578171091445427
F-Measure: 0.4705882352941176

 

Name Finder

CONLL 2002 Spanish Combined esp.testa

joern

 

Precision: 0.8982630272952854 
Recall: 0.5924713584288053
F-Measure: 0.7140039447731755

 

Name Finder

CONLL 2002 Spanish Combined esp.testb

joern

 

Precision: 0.9008
Recall: 0.7659863945578231
F-Measure: 0.8279411764705882

 

Name Finder

CONLL 2003 English Person eng.testa

jkosin

Precision: 0.901992661721591
Recall: 0.7263843648208469
F-Measure: 0.8047194918352375

Precision: 0.9352201257861635
Recall: 0.8072747014115093
F-Measure: 0.8665501165501166

 

Name Finder

CONLL 2003 English Person eng.testb

jkosin

Precision: 0.8977988745723299
Recall: 0.6821273964131107
F-Measure: 0.7752427693131103

Precision: 0.8873546511627907
Recall: 0.7551020408163265
F-Measure: 0.8159037754761109

 

Name Finder

CONLL 2003 English Organization eng.testa

jkosin

Precision: 0.8290322580645161
Recall: 0.6226696495152871
F-Measure: 0.711183505195638

Precision: 0.8528584817244611
Recall: 0.6785980611483967
F-Measure: 0.7558139534883722

 

Name Finder

CONLL 2003 English Organization eng.testb

jkosin

Precision: 0.818058934847256
Recall: 0.5394340758579169
F-Measure: 0.6501526888707977

Precision: 0.8263422818791947
Recall: 0.5930162552679109
F-Measure: 0.6905012267788293

 

Name Finder

CONLL 2003 English Location eng.testa

jkosin

Precision: 0.9584186939820742
Recall: 0.7408818726183996
F-Measure: 0.8357262402029991

Precision: 0.9283837056504599
Recall: 0.769188894937398
F-Measure: 0.8413218219708247

 

Name Finder

CONLL 2003 English Location eng.testb

jkosin

Precision: 0.9485177151120753
Recall: 0.7182254196642686
F-Measure: 0.8174619349330977

Precision: 0.9156180606957809
Recall: 0.7416067146282974
F-Measure: 0.8194766478966545

 

Name Finder

CONLL 2003 English Misc eng.testa

jkosin

Precision: 0.8492613111726685
Recall: 0.6052060737527115
F-Measure: 0.706757826338278

Precision: 0.8539007092198582
Recall: 0.6529284164859002
F-Measure: 0.7400122925629993

 

Name Finder

CONLL 2003 English Misc eng.testb

jkosin

Precision: 0.8979300499643112
Recall: 0.5299145299145299
F-Measure: 0.6664957615531857

Precision: 0.8599137931034483
Recall: 0.5683760683760684
F-Measure: 0.6843910806174958

 

Name Finder

CONLL 2003 English Combined eng.testa

jkosin

Precision: 0.8230655223984119
Recall: 0.8039380679905755
F-Measure: 0.8133893616650641

Precision: 0.8601818493738206
Recall: 0.8438236284079434
F-Measure: 0.8519242205420101

1000 iterations

Name Finder

CONLL 2003 English Combined eng.testb

jkosin

Precision: 0.7849405582672956
Recall: 0.7563739376770539
F-Measure: 0.7703925220469681

Precision: 0.8036415565869333
Recall: 0.7970963172804533
F-Measure: 0.8003555555555556

1000 iterations

Name Finder

CONLL 2003 German Person deu.testa

joern

Precision: 0.8272041489863272 
Recall: 0.22626695217701642 
F-Measure: 0.35533762893472637

Precision: 0.8602620087336245
Recall: 0.28122769450392576
F-Measure: 0.4238838084991931

 

Name Finder

CONLL 2003 German Person deu.testb

joern

Precision: 0.7535042735042735 
Recall: 0.2602510460251046 
F-Measure: 0.38687890773270717

Precision: 0.878
Recall: 0.3673640167364017
F-Measure: 0.5179941002949853

 

Name Finder

CONLL 2003 German Organization deu.testa

joern

Precision: 0.6615148726058698 
Recall: 0.29814665592264306 
F-Measure: 0.4110375194740828

Precision: 0.8365695792880259
Recall: 0.41659951651893634
F-Measure: 0.5562130177514794

 

Name Finder

CONLL 2003 German Organization deu.testb

joern

Precision: 0.690884820747521 
Recall: 0.3311772315653299 
F-Measure: 0.4477327413690855

Precision: 0.7942583732057417
Recall: 0.4294954721862872
F-Measure: 0.5575146935348446

 

Name Finder

CONLL 2003 German Location deu.testa

joern

Precision: 0.8779137529137528 
Recall: 0.32006773920406434 
F-Measure: 0.46910886680647634

Precision: 0.7362637362637363
Recall: 0.34038950042337
F-Measure: 0.4655471916618414

 

Name Finder

CONLL 2003 German Location deu.testb

joern

Precision: 0.741636798088411 
Recall: 0.3169082125603865 
F-Measure: 0.44406386065180703

Precision: 0.75
Recall: 0.3652173913043478
F-Measure: 0.4912280701754385

 

Name Finder

CONLL 2003 German Misc deu.testa

joern

Precision: 0.8151658767772512 
Recall: 0.12178217821782178 
F-Measure: 0.21190646707366545

Precision: 0.7213930348258707
Recall: 0.14356435643564355
F-Measure: 0.2394715111478117

 

Name Finder

CONLL 2003 German Misc deu.testb

joern

Precision: 0.8125 
Recall: 0.15074626865671642 
F-Measure: 0.2543095099748208

Precision: 0.6198830409356725
Recall: 0.1582089552238806
F-Measure: 0.2520808561236623

 

Name Finder

CONLL 2003 German Combined deu.testa

joern

Precision: 0.6622805891862553 
Recall: 0.28698530933167804 
F-Measure: 0.400445860424834

Precision: 0.7675205413243112
Recall: 0.32857438444030623
F-Measure: 0.46015647638365687

 

Name Finder

CONLL 2003 German Combined deu.testb

joern

Precision: 0.6632526799570968 
Recall: 0.33324258099646065 
F-Measure: 0.44360278183404916

Precision: 0.7553418803418803
Recall: 0.3849714130138851
F-Measure: 0.5100090171325519

 

POS Tagger

CONLL 2006 Danish

joern

Accuracy: 0.9511278195488722

Accuracy: 0.9511278195488722

 

POS Tagger

CONLL 2006 Dutch

joern

Accuracy: 0.9324977618621307

Accuracy: 0.9324977618621307

 

POS Tagger

CONLL 2006 Portuguese

joern

Accuracy: 0.9659110277825124

Accuracy: 0.9659110277825124

 

POS Tagger

CONLL 2006 Swedish

joern

Accuracy: 0.9275106082036775

Accuracy: 0.9275106082036775

 

Chunker

CONLL 2000

colen

 

Precision: 0.9255923572240226
Recall: 0.9220610430991112
F-Measure: 0.9238233255623465

Evaluator was not available in 1.5.0. To evaluate if something changed I compared the output of 1.5.0 and 1.5.1. The output changed a little because of a bug fixed in 1.5.1 (missing trailing closing bracket)

Chunker

Arvores Deitadas

colen

 

Precision: 0.9406086044071353
Recall: 0.9364814040952779
F-Measure: 0.9385404669668097

AD format for Chunker was not available for 1.5.0

The results of the tagging performance might differ compared to the
1.5.0 release since a precision bug in the calculation of the score has been fixed:
https://issues.apache.org/jira/browse/OPENNLP-59

Test UIMA Integration

The test ensures that the Analysis Engine can run and not not
crash trough simple runtime time code errors. We need to add
more sophisticated testing with the next releases.

Analysis Engine

Tester

Passed

Comment

Sentence Detector

joern

yes

Used to process millions of news articles

Sentence Detector Trainer

Tommaso

yes

Trained and tested with cmd line tool

Tokenizer ME

joern

yes

Used to process millions of news articles

Tokenizer Trainer

Tommaso

 

Trained and tested with cmd line tool

Name Finder

joern

yes

Used to process millions of news articles

Name Finder Trainer

Tommaso

yes

Trained and tested with cmd line tool

Chunker

joern

yes

as part of sample pear

Chunker Trainer

 

 

 

POS Tagger

joern

yes

as part of sample pear

POS Tagger Trainer

Tommaso

yes

Trained and tested with cmd line tool

Parser

 

 

 

createPear.sh

joern

no, retest with RC5

Test that pear is build and works. Now fixed after OPENNLP-143.

Sample PEAR

joern

yes

installed and run over sample text

Distribution Review

Please ensure that the listed files below are included in the distributions
and are in a good state.

Package

File or Test

Tester

Passed

Comment

Binary

LICENSE

joern

yes

AL 2.0 and BSD for JWNL

Binary

NOTICE

joern

yes

standard notice, dates are correct

Binary

README

joern

yes

 

Binary

RELEASE_NOTES.html

joern

yes

issue list is generated correctly

Binary

Test signatures: .md5, .sha1, .asc

joern

yes

rc7

Source

LICENSE

joern

yes

standard AL 2.0 file

Source

NOTICE

joern

yes

standard notice, dates are correct

Source

Test signatures: .md5, .sha1, .asc

joern

yes

rc7

Source

Can build from source?

joern

yes

Test should be done without jwnl and opennlp in local m2 repo.
Test was done on Ubuntu 10.10.

  • No labels