Test Plan for Apache OpenNLP 1.5.2

This page contains the test plan for the 1.5.2 release.

The 1.5.2 release does not introduce any changes to the feature
generation expect for the name finder which might generate different
token class features for words with special letters.

Compatibility Test with OpenNLP 1.5.0 SourceForge Models

The 1.5.0 SourceForge models must be fully compatible with the 1.5.2
release. In this test all the English models are tested for compatibility
on the English 300k sentences Leipzig Corpus. It is tested that
the output produced with the same model by both versions has the same md5 hash.

Component	Model	Perf 1.5.1	Perf 1.5.2	Tester	Passed	Comment
Sentence Detector	en-sent.bin	42186.7 sent/s		joern	no	It is assumed it did not pass because of OPENNLP-202. The diff showed that in the first 20 compared cases didn't made a mistake compared to 1.5.1.
Tokenizer	en-token.bin	3091.8 sent/s	2300.4 sent/s	joern	yes
Name Finder	en-ner-person.bin	614.4 sent/s	650.6 sent/s	joern	yes	output identical, measurement was done on a idle system, the new name finder is roughly 10% faster
POS Tagger	en-pos-maxent.bin	732.1 sent/s	816.9 sent/s	joern	yes
POS Tagger	en-pos-perceptron.bin	1110.6 sent/s		joern	no	Perceptron normalization was changed.
Chunker	en-chunker.bin	167,3 sent/s	166.4 sent/s	joern	yes
Parser	en-parser-chunking.bin	11.6 sent/s		joern	no	Could be a regression, reason must be identified!

Note: Test was done on MacBook Pro 13" 7.1, 2.66 GHz Core 2 Duo, 8GB Ram, 256GB SSD running OS X 10.6.6
and Java 1.6.0_26 64-Bit Server.The performance varies because light weight tasks have been performed in the background while testing.

Note: "Concurrent" in the comment means that both tests where started at the same time.

Regression Test Training with (private) English data

The training of both versions with the same data must produce
a model with identical output. The model output is tested with
the procedure from the previous test.

To pass the test the event hash and the model output must be identical.

Component	Model	Training Time 1.5.1	Tester
Sentence Detector	en-sent.bin	0m11.255s	joern
Tokenizer	en-token.bin	2m30.115s	joern
Name Finder	en-ner-person.bin		joern
POS Tagger	en-pos-maxent.bin		joern
POS Tagger	en-pos-perceptron.bin		joern
Parser	en-parser-chunking.bin	138m9.045s	joern

Note: Time was measured with the time command, the value is the "real" time value.

Performance test with public data

Test the tagging performance with all the publicly available training
and test data for various languages.

It is assumed that the training will be done with a cutoff of 5 and 100 iterations,
if different values are used please write them into the comment.

Component	Data	Tester	Tagging Perf 1.5.1	Comment
Sentence Detector
Tokenizer
Name Finder	CONLL 2002 Dutch Person ned.testa		Precision: 0.7906976744186046 Recall: 0.48364153627311524 F-Measure: 0.6001765225066196
Name Finder	CONLL 2002 Dutch Person ned.testb		Precision: 0.8527980535279805 Recall: 0.6384335154826958 F-Measure: 0.7302083333333333
Name Finder	CONLL 2002 Dutch Organization ned.testa		Precision: 0.8386075949367089 Recall: 0.38629737609329445 F-Measure: 0.5289421157684631
Name Finder	CONLL 2002 Dutch Organization ned.testb		Precision: 0.7784200385356455 Recall: 0.4580498866213152 F-Measure: 0.5767309064953604
Name Finder	CONLL 2002 Dutch Location ned.testa		Precision: 0.8362831858407079 Recall: 0.3945720250521921 F-Measure: 0.5361702127659574
Name Finder	CONLL 2002 Dutch Location ned.testb		Precision: 0.854251012145749 Recall: 0.5452196382428941 F-Measure: 0.665615141955836
Name Finder	CONLL 2002 Dutch Misc ned.testa		Precision: 0.8300492610837439 Recall: 0.4505347593582888 F-Measure: 0.5840554592720971
Name Finder	CONLL 2002 Dutch Misc ned.testb		Precision: 0.8373205741626795 Recall: 0.44229149115417016 F-Measure: 0.5788313120176405
Name Finder	CONLL 2002 Combined ned.testa		Precision: 0.7906976744186046 Recall: 0.48364153627311524 F-Measure: 0.6001765225066196
Name Finder	CONLL 2002 Dutch Combined ned.testb		Precision: 0.8527980535279805 Recall: 0.6384335154826958 F-Measure: 0.7302083333333333
Name Finder	CONLL 2002 Spanish Person esp.testa		Precision: 0.8982630272952854 Recall: 0.5924713584288053 F-Measure: 0.7140039447731755
Name Finder	CONLL 2002 Spanish Person esp.testb		Precision: 0.9008 Recall: 0.7659863945578231 F-Measure: 0.8279411764705882
Name Finder	CONLL 2002 Spanish Organization esp.testa		Precision: 0.8216258879242304 Recall: 0.6123529411764705 F-Measure: 0.7017189079878665
Name Finder	CONLL 2002 Spanish Organization esp.testb		Precision: 0.8009331259720062 Recall: 0.7357142857142858 F-Measure: 0.7669396872673119
Name Finder	CONLL 2002 Spanish Location esp.testa		Precision: 0.7481789802289281 Recall: 0.7306910569105691 F-Measure: 0.739331619537275
Name Finder	CONLL 2002 Spanish Location esp.testb		Precision: 0.8226221079691517 Recall: 0.5904059040590406 F-Measure: 0.6874328678839956
Name Finder	CONLL 2002 Spanish Misc esp.testa		Precision: 0.6446886446886447 Recall: 0.3955056179775281 F-Measure: 0.49025069637883006
Name Finder	CONLL 2002 Spanish Misc esp.testb		Precision: 0.6595744680851063 Recall: 0.36578171091445427 F-Measure: 0.4705882352941176
Name Finder	CONLL 2002 Spanish Combined esp.testa		Precision: 0.8982630272952854 Recall: 0.5924713584288053 F-Measure: 0.7140039447731755
Name Finder	CONLL 2002 Spanish Combined esp.testb		Precision: 0.9008 Recall: 0.7659863945578231 F-Measure: 0.8279411764705882
Name Finder	CONLL 2003 English Person eng.testa	jkosin	Precision: 0.9352201257861635 Recall: 0.8072747014115093 F-Measure: 0.8665501165501166	Must be re-done for rc2!
Name Finder	CONLL 2003 English Person eng.testb	jkosin	Precision: 0.8873546511627907 Recall: 0.7551020408163265 F-Measure: 0.8159037754761109	Must be re-done for rc2!
Name Finder	CONLL 2003 English Organization eng.testa	jkosin	Precision: 0.8528584817244611 Recall: 0.6785980611483967 F-Measure: 0.7558139534883722	Must be re-done for rc2!
Name Finder	CONLL 2003 English Organization eng.testb	jkosin	Precision: 0.8263422818791947 Recall: 0.5930162552679109 F-Measure: 0.6905012267788293	Must be re-done for rc2!
Name Finder	CONLL 2003 English Location eng.testa	jkosin	Precision: 0.9283837056504599 Recall: 0.769188894937398 F-Measure: 0.8413218219708247	Must be re-done for rc2!
Name Finder	CONLL 2003 English Location eng.testb	jkosin	Precision: 0.9156180606957809 Recall: 0.7416067146282974 F-Measure: 0.8194766478966545	Must be re-done for rc2!
Name Finder	CONLL 2003 English Misc eng.testa	jkosin	Precision: 0.8539007092198582 Recall: 0.6529284164859002 F-Measure: 0.7400122925629993	Must be re-done for rc2!
Name Finder	CONLL 2003 English Misc eng.testb	jkosin	Precision: 0.8599137931034483 Recall: 0.5683760683760684 F-Measure: 0.6843910806174958	Must be re-done for rc2!
Name Finder	CONLL 2003 English Combined eng.testa	jkosin	Precision: 0.8601818493738206 Recall: 0.8438236284079434 F-Measure: 0.8519242205420101	Must be re-done for rc2! 1000 iterations
Name Finder	CONLL 2003 English Combined eng.testb	jkosin	Precision: 0.8036415565869333 Recall: 0.7970963172804533 F-Measure: 0.8003555555555556	Must be re-done for rc2! 1000 iterations
Name Finder	CONLL 2003 German Person deu.testa		Precision: 0.8602620087336245 Recall: 0.28122769450392576 F-Measure: 0.4238838084991931
Name Finder	CONLL 2003 German Person deu.testb		Precision: 0.878 Recall: 0.3673640167364017 F-Measure: 0.5179941002949853
Name Finder	CONLL 2003 German Organization deu.testa		Precision: 0.8365695792880259 Recall: 0.41659951651893634 F-Measure: 0.5562130177514794
Name Finder	CONLL 2003 German Organization deu.testb		Precision: 0.7942583732057417 Recall: 0.4294954721862872 F-Measure: 0.5575146935348446
Name Finder	CONLL 2003 German Location deu.testa		Precision: 0.7362637362637363 Recall: 0.34038950042337 F-Measure: 0.4655471916618414
Name Finder	CONLL 2003 German Location deu.testb		Precision: 0.75 Recall: 0.3652173913043478 F-Measure: 0.4912280701754385
Name Finder	CONLL 2003 German Misc deu.testa		Precision: 0.7213930348258707 Recall: 0.14356435643564355 F-Measure: 0.2394715111478117
Name Finder	CONLL 2003 German Misc deu.testb		Precision: 0.6198830409356725 Recall: 0.1582089552238806 F-Measure: 0.2520808561236623
Name Finder	CONLL 2003 German Combined deu.testa		Precision: 0.7675205413243112 Recall: 0.32857438444030623 F-Measure: 0.46015647638365687
Name Finder	CONLL 2003 German Combined deu.testb		Precision: 0.7553418803418803 Recall: 0.3849714130138851 F-Measure: 0.5100090171325519
POS Tagger	CONLL 2006 Danish		Accuracy: 0.9511278195488722
POS Tagger	CONLL 2006 Dutch		Accuracy: 0.9324977618621307
POS Tagger	CONLL 2006 Portuguese		Accuracy: 0.9659110277825124
POS Tagger	CONLL 2006 Swedish		Accuracy: 0.9275106082036775
Chunker	CONLL 2000		Precision: 0.9255923572240226 Recall: 0.9220610430991112 F-Measure: 0.9238233255623465
Chunker	Arvores Deitadas		Precision: 0.9406086044071353 Recall: 0.9364814040952779 F-Measure: 0.9385404669668097

The results of the tagging performance might differ compared to the
1.5.0 release since a precision bug in the calculation of the score has been fixed:
https://issues.apache.org/jira/browse/OPENNLP-59
The results of the tagging performance may differ compared to the 1.5.1 release, since a bug was corrected in the event filtering.
(TODO: put jira issue here)

Test UIMA Integration

The test ensures that the Analysis Engine can run and not not
crash trough simple runtime time code errors. We need to add
more sophisticated testing with the next releases.

Analysis Engine	Tester	Passed	Comment
Sentence Detector
Sentence Detector Trainer			Trained and tested with cmd line tool
Tokenizer ME
Tokenizer Trainer			Trained and tested with cmd line tool
Name Finder
Name Finder Trainer			Trained and tested with cmd line tool
Chunker			as part of sample pear
Chunker Trainer
POS Tagger			as part of sample pear
POS Tagger Trainer			Trained and tested with cmd line tool
Parser
createPear.sh
Sample PEAR			installed and run over sample text

Distribution Review

Please ensure that the listed files below are included in the distributions
and are in a good state.

Package	File or Test	Tester	Passed	Comment
Binary	LICENSE			AL 2.0 and BSD for JWNL
Binary	NOTICE			standard notice, dates are correct
Binary	README	colen, jason, james, joern	yes	File was reviewed on the dev list.
Binary	RELEASE_NOTES.html			issue list is generated correctly
Binary	Test signatures: .md5, .sha1, .asc
Binary	JIRA issue list created	joern	no	generation failed!
Source	LICENSE			standard AL 2.0 file
Source	NOTICE			standard notice, dates are correct
Source	Test signatures: .md5, .sha1, .asc
Source	Can build from source?			Test should be done without jwnl and opennlp in local m2 repo. Test was done on Ubuntu 10.10.

Child pages

TestPlan1.5.2