As industrial use of Language Technology is flourishing and the market for competing software systems is expanding, there is an increasing need for the assessment of NLP components on the level of adequacy evaluation and quality assurance. However, effective and efficient assessment is often restricted by the lack of suitable test material and technology. While ISO norms and standards already exist for mainstream software, the basis and techniques for this are still largely lacking in the area of Language Technology. DiET will develop data, methods and tools for the glass-box evaluation of NLP components, building on the results of previous projects covering different aspects of assessment and evaluation. It will extend and develop test-suites with annotated test items for grammar, morphology and discourse, for English, French and German. DiET will provide user-support in terms of database technology, test-suite construction tools and graphic interfaces. The project will result in a tool-package for in-house and external quality assurance and evaluation, which will enable the commercial user to assess and compare Language Technology products.
- Systematic and controlled construction of test and evaluation data in three different languages (English, French, and German)
- Database technology for the administration and maintenance of the test suites
- Combination of corpora and test suites
- Tools allowing test data to be customised to new applications and domains
- Development and extension of evaluation methodology
- Support for standardisation of Language Technology systems
- Verification of the approach in different application areas
Partners
Aerospatiale France, IBM Heidelberg, Localisation Resources Centre Dublin, SRI International Cambridge, ISSCO Geneva
