Evaluating text categorization
David D. Lewis · DARPA Workshop on Speech and Natural Language (HLT) · 1991
While certain standard procedures are widely used for evaluating text retrieval systems and algorithms, the same is not true for text categorization. Omission of important data from reports is common and methods of measuring effectiveness vary widely. This has made judging the relative merits of techniques for text categorization difficult and has disguised important research issues.In this paper I discuss a variety of ways of evaluating the effectiveness of text categorization systems, drawing both on reported categorization experiments and on methods used in evaluating query-driven retrieval. I also consider the extent to which the same evaluation methods may be used with systems for text extraction, a more complex task. In evaluating either kind of system, the purpose for which the output is to be used is crucial in choosing appropriate evaluation methods.
Cited by 231 works. doi:10.3115/112405.112471
References
- Content Analysis for the Social Sciences and Humanities.
- Automatic Indexing: An Experimental Inquiry
- Another look at automatic text-retrieval systems
- SCISOR: extracting information from on-line news
- How evaluation guides AI research
- Retrieval test evaluation of a rule based automatic indexing (AIR/PHYS)
- A news story categorization system
- TOWARDS AUTOMATIC INDEXING: AUTOMATIC ASSIGNMENT OF CONTROLLED‐LANGUAGE INDEXING AND CLASSIFICATION FROM FREE INDEXING
- Plans for a task-oriented evaluation of natural language understanding systems
- The use of titles for automatic document classification