A loss function analysis for classification methods in text categorization
Fan Li, Yiming Yang · 2003
This paper presents a formal analysis of popular text classification methods, focusing on their loss functions whose minimization is essential to the optimization of those methods, and whose decomposition into the training-set loss and the model complexity enables cross-method comparisons on a common basis from an optimization point of view. Those methods include Support Vector Machines, Linear Regression, Logistic Regression, Neural Network, Naive Bayes, K-Nearest Neighbor, Rocchio-style and Multi-class Prototype classifiers. Theoretical analysis (including our new derivations) is provided for each method, along with evaluation results for all the methods on the Reuters-21578 benchmark corpus. Using linear regression, neural networks and logistic regression methods as examples, we show that properly tuning the balance between the training-set loss and the complexity penalty would have a significant impact to the performance of a classifier. In linear regression, in particular, the tuning of the complexity penalty yielded a result (measured using macro-averaged F1) that outperformed all text categorization methods ever evaluated on that benchmark corpus, including Support Vector Machines.
Cited by 67 works.
Authors
References
- The Nature of Statistical Learning Theory
- The Elements of Statistical Learning: Data Mining, Inference, and Prediction
- Text categorization with Support Vector Machines: Learning with many relevant features
- A comparison of event models for naive bayes text classification
- An Evaluation of Statistical Approaches to Text Categorization
- A re-examination of text categorization methods
- An example-based mapping method for text categorization and retrieval
- Text Categorization Based on Regularized Linear Classification Methods