Ensembles for supervised classification learning
David E. Rumelhart, O. Matan · 1996
This dissertation studies the use of multiple classifiers (ensembles or committees) in learning tasks. Both theoretical and practical aspects of combining classifiers are studied. First we analyze the representational ability of voting ensembles. A voting ensemble may perform either better or worse than each of its individual members. We give tight upper and lower bounds on the classification performance of a voting ensemble as a function of the classification performances of its individual members. Boosting is a method of combining multiple classifiers to form a strong classifier. Several issues concerning boosting are studied in this thesis. We study SBA, a hierarchical boosting algorithm proposed by Schapire, in terms of its representation and its search. We show that if the weak learner has low representational complexity, SBA's search may fail to boost or may give a sub-optimal solution. We present a rejection boosting algorithm that trades-off exploration and exploitation: It requires fewer pattern labels at the expense of lower boosting ability. Ensembles may be useful in gaining information. We study their use to minimize labeling costs of data and to enable improvements on performance over time. For that purpose a model for on-site learning is presented. The system learns by querying hard patterns while classifying easy ones. This model is related to query-based filtering methods, but takes into account that, in addition to labeling, filtering through the data has a cost. The Query-By-Committee algorithm is used as a good approximator of the model space for real-world domains. Results using this algorithm on a synthesized problem and a real-world OCR task using both a back-propagation network and a nearest neighbor classifier show that an on-site learner can perform as well as a classifier trained off-site, while achieving significant cost reduction.
Cited by 1 works.