Interactive deduplication using active learning
Sunita Sarawagi, Anuradha Bhamidipaty · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · 2002
Deduplication is a key operation in integrating data from multiple sources. The main challenge in this task is designing a function that can resolve when a pair of records refer to the same entity in spite of various data inconsistencies. Most existing systems use hand-coded functions. One way to overcome the tedium of hand-coding is to train a classifier to distinguish between duplicates and non-duplicates. The success of this method critically hinges on being able to provide a covering and challenging set of training pairs that bring out the subtlety of deduplication function. This is non-trivial because it requires manually searching for various data inconsistencies between any two records spread apart in large lists.We present our design of a learning-based deduplication system that uses a novel method of interactively discovering challenging training pairs using active learning. Our experiments on real-life datasets show that active learning significantly reduces the number of instances needed to achieve high accuracy. We investigate various design issues that arise in building a system to provide interactive response, fast convergence, and interpretable output.
Cited by 692 works. doi:10.1145/775047.775087
Authors
References
- C4.5: Programs for Machine Learning
- A Tutorial on Support Vector Machines for Pattern Recognition
- Data mining and knowledge discovery: making sense out of data
- A TUTORIAL ON SUPPORT VECTOR MACHINES FOR PATTERN RECOGNITION, DATA MINING AND KNOWLEDGE DISCOVERY
- Digital libraries and autonomous citation indexing
- Query by committee
- A guided tour to approximate string matching
- Improving Generalization with Active Learning
- Data Mining Using MLC a Machine Learning Library in C
- Selective Sampling Using the Query by Committee Algorithm