Gary M. Weiss's Research on Rare Events

My work on rarity shows a natural progression from fundamental work on rare cases and small disjuncts to work on class distribution and then changing class distributions. However, in between those two research areas I worked on the problem of predicting very rare events. This research was motivated by the problem of predicting telecommunication equipment failures from sequences of alarm messages, which I faced while working at AT&T Labs. In this work the events to be predicted were extremely rare and this impacted the requirements for the learning system. Furthermore, because the underlying data was time-series data and because virtually all classification programs require an example-based representation of the data, I faced a sizeable challenge. Because the approach of aggregating the data from time-series data to examples using a sliding window approach would most likely obscure subtle patterns in the data, I designed and implemented Timeweaver, a new genetic algorithm-based classification system that could operate natively on time-series data. The fitness function that I utilized, combined with the search capabilities of the genetic algorithm, made it possible to find subtle patterns in the data that were useful for predicting rare events. This work was described in two main papers, one of which focused on the general problem of event prediction and how to formulate this as a classification problem (Weiss & Hirsh, 1998) and a second paper that focused on the genetic-algorithm based prediction system (Weiss, 1999). This work was notable because it was one of the early efforts in using genetic algorithms for data mining and because the event prediction problem had not previously been formulated clearly and comprehensively as a classification problem. These papers have both been cited quite substantially and the KDD paper on event prediction (Weiss & Hirsh, 1998) has been cited by over 125 other papers.




top | Home | CIS Dept. | Fordham