Toward supervised anomaly detection

Research output: Journal contributionsJournal articlesResearchpeer-review

Authors

Anomaly detection is being regarded as an unsupervised learning task as anomalies stem from adversarial or unlikely events with unknown distributions. However, the predictive performance of purely unsupervised anomaly detection often fails to match the required detection rates in many tasks and there exists a need for labeled data to guide the model generation. Our first contribution shows that classical semi-supervised approaches, originating from a supervised classifier, are inappropriate and hardly detect new and unknown anomalies. We argue that semi-supervised anomaly detection needs to ground on the unsupervised learning paradigm and devise a novel algorithm that meets this requirement. Although being intrinsically non-convex, we further show that the optimization problem has a convex equivalent under relatively mild assumptions. Additionally, we propose an active learning strategy to automatically filter candidates for labeling. In an empirical study on network intrusion detection data, we observe that the proposed learning methodology requires much less labeled data than the state-of-the-art, while achieving higher detection accuracies.

Original languageEnglish
JournalJournal of Artificial Intelligence Research
Volume46
Pages (from-to)235-262
Number of pages28
ISSN1076-9757
DOIs
Publication statusPublished - 20.02.2013
Externally publishedYes

    Research areas

  • Informatics - learning strategies, Detection accuracy, Empirical studies, Network intrusion detection, Optimization problems, redictive performance, Supervised classifiers, Unsupervised anomaly detection
  • Business informatics

DOI

Recently viewed

Publications

  1. Effects of strategy instructions on learning from text and pictures
  2. Prior entry explains order reversals in the attentional blink
  3. Foundational Aspects of Polycentric Governance
  4. Determination of the antifungal agent posaconazole in human serum by HPLC with parallel column-switching technique
  5. Welteis
  6. Ästhetikkolumne
  7. States of Comparability
  8. Article 21 Formal Validity
  9. "If you like something, you want it to develop."
  10. Erwiderung einer Erwiderung
  11. Video Game Microtransactions & Loot Boxes - An Empirical Study on the Effectiveness of Social Responsibility Measures
  12. Composing with the terra fluida of interaction: new paths for CCO research as relational practice
  13. Plutonium Worlds
  14. Investigating Factors on R estorative Sleep Quality and its Relationship with Personal Work Performance - An Analysis of Diary Data
  15. Mapping of Innovation Relations
  16. A practical perspective on repatriate knowledge transfer
  17. Relationship between pH-values and nutrient availability in forest soils - the consequences for the use of ecograms in forest ecology
  18. § 22 Level Playing Field and Sustainable Development
  19. Work availability types and well-being in Germany–a latent class analysis among a nationally representative sample
  20. Implementierung eines Fehlerpräventionsprogramms für gefahrenintensive Arbeitsprozesse
  21. The use of knowledge in inter-organisational knowledge-networks
  22. Optimal grazing management rules in semi-arid rangelands with uncertain rainfall
  23. Methodological and empirical insights from gender vulnerability and adaptation responses to climate change in South Asia–a systematic review
  24. Permeable reactive barrier technologies for groundwater remediation in Germany: Recent progress and new developments
  25. Machine Learning Analysis in the Diagnostics of the Dynamics of Ball Bearing with Different Radial Internal Clearance
  26. Existential insecurity and deference to authority
  27. Functionality or Aesthetics?
  28. Using a Bivariate Polynomial in an EKF for State and Inductance Estimations in the Presence of Saturation Effects to Adaptively Control a PMSM
  29. Editorial: Courts in Context. An Empirical Re-Evaluation of Categorization in the Asylum Regime