Toward supervised anomaly detection

Research output: Journal contributionsJournal articlesResearchpeer-review

Standard

Toward supervised anomaly detection. / Görnitz, Nico; Kloft, Marius; Rieck, Konrad et al.
In: Journal of Artificial Intelligence Research, Vol. 46, 20.02.2013, p. 235-262.

Research output: Journal contributionsJournal articlesResearchpeer-review

Harvard

APA

Vancouver

Görnitz N, Kloft M, Rieck K, Brefeld U. Toward supervised anomaly detection. Journal of Artificial Intelligence Research. 2013 Feb 20;46:235-262. doi: 10.1613/jair.3623

Bibtex

@article{a8f717d595bd41368cab4a7c7732ac50,
title = "Toward supervised anomaly detection",
abstract = "Anomaly detection is being regarded as an unsupervised learning task as anomalies stem from adversarial or unlikely events with unknown distributions. However, the predictive performance of purely unsupervised anomaly detection often fails to match the required detection rates in many tasks and there exists a need for labeled data to guide the model generation. Our first contribution shows that classical semi-supervised approaches, originating from a supervised classifier, are inappropriate and hardly detect new and unknown anomalies. We argue that semi-supervised anomaly detection needs to ground on the unsupervised learning paradigm and devise a novel algorithm that meets this requirement. Although being intrinsically non-convex, we further show that the optimization problem has a convex equivalent under relatively mild assumptions. Additionally, we propose an active learning strategy to automatically filter candidates for labeling. In an empirical study on network intrusion detection data, we observe that the proposed learning methodology requires much less labeled data than the state-of-the-art, while achieving higher detection accuracies.",
keywords = "Informatics, learning strategies, Detection accuracy, Empirical studies, Network intrusion detection, Optimization problems, redictive performance, Supervised classifiers, Unsupervised anomaly detection, Business informatics",
author = "Nico G{\"o}rnitz and Marius Kloft and Konrad Rieck and Ulf Brefeld",
year = "2013",
month = feb,
day = "20",
doi = "10.1613/jair.3623",
language = "English",
volume = "46",
pages = "235--262",
journal = "Journal of Artificial Intelligence Research",
issn = "1076-9757",
publisher = "Morgan Kaufmann Publishers, Inc.",

}

RIS

TY - JOUR

T1 - Toward supervised anomaly detection

AU - Görnitz, Nico

AU - Kloft, Marius

AU - Rieck, Konrad

AU - Brefeld, Ulf

PY - 2013/2/20

Y1 - 2013/2/20

N2 - Anomaly detection is being regarded as an unsupervised learning task as anomalies stem from adversarial or unlikely events with unknown distributions. However, the predictive performance of purely unsupervised anomaly detection often fails to match the required detection rates in many tasks and there exists a need for labeled data to guide the model generation. Our first contribution shows that classical semi-supervised approaches, originating from a supervised classifier, are inappropriate and hardly detect new and unknown anomalies. We argue that semi-supervised anomaly detection needs to ground on the unsupervised learning paradigm and devise a novel algorithm that meets this requirement. Although being intrinsically non-convex, we further show that the optimization problem has a convex equivalent under relatively mild assumptions. Additionally, we propose an active learning strategy to automatically filter candidates for labeling. In an empirical study on network intrusion detection data, we observe that the proposed learning methodology requires much less labeled data than the state-of-the-art, while achieving higher detection accuracies.

AB - Anomaly detection is being regarded as an unsupervised learning task as anomalies stem from adversarial or unlikely events with unknown distributions. However, the predictive performance of purely unsupervised anomaly detection often fails to match the required detection rates in many tasks and there exists a need for labeled data to guide the model generation. Our first contribution shows that classical semi-supervised approaches, originating from a supervised classifier, are inappropriate and hardly detect new and unknown anomalies. We argue that semi-supervised anomaly detection needs to ground on the unsupervised learning paradigm and devise a novel algorithm that meets this requirement. Although being intrinsically non-convex, we further show that the optimization problem has a convex equivalent under relatively mild assumptions. Additionally, we propose an active learning strategy to automatically filter candidates for labeling. In an empirical study on network intrusion detection data, we observe that the proposed learning methodology requires much less labeled data than the state-of-the-art, while achieving higher detection accuracies.

KW - Informatics

KW - learning strategies

KW - Detection accuracy

KW - Empirical studies

KW - Network intrusion detection

KW - Optimization problems

KW - redictive performance

KW - Supervised classifiers

KW - Unsupervised anomaly detection

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=84875512265&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/5182cca1-961d-3f09-a144-35dd9cc37f97/

U2 - 10.1613/jair.3623

DO - 10.1613/jair.3623

M3 - Journal articles

AN - SCOPUS:84875512265

VL - 46

SP - 235

EP - 262

JO - Journal of Artificial Intelligence Research

JF - Journal of Artificial Intelligence Research

SN - 1076-9757

ER -

DOI

Recently viewed

Publications

  1. The Computational Turn, or, a New Weltbild
  2. Archival research on carbon reporting quality. A review of determinants and consequences for firm value
  3. Community and Training in NFDI4DS
  4. Kriminalisierung und Versicherheitlichung von Migration. Editorial
  5. Assoggettamento/Soggettivazione
  6. On the micro-structure of the German export boom
  7. The Measurement of Grip-Strength in Automobiles
  8. Front in the mouth, front in the word
  9. Intra- and interspecific hybridization in invasive Siberian elm
  10. Design und Methode der Studie
  11. Benchmarking question answering systems
  12. Logistisches Montagecontrolling
  13. Seabirds as a subsistence and cultural resource in two remote Alaskan communities
  14. Das Bild im Monitor
  15. Stakeholder Governance – An analysis of BITC Corporate Responsibility Index Data on Stakeholder Engagement and Governance
  16. Call for Submissions Business Ethics Quarterly Special Issue on
  17. Benno Reifenberg (1892-1970)
  18. Article 21 Formal Validity
  19. The Shareholder Value Effect of System Overloads: An Analysis of Investor Responses to the 2003 Blackout in the US
  20. § 18
  21. Foreign and Domestic Takeovers in Germany: First Comparative Evidence on the Post-acquisition Target Performance using new Data
  22. Article 28 Relationship with Existing International Conventions
  23. Ästhetische Operationen
  24. Der Autobiograph als Botaniker
  25. Academic staff development as a catalyst for curriculum change towards education for sustainable development: an output perspective
  26. Franchising as a Strategy for Combining Small and Large Group Advantages (Logics) in Social Entrepreneurship
  27. Systematic Theology
  28. Theosophie
  29. Empathy and Donation Behavior Toward Happy and Sad Chimpanzees