Joint optimization of an autoencoder for clustering and embedding

Research output: Journal contributionsJournal articlesResearchpeer-review

Standard

Joint optimization of an autoencoder for clustering and embedding. / Boubekki, Ahcène; Kampffmeyer, Michael; Brefeld, Ulf et al.
In: Machine Learning, Vol. 110, No. 7, 01.07.2021, p. 1901-1937.

Research output: Journal contributionsJournal articlesResearchpeer-review

Harvard

APA

Vancouver

Boubekki A, Kampffmeyer M, Brefeld U, Jenssen R. Joint optimization of an autoencoder for clustering and embedding. Machine Learning. 2021 Jul 1;110(7):1901-1937. doi: 10.1007/s10994-021-06015-5

Bibtex

@article{5d22629599494a47993f95b18dfd860c,
title = "Joint optimization of an autoencoder for clustering and embedding",
abstract = "Deep embedded clustering has become a dominating approach to unsupervised categorization of objects with deep neural networks. The optimization of the most popular methods alternates between the training of a deep autoencoder and a k-means clustering of the autoencoder{\textquoteright}s embedding. The diachronic setting, however, prevents the former to benefit from valuable information acquired by the latter. In this paper, we present an alternative where the autoencoder and the clustering are learned simultaneously. This is achieved by providing novel theoretical insight, where we show that the objective function of a certain class of Gaussian mixture models (GMM{\textquoteright}s) can naturally be rephrased as the loss function of a one-hidden layer autoencoder thus inheriting the built-in clustering capabilities of the GMM. That simple neural network, referred to as the clustering module, can be integrated into a deep autoencoder resulting in a deep clustering model able to jointly learn a clustering and an embedding. Experiments confirm the equivalence between the clustering module and Gaussian mixture models. Further evaluations affirm the empirical relevance of our deep architecture as it outperforms related baselines on several data sets.",
keywords = "Clustering, Deep autoencoders, Embedding, Gaussian mixture models, k-means, Informatics, Business informatics",
author = "Ahc{\`e}ne Boubekki and Michael Kampffmeyer and Ulf Brefeld and Robert Jenssen",
year = "2021",
month = jul,
day = "1",
doi = "10.1007/s10994-021-06015-5",
language = "English",
volume = "110",
pages = "1901--1937",
journal = "Machine Learning",
issn = "0885-6125",
publisher = "Springer Netherlands",
number = "7",

}

RIS

TY - JOUR

T1 - Joint optimization of an autoencoder for clustering and embedding

AU - Boubekki, Ahcène

AU - Kampffmeyer, Michael

AU - Brefeld, Ulf

AU - Jenssen, Robert

PY - 2021/7/1

Y1 - 2021/7/1

N2 - Deep embedded clustering has become a dominating approach to unsupervised categorization of objects with deep neural networks. The optimization of the most popular methods alternates between the training of a deep autoencoder and a k-means clustering of the autoencoder’s embedding. The diachronic setting, however, prevents the former to benefit from valuable information acquired by the latter. In this paper, we present an alternative where the autoencoder and the clustering are learned simultaneously. This is achieved by providing novel theoretical insight, where we show that the objective function of a certain class of Gaussian mixture models (GMM’s) can naturally be rephrased as the loss function of a one-hidden layer autoencoder thus inheriting the built-in clustering capabilities of the GMM. That simple neural network, referred to as the clustering module, can be integrated into a deep autoencoder resulting in a deep clustering model able to jointly learn a clustering and an embedding. Experiments confirm the equivalence between the clustering module and Gaussian mixture models. Further evaluations affirm the empirical relevance of our deep architecture as it outperforms related baselines on several data sets.

AB - Deep embedded clustering has become a dominating approach to unsupervised categorization of objects with deep neural networks. The optimization of the most popular methods alternates between the training of a deep autoencoder and a k-means clustering of the autoencoder’s embedding. The diachronic setting, however, prevents the former to benefit from valuable information acquired by the latter. In this paper, we present an alternative where the autoencoder and the clustering are learned simultaneously. This is achieved by providing novel theoretical insight, where we show that the objective function of a certain class of Gaussian mixture models (GMM’s) can naturally be rephrased as the loss function of a one-hidden layer autoencoder thus inheriting the built-in clustering capabilities of the GMM. That simple neural network, referred to as the clustering module, can be integrated into a deep autoencoder resulting in a deep clustering model able to jointly learn a clustering and an embedding. Experiments confirm the equivalence between the clustering module and Gaussian mixture models. Further evaluations affirm the empirical relevance of our deep architecture as it outperforms related baselines on several data sets.

KW - Clustering

KW - Deep autoencoders

KW - Embedding

KW - Gaussian mixture models

KW - k-means

KW - Informatics

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85109174419&partnerID=8YFLogxK

U2 - 10.1007/s10994-021-06015-5

DO - 10.1007/s10994-021-06015-5

M3 - Journal articles

AN - SCOPUS:85109174419

VL - 110

SP - 1901

EP - 1937

JO - Machine Learning

JF - Machine Learning

SN - 0885-6125

IS - 7

ER -

Documents

DOI

Recently viewed

Publications

  1. Database Publishing Without Databases
  2. Agile knowledge graph testing with TESTaLOD
  3. How to get really smart: Modeling retest and training effects in ability testing using computer-generated figural matrix items
  4. Semi-supervised learning for structured output variables
  5. Evolutionary generation of dispatching rule sets for complex dynamic scheduling problems
  6. Different complex word problems require different combinations of cognitive skills
  7. Simultaneous Constrained Adaptive Item Selection for Group-Based Testing
  8. Noise Detection for Biosignals Using an Orthogonal Wavelet Packet Tree Denoising Algorithm
  9. Applying Bayesian Parameter Estimation to A/B Tests in e-Business Applications
  10. Problem structuring for transitions
  11. Diffusion-driven microstructure evolution in OpenCalphad
  12. How to combine collaboration scripts and heuristic worked examples to foster mathematical argumentation - when working memory matters
  13. An Improved Approach to the Semi-Process-Oriented Implementation of Standardised ERP-Systems
  14. Global temporal typing patterns in foreign language writing
  15. Gain Scheduling Controller for Improving Level Control Performance
  16. Retest effects in matrix test performance
  17. Anatomy of Haar Wavelet Filter and Its Implementation for Signal Processing
  18. Template-based Question Answering using Recursive Neural Networks
  19. Sequencing and fading worked examples and collaboration scripts to foster mathematical argumentation - working memory capacity matters for fading
  20. Four Methods to Distinguish between Fractal Dimensions in Time Series through Recurrence Quantification Analysis