A Method of Moments Embedding Constraint and its Application to Semi-Supervised Learning

Published

June 17, 2024

Author(s)

Michael Majurski, Parniyan Farvardin, Sumeet Menon, David Chapman

Abstract

Discriminative deep learning models with a linear+softmax final layer have a problem: the latent space only predicts the conditional probabilities $p(y|x)$ but not the full joint distribution $p(y,x)$, which necessitates a generative approach. The conditional probability cannot detect outliers, causing outlier sensitivity in softmax networks. This exacerbates model over-confidence impacting many problems: from hallucinations, to confounding biases, and dependence on large datasets. We introduce a novel embedding constraint based on the Method of Moments (MoM). We investigate the use of polynomial moments ranging from 1st through 4th order hyper-covariance matrices. Furthermore, we use this embedding constraint to train an Axis-aligned Gaussian Mixture Model (AAGMM) final layer, which learns not only the conditional, but also the joint distribution of the latent space. We demonstrate our approach by extending FixMatch based semi-supervised image classification. We find our MoM constraint with the AAGMM layer is able to match or improve upon the reported FixMatch accuracy, while also modeling the joint distribution, thereby reducing outlier sensitivity. Future work explores potential applications for this layer and embedding constraint, and how/why this MoM technique can overcome theoretical limitations of other existing methods including the approximate KL-divergence constraint of variational autoencoders.

Proceedings Title

IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR)

Conference Dates

June 17-21, 2024

Conference Location

Seattle, WA, US

Conference Title

Computer Vision and Pattern Recognition

Pub Type

Conferences

Download Paper

Local Download

Keywords

Deep learning, semi-supervised, latent embedding constraint

Citation

Majurski, M. , Farvardin, P. , Menon, S. and Chapman, D. (2024), A Method of Moments Embedding Constraint and its Application to Semi-Supervised Learning, IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR) , Seattle, WA, US, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=957042 (Accessed May 21, 2025)