MSS-PAE: Saving Autoencoder-based Outlier Detection from Unexpected Reconstruction

: Tan, Xu; Yang, Jiawei; Chen, Junqi; Rahardja, Sylwan; Rahardja, Susanto

Publisher: Elsevier BV

: London

: 2025

: Pattern Recognition

: PATTERN RECOGN

: 111467

: 163

: 14

: 0031-3203

: 1873-5142

DOI: https://doi.org/10.1016/j.patcog.2025.111467

: https://doi.org/10.1016/j.patcog.2025.111467

The Autoencoder (AE) is popular in Outlier Detection (OD) now due to its strong modeling ability. However, AE-based OD methods face the unexpected reconstruction problem: outliers are reconstructed with low errors, impeding their distinction from inliers. This stems from two aspects. First, AE may overconfidently produce good reconstructions in regions where outliers or potential outliers exist while using the mean squared error. To address this, the aleatoric uncertainty was introduced to construct the Probabilistic Autoencoder (PAE), and the Weighted Negative Log-Likelihood (WNLL) was proposed to enlarge the score disparity between inliers and outliers. Second, AE focuses on global modeling yet lacks the perception of local information. Therefore, the Mean-Shift Scoring (MSS) method was proposed to utilize the local relationship of data to reduce the false inliers caused by AE. Moreover, experiments on 32 real-world OD datasets proved the effectiveness of the proposed methods. The combination of WNLL and MSS achieved 45% relative performance improvement compared to the best baseline. In addition, MSS improved the detection performance of multiple AE-based outlier detectors by an average of 20%. The proposed methods have the potential to advance AE's development in OD.

:
This work has been co-funded by the European Union’s Horizon Europe research and innovation programme for research and innovation 2021-2027 under Marie Skłodowska-Curie grant agreement n°[101126611].