Advanced Data Analysis with Autoencoders

Overview

Autoencoders are a fascinating and versatile tool in machine learning, designed primarily for dimensionality reduction and unsupervised learning. They work by learning to encode input data into a compressed form and then reconstruct it back to its original state without needing labeled data.

This blog will explore the fundamental concepts of autoencoders, including their architecture, types, and key functionalities. We will also discuss practical considerations such as the choice of functions and loss functions, the link between autoencoders and Principal Component Analysis (PCA), and strategies to handle issues like overfitting. Whether you are new to autoencoders or looking to deepen your understanding, this guide will provide a comprehensive introduction to these powerful neural network models.

Introduction

An autoencoder is a neural network used for dimensionality reduction and unsupervised learning, also known as feature learning. It applies backpropagation with the target values set to match the input values. The autoencoder aims to reconstruct the input data X from X without requiring labels.

hw,b (X)≈X or X̂= X

In other words, this process seeks to learn an approximation of the identity function.

X̂(n)=Q^(-1) QX(n)

Although learning the identity function might seem straightforward, imposing constraints on the network, such as limiting the number of hidden units, can reveal interesting structures within the data.

A Simple Autoencoder with One Hidden Layer

In a basic autoencoder with one hidden layer, the goal is to minimize the loss function, defined as:

L(X_i,(X_i ) ̂ )=|(|X_i-(X_i ) ̂ |)|^2

We want X ̂(n)=X_i

• X represents the input layer,

• H denotes the hidden layer output,

• X′ is the output layer,

• b is the bias for the input layer,

• c is the bias for the hidden layer,

Z=H=g(W_1^T X_i+b)
(X_i ) ̂=f(W_2^T h+c)

The objective is to achieve Loss, L=0, indicating perfect input reconstruction.

Case 1: Undercomplete Autoencoder

An autoencoder is considered under complete when the dimension of the hidden layer H is smaller than the dimension of the input X.

dim(h)<dim(X_i )

If the autoencoder can perfectly reconstruct X from H, then H provides a loss-free encoding of X, capturing all the significant characteristics of X.

Case 2: Overcomplete Autoencoder

An autoencoder is termed overcomplete if it learns a trivial encoding by copying X into H and then H into X_i.

dim(h)≥dim(X_i )

This identity encoding does not provide valuable insights into the data’s important characteristics.

The Choice of Functions

• For binary inputs:

When the input is binary, i.e. X_i∈(0,1), then the encoder function is typically a sigmoid function, and the decoder function is usually a logistic function, as it naturally constrains outputs to the range [0, 1].

Logistics function:

(X_i ) ̂=Logistic(W_2^T h+c)

• For real-number inputs:

When inputs are a real number, i.e. X_i∈R, the encoder function is generally linear, while the decoder function is often a sigmoid function.

Linear Function:

(X_i ) ̂=W_2^T h+c

The Choice of Loss Function

• For real-number inputs: The autoencoder aims to close the reconstruction to the original input. This is formalized using the squared error loss function. Training is carried out similarly to a regular feedforward network using backpropagation.

min_(w_1 ) ,_(w_2 ) ,_b ,_c 1/m ∑_1^m▒∑_1^n▒((X_i ) ̂,_j-X_i ,_j )

Using backpropagation, we can then train the autoencoder just like a regular feed forward network.

All we need is a formula for ∂L(θ)/(∂W_2 ) and ∂L(θ)/(∂W_1 )

We also need ∂h(θ)/∂b and (∂h(θ))/∂c

L(θ)=((X_i ) ̂-X_i )^T ((X_i ) ̂-X_i )

• For binary inputs: A logistic decoder produces outputs between 0 and 1, which can be interpreted as probabilities. Cross-entropy loss is commonly used for binary inputs.

σ(z)=1/(1+e^(-z) )

Since outputs are between 0 and 1, we could interpret them as probabilities. So whatever you are reconstructing tells you that suppose the reconstruction value is 0.8, 0.8 tells you that the output should have been 1. If output is 0.2, then 0.2 output was 0.

In practice, we use cross entropy loss for binary inputs.

For a Single n dimensional input, we can use the following loss function:

min(-∑_(j=1)^n▒(x_i ,_j logx_i ,_j+(1-(x_i ) ̂,_j )log(1-(x_i ) ̂,_j )) )

Link Between PCA and Autoencoders

The encoder component of an autoencoder can be equivalent to Principal Component Analysis (PCA) if:

• A linear encoder is used,

• A linear decoder is applied,

• The squared error loss function is employed,

• Inputs are normalized to have zero mean.

(x_i ) ̂,_j=1/√m (x_i ,_j-1/m ∑_(k=1)^m▒〖x_k ,_j 〗)

Normalization ensures that each dimension of the data has a zero mean.

Let X’ be the zero mean data matrix, then what the above normalization gives us is X=1/√m X^’

Now X^T X=1/m (X^’ )^T X^’ is the covariance matrix.

Regularization in Autoencoders

Overfitting occurs when there are a large number of parameters in the model. Overfitting is a common issue in the case of overcomplete autoencoders, which have many parameters. To mitigate overfitting, regularization is applied.

While poor generalization can also occur with undercomplete autoencoders, it is a more significant problem with overcomplete autoencoders. In these cases, the model might learn to simply copy the input to the hidden layer and then the hidden layer back to the output.

Regularization needs to be introduced to address poor generalization. The simplest approach is to add an L2 regularization term to the objective function, which allows for derivative calculation.

min_θ ,_(w_1 ) ,_(w_2 ) ,_(b_c ) (1/m ∑_(i=1)^m▒∑_(j=1)^n▒((X_i ) ̂,_j-X_i ,_j )^2 )+λ|(|θ|)|^2

Theta (θ) represents all the parameters in the model. The regularization term prevents the model from achieving a zero error on the training data, which ensures that it does not simply memorize the data. By not allowing the model to memorize the training data perfectly, regularization helps improve its ability to generalize well to unseen test data.
This is very easy to implement and just adds a term λw to the gradient δL(θ)/δw
and similarly for other parameters.

Another technique to prevent overfitting is weight tying. In weight tying, the weights of the encoder and decoder are constrained to be equal, meaning  .

This effectively reduces the number of parameters in the network by forcing the model to learn a single set of weights for encoding and decoding. Imposing this constraint prevents the model from learning two independent sets of weights, which could lead to overfitting.

Denoising Autoencoders

A denoising autoencoder introduces noise to the input data through a probabilistic process P((X_x ) ̃,_j│X_i ,_j )
before feeding it into the network.
A common corruption technique for binary inputs involves flipping each bit with probability q while retaining it with probability 1−q.

P((X_i ) ̃,_j=0│X_i ,_j )
P((X_i ) ̃,_j=X_i ,_j )=1-q

Sparse Autoencoders

A sparse autoencoder aims to ensure that neurons are inactive for most inputs, meaning their average activation is close to zero. The output values range between 0 and 1 for hidden neurons with sigmoid activation. A neuron is considered activated if its output is close to 1.

Conclusion

Autoencoders represent a powerful tool in machine learning, offering valuable capabilities for dimensionality reduction and feature learning. By learning to compress and reconstruct data, autoencoders can uncover hidden patterns and structures in datasets. Their versatility extends to handling various types of data, including binary and real numbers, and they can be adapted to address specific challenges such as overfitting and noise. Understanding the different types of autoencoders and their applications—from undercomplete to overcomplete and from denoising to sparse—enables practitioners to leverage these models effectively for a wide range of tasks. As the field evolves, autoencoders will play a crucial role in advancing data representation and analysis techniques.

Drop a query if you have any questions regarding Autoencoders and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is the first Indian Company to win the prestigious Microsoft Partner 2024 Award and is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, AWS GenAI Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, Amazon ECS Service Delivery Partner, AWS Glue Service Delivery Partner, Amazon Redshift Service Delivery Partner, AWS Control Tower Service Delivery Partner, AWS WAF Service Delivery Partner and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.