Autoencoders

An Autoencoder is a neural network trained to compress input into a low-dimensional latent space and reconstruct it: Input x → Encoder → Latent z → Decoder → Reconstructed x̂ Loss: Reconstruction loss = ||x − x̂||² The bottleneck forces the network to learn the most important features. The latent space z is the learned representation.

Undercomplete AE — Bottleneck smaller than input. Forces compression. Learns PCA-like linear representations. Denoising AE — Corrupts input with noise, trains to reconstruct clean version. Learns more robust features. Sparse AE — Adds L1 penalty on activations. Learns sparse representations (few neurons active). Variational AE (VAE) — Learns a probability distribution in latent space. Enables generation of new samples. Foundation of modern generative models.

VAE encodes input as μ and σ (mean and std of a Gaussian). Sample z ~ N(μ, σ²). VAE Loss = Reconstruction Loss + KL Divergence KL divergence: regularizes the latent space to be close to N(0,1), enabling smooth interpolation and generation.