Vision Transformers (ViT)

ViT (Dosovitskiy et al., 2020) applies the Transformer architecture directly to image patches, without any convolutional layers. Key idea: split the image into 16×16 patches → flatten each patch → project to d_model dimensions → apply standard Transformer encoder. A 224×224 image with 16×16 patches gives (224/16)² = 196 patch tokens + 1 [CLS] token = 197 total tokens.

CNNs: + Strong inductive biases (locality, translation invariance) — good with less data + Computationally efficient (shared local filters) − Limited global context (fixed receptive field) ViT: + Global self-attention from the first layer — sees full image context + Scales better with data and compute − Needs large datasets or heavy augmentation (no inductive bias) − Quadratic attention cost O(n²) — expensive for high-resolution

DeiT — Data-efficient ViT via knowledge distillation from CNN teacher. Trains well on ImageNet alone. Swin Transformer — Hierarchical ViT with shifted window attention. O(n) cost. State-of-the-art on detection/segmentation. MAE (Masked Autoencoder) — Self-supervised ViT pretraining by masking 75% of patches and reconstructing. Learns rich representations. DINO / DINOv2 — Self-supervised ViT features. Excellent zero-shot performance.

ViT-Base/16 implementation with patch embedding, positional encoding, and Transformer encoder.