Deep Learning Interview Questions

Backpropagation computes gradients of the loss with respect to all weights using the chain rule. Two passes: Forward: compute activations layer by layer, cache intermediate values. Backward: starting from the loss, propagate ∂L/∂w backward using ∂L/∂wᵢ = ∂L/∂aⱼ · ∂aⱼ/∂zⱼ · ∂zⱼ/∂wᵢ. Finally: w ← w - α·∂L/∂w. PyTorch autograd does this automatically via loss.backward().

In deep networks, gradients are multiplied by the derivative of activation functions at each layer during backpropagation. Sigmoid derivative ≤ 0.25, so after 10 layers: 0.25¹⁰ ≈ 0.000001 — gradients vanish. Solutions: • ReLU activation — derivative is 1 for positive values (no shrinkage) • Residual connections (ResNets) — gradients bypass layers via skip connections • Batch normalization — re-normalizes activations • Gradient clipping — caps gradient norm • LSTM gates — control gradient flow in RNNs

BatchNorm normalizes layer inputs across the batch to have zero mean and unit variance, then applies learned scale (γ) and shift (β): x̂ = (x - μ_B) / √(σ²_B + ε); y = γ·x̂ + β Benefits: • Reduces internal covariate shift → stable training • Allows much higher learning rates • Acts as mild regularizer (adds noise via batch statistics) • Reduces sensitivity to weight initialization Important: model.eval() uses running statistics from training, not batch statistics.

CNNs: best for data with spatial or local structure (images, audio spectrograms). Translation invariant — same pattern detected anywhere. Parallelizable. RNNs/LSTMs: best for sequential data with temporal dependencies (time series, text, audio). Handles variable-length sequences. Sequential computation (slower). Today: Transformers have largely replaced LSTMs for NLP. For images: Vision Transformers (ViT) compete with CNNs. But CNNs still win on small datasets and edge devices.

Attention computes a weighted sum of values (V) based on similarity between queries (Q) and keys (K): Attention(Q,K,V) = softmax(QKᵀ / √d_k) · V Q·Kᵀ: similarity between every pair of positions → (seq_len × seq_len) scores √d_k: scaling to prevent softmax saturation softmax: normalize to probabilities (attention weights) ·V: weighted combination of value vectors Multi-head: run H parallel attentions with different projections, concatenate results. Each head learns different relationships (syntax, semantics, coreference).

1. More data or data augmentation 2. Dropout — randomly zero neurons (p=0.1-0.5) 3. L1/L2 regularization (weight_decay in PyTorch) 4. Batch Normalization — mild regularization effect 5. Early stopping — stop when validation loss stops improving 6. Reduce model capacity (fewer layers/neurons) 7. Transfer learning — pretrained features are already generalizable 8. Label smoothing — soft targets instead of hard 0/1