Hyperparameter Tuning
Parameters vs Hyperparameters
Parameters — Learned from data during training (weights, biases). Hyperparameters — Set before training, control the learning process (learning rate, n_estimators, max_depth, regularization strength).
Grid Search
Exhaustively tries every combination in a defined grid. Guaranteed to find the best in the search space but exponentially slow as dimensions grow.
Random Search
Randomly samples combinations from the parameter space. Empirically finds nearly as good a solution as grid search in much less time — especially when only a few hyperparameters really matter.
Bayesian Optimization with Optuna
Builds a probabilistic model of the objective function and intelligently chooses the next point to evaluate. Much more efficient than random or grid search for expensive models.
For neural networks, use Optuna + PyTorch Lightning. For fast iteration, start with RandomizedSearchCV before Optuna.
Finished reading? Mark it complete to earn your XP.