Decision Trees & Random Forests
Decision Trees
A Decision Tree splits data recursively by choosing the feature and threshold that best separates classes at each node. Splitting criteria: • Gini Impurity = 1 − Σ pᵢ² (used in sklearn by default) • Information Gain / Entropy = −Σ pᵢ log₂(pᵢ) The tree grows until leaves are pure or a stopping condition is met (max_depth, min_samples_split).
Random Forests
Random Forest = Ensemble of N decision trees trained with two key tricks: 1. Bootstrap Aggregating (Bagging) — Each tree trains on a random sample WITH replacement. 2. Feature Randomness — At each split, only √p features are considered (not all p). Final prediction: majority vote (classification) or mean (regression). Result: Much lower variance than a single tree, while keeping low bias.
Feature Importance
Random Forests provide built-in feature importance — how much each feature reduces impurity across all trees. Useful for feature selection.
Decision Tree & Random Forest Example
Deep trees overfit badly. Random Forests avoid this via ensembling — reducing variance without increasing bias.
Finished reading? Mark it complete to earn your XP.