Neura saysGradient boosting builds trees sequentially, each fixing the last — XGBoost, LightGBM.
Where forests build trees independently, boosting builds them in sequence: each new tree predicts the residual errors of the ensemble so far. Done carefully (with learning rate and regularization), it's often the most accurate model for tabular data — XGBoost and LightGBM dominate Kaggle. The cost: more tuning and a risk of overfitting if unchecked.
Power-ups you unlock
Sequential trees, each correcting prior errors
Often the top tabular performer (XGBoost/LightGBM)
Learning rate + regularization control it
More tuning; can overfit if unchecked
The Overfit Ogre attacks — common mistakes
No regularization/early stopping → overfit
Confusing boosting (sequential) with bagging (parallel)
Throwing it at tiny datasets unnecessarily
Boss battleContrast how random forests and boosting build their trees.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>forest: trees built independently, then averaged
boosting: tree2 fixes tree1’s errors, tree3 fixes tree2 ...</pre></body></html>