Differences between Bagging and Boosting

In machine learning, Bagging and Boosting are two popular ensemble learning techniques used to improve the performance of models. Ensemble learning combines multiple weak models (often called base learners) to create a stronger predictive model. While both methods aim to enhance accuracy and reduce errors, they work in different ways.

Bagging

Bagging, short for Bootstrap Aggregating, is an ensemble learning technique that reduces variance by training multiple models independently on different random subsets of the training data. The final prediction is obtained by averaging (for regression) or majority voting (for classification) over all the models.

• Randomly selects multiple subsets of data with replacement (bootstrap sampling).
• Trains multiple models (usually of the same type) independently on each subset.
• Combines the predictions of all models to make the final prediction.
• Popular example: Random Forest (uses decision trees as base learners).

Bagging is useful when a model suffers from high variance and overfitting, as it stabilizes predictions.

Boosting

Boosting is an ensemble learning technique that reduces bias by training models sequentially. Each new model attempts to correct the errors made by the previous models, making the ensemble stronger over iterations.

• Starts by training a weak model on the entire dataset.
• Identifies and gives higher weight to misclassified instances.
• Trains the next model to focus on correcting errors made by previous models.
• Final prediction is obtained by combining all weak models.
• Popular examples: AdaBoost, Gradient Boosting, XGBoost.

Boosting is effective when a model has high bias and underfitting, as it improves prediction accuracy.

Bagging vs Boosting

	Bagging	Boosting
Objective	Reduces variance and overfitting.	Reduces bias and improves accuracy.
Training Process	Trains models independently in parallel.	Trains models sequentially, correcting errors.
Data Sampling	Bootstrap sampling (random subsets with replacement).	Uses entire dataset, adjusting weights dynamically.
Final Prediction	Combines predictions using averaging or majority voting.	Combines predictions using weighted sum.
Best Used For	High variance models prone to overfitting.	High bias models needing better accuracy.
Popular Algorithms	Random Forest, Bagging Classifier.	AdaBoost, Gradient Boosting, XGBoost.