Ensemble Learning - Bagging

In ensemble algorithms, bagging methods form a class of algorithms which build several instances of a black-box estimator on random subsets of the original training set and then aggregate their individual predictions to form a final prediction.

These methods are used as a way to reduce the variance of a base estimator (e.g., a decision tree), by introducing randomization into its construction procedure and then making an ensemble out of it.

In many cases, bagging methods constitute a very simple way to improve with respect to a single model, without making it necessary to adapt the underlying base algorithm. As they provide a way to reduce overfitting, bagging methods work best with strong and complex models (e.g., fully developed decision trees), in contrast with boosting methods which usually work best with weak models (e.g., shallow decision trees).

Bagging methods come in many flavors but mostly differ from each other by the way they draw random subsets of the training set:

• When random subsets of the dataset are drawn as random subsets of the samples, then this algorithm is known as Pasting.
• When samples are drawn with replacement, then the method is known as Bagging.
• When random subsets of the dataset are drawn as random subsets of the features, then the method is known as Random Subspaces.
• Finally, when base estimators are built on subsets of both samples and features, then the method is known as Random Patches.

To learn more about various Ensemble Learning techniques like Bagging, and their implementation, visit this link.