Machine Learning Basics
Which of the following correctly describes the difference between bagging (random forests and the like) and boosting in ensemble learning?
- ABagging trains models independently on several training sets made by sampling with replacement and combines their results by averaging or majority vote, while boosting adds models one after another so that each makes up for the errors of the previous ones
- BBagging adds models one at a time while emphasizing the data the previous model got wrong, while boosting trains mutually independent models in parallel on data sampled with replacement and takes a majority vote
- CBoth bagging and boosting are methods that build one large decision tree and then cut off unneeded branches to prevent overfitting
Show answer and explanation
Answer: A. Bagging trains models independently on several training sets made by sampling with replacement and combines their results by averaging or majority vote, while boosting adds models one after another so that each makes up for the errors of the previous ones
Bagging creates slightly different training sets by drawing randomly from the original data with duplicates allowed (bootstrapping), and combines the predictions of the many models trained on them by averaging or majority vote. The models are independent, so they can be trained in parallel, and the main effect is to reduce variance. A random forest is a bagging method that also chooses the features used for splitting at random, increasing the diversity of the decision trees. Boosting adds weak models one after another so that each makes up for the errors of the models so far, and it mainly reduces bias.
- A. Correct. Bagging averages or votes over independent models, and boosting adds models in sequence to make up for errors.
- B. The descriptions of bagging and boosting are reversed.
- C. This describes pruning a decision tree, not ensemble learning, which combines several models.