Machine Learning Basics
Which of the following steps in building a machine learning model causes data leakage (information from the evaluation data slipping into training)?
- AComputing the mean and standard deviation for standardization from the training data only, and standardizing the validation and test data with the same values
- BComputing the mean and standard deviation for standardization from all the data and standardizing it, and then splitting it into training and test data
- CSplitting time-series data so that data before a certain date is used for training and data from that date on is used for testing
Show answer and explanation
Answer: B. Computing the mean and standard deviation for standardization from all the data and standardizing it, and then splitting it into training and test data
Data leakage is information that should not be available at prediction time, or information from the evaluation data, entering the training process. Computing the mean and standard deviation over all the data before splitting lets statistics of the test data reach training through preprocessing, which can make performance look better than it really is. The correct procedure is to compute preprocessing parameters from the training data only and apply those values to the validation and test data. For time-series data, a random split would mean predicting the past with future information, so the data is split in time order.
- A. This is the correct procedure for preventing leakage: preprocessing parameters are computed from the training data only.
- B. Correct. The mean and standard deviation of the test data enter training through preprocessing.
- C. This is the correct way to split time-series data, keeping future information out of training.