Machine Learning Basics
Which of the following describes transfer learning, in which a model trained on a large amount of data is used for a different task with little data?
- AStarting from the weights of the trained model, the output layer is replaced to suit the new task, or part or all of the model is retrained on the new data (fine-tuning)
- BThe trained model is not used; all layers are retrained from randomly initialized weights using only the small amount of data for the new task
- CA small model is trained to imitate the outputs of a large model so that the same task can be solved with less computation and memory
Show answer and explanation
Answer: A. Starting from the weights of the trained model, the output layer is replaced to suit the new task, or part or all of the model is retrained on the new data (fine-tuning)
Transfer learning applies the features that a model learned on one task to a different task. For example, the layers near the input of a CNN trained on a large number of images can extract general features such as edges and textures, so only the output layer is replaced and trained for the new task, or the whole network is retrained with a small learning rate (fine-tuning). It tends to reach high performance with less data and computation than training from scratch, and it is also used to adapt language models to specific uses.
- A. Correct. The trained weights are the starting point and are retrained to suit the new task.
- B. This trains from scratch and does not make use of the features the trained model acquired.
- C. This describes knowledge distillation, a method for making a model for the same task smaller.