Machine Learning Basics
Let p be the probability that a model outputs for the correct class. Which of the following describes the cross-entropy loss −log p used to train classifiers?
- AThe closer p is to 1, the closer the loss is to 0, and as p approaches 0 the loss grows sharply
- BRegardless of the value of p, the loss is proportional to the square of the difference between the correct answer and the prediction
- CThe loss is largest when p is 0.5 and becomes smaller as p approaches 0 or 1
Show answer and explanation
Answer: A. The closer p is to 1, the closer the loss is to 0, and as p approaches 0 the loss grows sharply
The cross-entropy loss is −log 1 = 0 when the probability p assigned to the correct class is 1, grows as p becomes smaller, and grows without limit as p approaches 0. With the natural logarithm it is about 0.69 for p = 0.5 and about 2.3 for p = 0.1. Because it heavily penalizes confident wrong predictions, it is widely used for classification models that output probabilities through the softmax or sigmoid function. The mean squared error, which uses squared errors, is used mainly for regression, which predicts continuous values.
- A. Correct. The loss is 0 when p is 1 and grows as p approaches 0.
- B. Using the square of the difference is the mean squared error.
- C. The loss decreases as p increases; it is not largest at p = 0.5.