Machine Learning Basics
Which of the following describes the self-attention used in the Transformer?
- AIt reads the elements of a sequence one at a time from the beginning and combines the information of the whole sequence by passing each element's hidden state to the next element
- BIt computes weights from the similarity between each token's query and the keys of all tokens and takes the weighted sum of the values, so the computation for all positions of the sequence can be done in parallel
- CFor each token it stacks many layers of convolutions that look only at a fixed number of neighboring tokens, so that relationships with distant tokens are not handled
Show answer and explanation
Answer: B. It computes weights from the similarity between each token's query and the keys of all tokens and takes the weighted sum of the values, so the computation for all positions of the sequence can be done in parallel
In self-attention, a query (Q), key (K) and value (V) are made from each token's vector, the dot products of a query with all keys are turned into weights by the softmax function, and the weighted sum of the values is computed. Every token can take in information from distant tokens directly, and because the computation can be expressed as matrix products and does not need to wait for the result of the previous time step as in an RNN, all positions of the sequence can be computed in parallel during training. The computation itself contains no word-order information, so it is combined with positional encoding, which adds position information. On the other hand, the amount of computation grows with the square of the sequence length.
- A. This is how an RNN processes a sequence. It computes in time order, so it is hard to parallelize.
- B. Correct. The relationships between all pairs of tokens are obtained as weights and computed in parallel with matrix products.
- C. Self-attention can directly handle relationships with every token in the sequence, not just nearby ones.