Machine Learning Basics
Which of the following describes how a large language model (LLM) processes text?
- AIt converts text into character codes one character at a time, treats characters with close code values as close in meaning, and predicts the next character
- BIt splits text into tokens (units such as words or parts of words), converts each token into an embedding vector for processing, and predicts the probability of the next token
- CIt replaces every word in the text with a dictionary headword before processing, so it cannot process text containing words or new expressions not in the dictionary
Show answer and explanation
Answer: B. It splits text into tokens (units such as words or parts of words), converts each token into an embedding vector for processing, and predicts the probability of the next token
An LLM handles text by splitting it into units called tokens. A token is not necessarily a whole word: frequent words become a single token and rare words are split into several pieces, so words not in any dictionary can still be handled. Each token is converted into an embedding vector obtained through training, and tokens with similar meanings or usage end up with similar vectors. The model predicts the probability of the next token from the tokens so far and generates text by repeatedly appending the chosen token.
- A. The size of a character code has nothing to do with meaning, so embedding vectors obtained through training are used instead.
- B. Correct. It splits text into tokens, turns them into embedding vectors and predicts the next token.
- C. Words can be split into tokens such as parts of words, so words not in the dictionary can be handled.