How Do Large Language Models Work? Generative AI Explained

In everyday life

You type a question, and an AI answers in sentences. People also use it to draft emails, summarize long documents and translate other languages. AI that creates new text, images and so on like this is called "generative AI".

This article covers the kind of generative AI that writes text. At its center is the "large language model". It is also called an LLM, from the first letters of the English name. The news usually leads with product names, but the basic workings inside are shared.

Many details of each product are not made public. So this article sticks to ideas common to many models that can be checked in research papers and official documents.

What it does: guess what comes next

A language model is a calculation that gives the probability of what is likely to come next in a piece of text. A probability is a number from 0 to 100% for how likely each option is. What "large" means is explained later.

Say the input is "Today the weather is really". The model gives a probability to each word that could come next. "nice" gets a high one, "bad" a lower one, and unrelated words almost zero.

Nobody wrote these probabilities into a table. They come from a way of calculating that was tuned little by little while the model learned from huge amounts of text. How it learns is covered later, in "Three common stages of training".

Tokens: the pieces text is split into

The options in the figure are not always whole words. A language model splits text into small pieces and works with those. These pieces are called "tokens". A token can be a single character or a group of several characters.

For example, "The weather is unbelievable" might be split into "The", " weather", " is", " un", "believ" and "able". This is only an illustration, not the output of a real model. Each model has its own way of splitting. The same sentence can give a different number of tokens in a different model.

How much text a model can handle at once, and what a service charges, are often counted in tokens. Token counts don't match character or word counts, so treat them as a rough guide.

Writing one token at a time

An answer is not produced all at once. The model first picks the next token and adds it to the end of the text. Then it uses the whole, longer text to pick the next token. This repeats until it picks a special token that means "the end," or reaches a set length.

The next token is not always the one with the highest probability. Often it is chosen like a lottery, where each option's chance of winning matches its probability. This way of choosing is called "sampling". It is the main reason the same question gets a slightly different answer each time.

Always picking the top option tends to make text dull, with the same phrases repeated. A study by Holtzman and colleagues (posted in 2019) looked at this problem. It proposed cutting off low-probability options before drawing. How lopsided the lottery is can also be changed with a setting called "temperature". A lower temperature makes the top options even more likely, so answers vary less.

Parameters: huge numbers of values tuned by training

Letters can't be calculated with as they are. So the model first turns each token into a list of numbers. It then multiplies those numbers by fixed values and adds them up, over many layers. The values that decide things like how much weight each input gets are called "parameters".

People don't set parameters one by one. They start out as random values and are corrected little by little so the predictions come out right. What the model has learned is packed into the final set of parameter values. The multiply-and-add math is explained in "Why Are GPUs Used for AI?"

The "large" in large language model mainly means it has a very large number of these parameters. For example, the largest Llama 3.1 model, released by Meta in 2024, has about 405 billion parameters. Many models keep their insides private, so their counts can't be checked from outside. That is why a publicly released model is used as the example here.

Three common stages of training

Large language models are usually trained in several stages. Here is a common three-stage flow. How the stages are divided, and the methods used, differ from developer to developer.

The first stage is "pre-training". It starts with huge amounts of text, such as web pages and books. The model sees part of a passage and predicts the next token. When it is wrong, its parameters are corrected a little. The right answer is already in the text, so nobody has to write answers by hand. Llama 3 was trained on about 15 trillion tokens of text (2024 paper).

After pre-training, the model is good at continuing text. But it won't necessarily answer your question. It may even continue a question by writing a list of similar questions.

So in the second stage, the already-trained model gets a little extra training for a specific goal. This is called "fine-tuning". Here it is shown questions paired with examples of good answers (for example, ones written by people). Because it learns from correct examples, this is called "supervised" fine-tuning.

The third stage matches the model to human preferences. The model writes several answers to the same question, and people compare which is better. Using these judgments, the model is pushed toward answers people prefer. A well-known approach uses reinforcement learning, which learns from scores given to results. It is called RLHF (reinforcement learning from human feedback).

This three-stage flow was laid out as a whole in a 2022 paper by OpenAI researchers. Meta, in its 2024 paper, explained that it chose a method called DPO for the third stage instead. DPO learns directly from the comparison results, without reinforcement learning. Meta chose it because it needed less compute for large models and performed better. Not every model follows exactly these three stages.

The Transformer and attention

Most of today's large language models use a design for the calculation called the "Transformer". Researchers at Google introduced it in 2017, in a paper titled "Attention Is All You Need". It was first designed to translate text into another language.

Before that, most text AI read tokens one at a time, from the beginning. Each step had to wait for the previous result, so the work was hard to run in parallel. A Transformer can process all the tokens of existing text at the same time. That suits GPUs, which calculate with huge numbers of circuits at once. When writing an answer, though, it still adds one token at a time.

At the core of the Transformer is attention. In a model that writes text, each token works out how much to focus on each earlier token. It then takes in information from those tokens, in proportion to that focus.

For example, in "Sam lent me a pen. It was blue.", what "It" refers to matters. If "It" can focus strongly on "pen," as in the figure, the model can more easily write a sensible continuation. But the figure is a simplified example for explanation. Working out which layer of a real model focuses on what is not easy for people.

The context window: how much it can see at once

When writing an answer, there is a limit on how much text the model can use as clues. It is a limit on the total tokens of the question, the conversation so far and the answer being written. This is called the "context window".

The limit differs by model. For example, Llama 3.1 can handle up to 128K (about 130,000) tokens (2024). Anything beyond the limit is invisible to the model. So some services drop the older parts of a long conversation, or pass along a summary.

What you tell it during a conversation is used as clues in that conversation. But chatting does not rewrite the model's parameters on the spot. Some services do, however, save what you enter and use it for later training. This is covered in "Using it safely".

Why does it make believable mistakes?

A language model is trained to write believable continuations. The model itself has no way to check whether what it writes is true. So it sometimes writes baseless content in smooth, fluent sentences. This is called a "hallucination".

A 2024 guide from the US National Institute of Standards and Technology (NIST) describes this as "the production of confidently stated but erroneous content". For example, the title of a book that doesn't exist, or a number found nowhere, can appear looking real.

The model is not looking things up in a dictionary of answers. The patterns of the text it learned from are simply blended into its parameter values. It doesn't know, on its own, about events after its training text was collected.

One fix is to connect the model to outside tools such as search, and have it read what it finds before answering. A 2020 paper by Lewis and colleagues named this "retrieval-augmented generation (RAG)". It can reduce mistakes, but not remove them. The sources may contain errors, and the model may misread them.

Training and inference: computing and electricity

Training, which sets the parameters, is done by the developer running large numbers of GPUs for a long time. Using a finished model to produce answers is called inference. Inference happens every time someone uses it. "AI Training vs Inference" compares the two in a table.

During inference, a huge number of parameters is read from memory and used for every token produced. So the speed of moving data from memory matters, not only calculation speed. "What Is HBM?" explains why AI GPUs use a special kind of memory.

This computing happens in dedicated buildings called data centers, using large amounts of electricity. A 2025 report by the International Energy Agency (IEA) found that data centers used about 1.5% of the world's electricity in 2024. The figure includes uses other than AI, but the IEA names AI as the biggest driver of growth in data-center electricity. Data centers are explained in "What Is Cloud Computing?"

Using it safely

Check important details against the original source. Answers involving numbers, dates, law, health or money especially need backing from official sources. Even if an answer gives a source name or URL, open it. Check that it exists and really says that.

Be careful about what you type in, too. Depending on the service's terms and settings, what you enter may be used for later training. Don't enter company secrets or other people's personal information until you have checked how it is handled. For example, Japan's Personal Information Protection Commission issued a caution in June 2023. It warned that personal information entered could be used to train AI.

At work, follow your company's rules and use services whose contracts set out how your input is handled. It is the same idea as using the cloud: you are trusting your data to an outside company. When you use an AI's answer, it is still a person who checks it and decides whether to use it.

Common myths

By the numbers

Model sizes and context limits change every year, so read these numbers as "one example at that time".

Learn more

• Compare in a table: AI Training vs Inference, Supervised vs Unsupervised vs Reinforcement Learning (the ways of learning behind pre-training and RLHF)
• A term about learning: Transfer learning (reusing a trained model for a different purpose)
• Terms about how it works: KV cache (memory that keeps the results for earlier tokens during inference), RNN (the read-in-order approach that was common before the Transformer)
• Count the calculations: "Matrix Multiplication FLOPs and Arithmetic Intensity"
• Read from the hardware side: "Why Are GPUs Used for AI?", "What Is HBM?"