Inside a Language Model: Attention, Tokens, and Why It Hallucinates

There is a version of the large language model story told entirely in metaphor — the model "understands", "reasons", "knows" — and a version told in mechanism. The mechanistic version is not much harder to follow, and it makes the technology's behaviour far more predictable. What follows is the mechanism, in the order the computation actually happens.

Text becomes tokens

A model does not see characters or words. It sees integers.

A tokeniser splits input text into subword units drawn from a fixed vocabulary, typically between 30,000 and a few hundred thousand entries, learned from a corpus by an algorithm such as byte-pair encoding. Common words become single tokens. Rare words fragment: an unusual surname might become four pieces. Whitespace and punctuation carry meaning; leading spaces are often part of a token.

This detail explains a whole family of otherwise puzzling behaviours. Asking a model to count the letters in a word is asking it to introspect about a spelling it never directly observed — it saw a token identifier, not a character sequence. Arithmetic on long numbers is unreliable partly because digits group into tokens inconsistently. Text in languages underrepresented in the tokeniser's training corpus consumes far more tokens per sentence, which raises cost and consumes context.

Each token identifier indexes into an embedding table, producing a vector of some thousands of dimensions. From here on, the model manipulates vectors.

Attention: the actual innovation

The transformer architecture, introduced in 2017, replaced sequential recurrence with a mechanism that lets every position in the sequence look directly at every earlier position.

For each token, the model computes three projections of its current representation: a query, a key, and a value. To decide how much position i should draw from position j, it takes the dot product of query i with key j, scales it, and normalises across all j with a softmax. The resulting weights are used to average the value vectors. In plain terms: each token asks a question, every earlier token advertises what it offers, and the token collects a weighted blend of what best matches its question.

Two properties follow. First, path length between any two positions is one, so long-range dependencies do not have to survive many sequential steps — the vanishing-gradient problem that limited recurrent networks largely disappears. Second, the computation across positions is parallel, which is what made training on very large corpora economically feasible on modern accelerators.

The cost is quadratic. Comparing every position with every other position means attention scales with the square of sequence length in both compute and memory. Extending context from 8,000 to 128,000 tokens is not sixteen times more expensive for the attention component; it is roughly 256 times. An enormous amount of systems engineering — memory-efficient attention kernels, sparse and sliding-window patterns, grouped and multi-query key-value sharing, key-value cache compression — exists to soften this scaling, and long-context capability is mostly a story about those techniques rather than about the core mechanism.

Attention is run in multiple parallel heads, each with its own projections, allowing different heads to specialise: some track syntactic dependencies, some copy earlier tokens, some attend to positional structure. Interpretability research has identified reusable circuits — most famously "induction heads", which detect a repeated pattern and predict its continuation, and which appear to underpin much of a model's in-context learning ability.

The model is not retrieving facts from a database. It is computing a probability distribution over the next token, conditioned on everything currently in its context window.

Between the attention layers

Each transformer block pairs attention with a position-wise feed-forward network — typically two linear transformations with a nonlinearity between them, expanding to several times the model's width and projecting back. This is where most parameters live.

The useful intuition, supported by interpretability work, is a division of labour: attention moves information between positions, while feed-forward layers transform information within a position, acting as a large associative store of learned patterns and factual regularities. Mixture-of-experts architectures exploit this by replacing the dense feed-forward layer with many expert sub-networks and routing each token to only a couple of them, which increases total parameters without proportionally increasing computation per token.

Residual connections carry each layer's input forward alongside its output, and normalisation layers keep activations in a stable range. The residual stream is best pictured as a shared workspace that every layer reads from and writes into incrementally — not a pipeline that transforms text into an answer in one pass, but dozens of small edits to a running representation.

At the top, a final linear layer projects the representation onto the vocabulary, producing a score for every possible next token. Softmax turns scores into probabilities.

Sampling: where randomness enters

The model outputs a distribution, not a token. Something must choose.

Greedy decoding always takes the highest-probability token, which is deterministic but tends to produce repetitive text. Temperature scaling divides the scores before softmax: temperatures below one sharpen the distribution toward confident choices, above one flatten it toward diversity. Top-k restricts sampling to the k most likely tokens; nucleus sampling restricts to the smallest set whose cumulative probability exceeds a threshold, which adapts better to distributions of varying sharpness.

This is why the same prompt yields different answers, and why "make it more creative" and "make it more accurate" pull against each other at the level of a single knob. It is also why apparent confidence in phrasing carries no information about correctness: fluency is a property of the distribution the model learned, not of the truth of the sampled sequence.

Training in three stages

Pretraining optimises one objective across a very large corpus: predict the next token. Nothing more sophisticated is required to produce grammar, factual association, translation ability, and code competence — these emerge because predicting text well requires modelling the processes that generated it. This stage dominates compute cost and is where most capability originates.

Scaling analyses established that loss falls predictably with model size, data volume, and compute, and later work corrected an important imbalance: earlier large models were substantially undertrained relative to their parameter count, and compute is better spent on more data for a smaller model than the field initially assumed. That correction, plus the observation that inference cost scales with parameters, is why recent releases emphasise smaller, longer-trained models.

Supervised fine-tuning then trains on curated demonstrations of the desired behaviour — instructions paired with good responses — converting a raw text continuer into something that answers questions.

Preference optimisation aligns outputs with human judgement. Annotators rank candidate responses; that data trains either a reward model used with reinforcement learning, or is used directly through methods that optimise a preference objective without a separate reward model. This stage shapes tone, refusals, formatting, and helpfulness. It also introduces characteristic artefacts: excessive hedging, unnecessary preambles, and a tendency to agree with the user, all of which are learned from what annotators rewarded.

More recently, models are additionally trained to produce extended intermediate reasoning before answering, with reinforcement learning against verifiable outcomes in domains such as mathematics and programming where correctness can be checked automatically. This measurably improves performance on multi-step problems, at the cost of substantially more inference computation.

Why hallucination is structural

Ask why a model states a plausible falsehood with confidence, and the answer is in the objective function.

Pretraining rewards likely continuations. If a question resembles patterns where a citation follows, the highest-probability continuation is a plausibly formatted citation — and a fabricated one is, to the model, indistinguishable in form from a real one it never memorised. There is no step in the process at which the model consults a source and verifies. Its parameters encode statistical associations, compressed lossily across an enormous corpus.

Preference training can make this worse before it makes it better. If annotators prefer confident, complete answers, the training signal penalises appropriate uncertainty. A model that says "I don't know" loses to one that guesses fluently, unless the evaluation explicitly rewards calibrated abstention.

Practical mitigations therefore change the setup rather than the model. Retrieval-augmented generation fetches relevant documents and places them in context, converting a recall task into a reading-comprehension task, which models do far better. Tool use delegates arithmetic, code execution, and lookup to systems that are actually correct. Structured output constraints and verifier models catch a class of errors at the interface. None of these eliminate hallucination; all of them reduce its rate substantially in narrow domains.

What the model does not have

Three absences explain a great deal of user frustration.

No persistent memory. Unless an application stores and re-injects history, each request begins from the same fixed weights. "Remembering" across sessions is a retrieval feature built around the model.

No introspective access. Asked why it produced an answer, a model generates a plausible explanation in the same way it generates any other text. Stated reasoning may or may not correspond to the computation that produced the answer, and research has shown cases where it demonstrably does not.

No knowledge of time. Weights are frozen at training; the model has no clock and no awareness that its information is stale unless told.

Reading the field without being misled

A few habits make evaluation claims easier to interpret. Check whether a benchmark could plausibly appear in training data, since contamination inflates scores in ways that do not generalise. Prefer held-out or continuously refreshed evaluations. Compare cost per solved task rather than cost per token, because a cheaper model that needs three attempts is not cheaper. Treat any comparison that changes quantisation, context length, or sampling settings between systems as uninformative.

None of this diminishes what these systems do. A single architecture, trained on one objective, produces usable translation, summarisation, code synthesis, and explanation — and does so well enough to be economically significant. That is a substantial result. It is simply a different result from understanding, and the distinction is the most useful thing a practitioner can hold onto.