Transformer

The transformer was introduced by Google researchers in 2017, in a paper titled "Attention Is All You Need". It replaced older designs that read text one word at a time and struggled to remember the start of a long sentence by the time they reached the end.

The "T" in GPT stands for transformer, and the same design is used well beyond chat: in translation, speech recognition, image models and code tools. It has nothing to do with electrical transformers or the toy robots. Its success is the main reason language models improved so quickly after 2017.

How it works

Text is first split into tokens, and each token becomes a list of numbers. Attention then lets every token look at every other token and decide how much each one matters for its meaning. In "the bank raised its rates", attention helps "bank" link to "rates" and so mean a financial institution, not a riverbank.

A transformer stacks many layers of attention, each refining the picture. Because the tokens are processed in parallel, transformers train efficiently on modern chips, which is what made today's very large models practical. Before training, a transformer knows nothing. During pre-training it reads enormous amounts of text and learns to predict the next token, which forces it to absorb grammar, facts and styles of writing along the way. Further training then teaches it to follow instructions and hold a conversation. The design itself is shared widely; what separates one company's model from another is mostly the data, the scale and the training choices.

Most chat models use a decoder-style transformer: it reads everything so far and predicts one next token, adds it to the input, and repeats until the reply is finished. The cost of attention grows quickly with input length, because every token is compared with every other one. That is the practical reason models have a context window limit and why very long inputs are slower and more expensive to process.

Example

When a travel agency pastes a long customer email into an AI assistant and asks for a summary, the transformer inside connects a date mentioned in the first line with a hotel request in the last paragraph. Older models often lost that link across long text.

The same model still has limits the agency can test. If the email thread is long enough to pass the context window, the earliest messages may be dropped or summarised poorly. Pasting a deliberately long thread and asking about a detail from the first message is a quick way to see where a given tool starts to lose track.

Why it matters for your business

You do not need to know transformer mathematics to use AI tools. The useful takeaway is that these models read your whole input in context, up to their context window, so clear, well-ordered prompts and pages with headings and plain sentences are easier for them to interpret correctly. It also explains why putting the most important instruction or fact clearly near the start or end of a long prompt, rather than burying it in the middle, tends to produce more reliable answers, and why splitting a very long task into smaller steps often works better than one giant request.

Common mistakes

• Thinking "transformer" is a product or brand rather than a general model design.
• Assuming attention means the model understands meaning like a person rather than learning statistical relationships.
• Pasting more text than the model's context window allows and expecting it to use all of it.

Questions people ask

What does GPT stand for?

GPT stands for Generative Pre-trained Transformer. Generative means it produces text, pre-trained means it first learned from a huge general dataset before any specific tuning, and transformer is the neural network design it is built on. The term is used by OpenAI but describes a general approach many models share.

What is attention in a transformer?

Attention is the step where each token weighs how relevant every other token in the input is to its own meaning. It lets the model connect words that are far apart, resolve words with several meanings and keep track of who or what a sentence refers to.

Are all AI models transformers?

No. Transformers dominate language models and are common in image and speech systems, but many other designs exist, including diffusion models for images and simpler models for tabular business data. Researchers are also testing newer architectures that aim to handle very long inputs more cheaply.