Transformer
What is Transformer?
The neural network design behind almost every modern language model. Introduced in a landmark 2017 paper, its key idea (self-attention) lets the model weigh how strongly every word relates to every other word in a passage, all at once. That made training on huge text collections practical, which set off the whole LLM era.
Why it matters
The reason the transformer, rather than the ideas before it, set off the last decade of AI is mostly about training speed. Earlier designs read text in order, one word after another, which made them slow to train and hard to scale. The transformer looks at a whole passage at once, so training could be spread across many processors and run on far larger text collections than before. That single shift in efficiency is what made today’s large models practical to build. The same design carries a built-in cost, though: the work of relating every word to every other grows sharply as the text gets longer, and this is a large part of the reason very long inputs stay slow and expensive to process.
In practice
Anyone who has watched costs climb when feeding a model very long documents has met this limit first-hand. Because the transformer compares every word with every other, doubling the input more than doubles the work, so a system that cheaply handles a page can turn expensive on a whole contract. Teams design around it by retrieving only the relevant passages instead of dumping everything in, which is often both cheaper and more accurate than relying on an ever-longer context.
