Small language model

There is no official size line. In practice, models from under a billion to a few billion parameters are usually called small, while frontier large language models are many times larger and run in big data centres. The labels move as technology improves: today's small model can outperform an older large one.

"Small" does not mean simple. A well-trained small model can summarise, classify, extract information and answer questions on a focused topic very well. It is weaker at broad knowledge, long complex reasoning and unusual requests. For many routine business tasks, that trade is worth making.

How it works

Small models use the same basic design as large ones, usually a transformer, but with fewer layers and parameters. Many are trained with help from larger models, a process called distillation, where the small model learns to imitate a large model's answers. Careful choice of training data also matters more at small sizes, because there is less capacity to waste.

Because there are fewer parameters, each answer takes less computing power. That allows small models to run on ordinary hardware, sometimes on the device itself, so data never leaves it. Many are released as open-weight models, which can be downloaded and hosted privately.

A small model becomes especially useful when it is narrowed to one job, through fine-tuning on examples of that task or by giving it the right documents at question time. For broad, open-ended questions, a large model usually remains the better choice. Small models are also the ones that run on phones and laptops for features such as offline transcription, text suggestions and summarising notes, where sending everything to a data centre would be slow, costly or unwanted for privacy reasons. Their speed makes them useful as helpers inside larger systems too, for example sorting requests before a larger model handles the hard ones.

Example

A medical clinic wants to sort incoming patient messages into booking, prescription, billing and urgent, without sending patient data to an outside AI provider. A small open-weight model running on a server the clinic controls handles the sorting. Its answers on general medical questions would be unreliable, but it does not need to answer them; it only labels messages.

Before switching over, staff label two hundred past messages by hand and compare the model's labels. It matches on most and occasionally mistakes urgent messages for booking requests, so the clinic adds a rule that any message mentioning pain or bleeding is always flagged for a person, whatever the model says. Running costs are a fixed server bill rather than a charge per message, which suits a steady, predictable volume.

Why it matters for your business

Small models can cut running costs, respond faster and keep sensitive data in-house, which matters for clinics, lawyers and anyone with strict privacy duties. The trade-off is capability and effort: they handle fewer kinds of task, and hosting your own model needs technical skill and maintenance. If you use AI only through a chat app for occasional writing, this choice will rarely affect you; it matters when you build a tool that runs thousands of times.

Common mistakes

• Choosing the largest available model for a narrow, repetitive task a small one would handle more cheaply.
• Expecting a small model to answer broad general-knowledge questions as well as a frontier model.
• Assuming self-hosting is automatically private without securing the server it runs on.
• Skipping a test on real examples before replacing a larger model.

Questions people ask

What is the difference between a small and a large language model?

Size, measured in parameters, and what follows from it. Large models know more and handle complex, varied tasks better. Small models are cheaper and faster, can run on ordinary hardware or devices, and work well on focused tasks, especially when tuned or given the right documents.

Can a small language model run on my laptop?

Many can. Free tools let you download and run small open-weight models on a reasonably modern computer, often without an internet connection. Speed depends on your memory and processor. It is a good way to test whether a small model handles your task.

When should a business use a small language model?

When the task is narrow and repetitive, such as sorting, tagging or extracting fields, when it runs at high volume so costs add up, or when data must stay on your own systems. For varied, open-ended work, a large model through an assistant or API is usually simpler.