Small Language Models (SLMs): When to Choose Them Over LLMs

A small language model, or SLM, is a language model with a compact configuration, evaluated for defined tasks and resources. In a business, it can be worth considering for message classification, field extraction or preparing replies under clear rules. The decision follows output quality and the cost of the complete process, not model size alone.

A support assistant identifying an enquiry type does not have the same task as a system analysing a complex file and proposing a strategy. Sometimes a compact model is useful, sometimes a model performing better on the problem, and some applications use both. The architecture needs to reflect these differences rather than treating every request identically.

This guide explains the relationship between SLMs and LLMs, required memory, quantisation, hosting and evaluation. Examples are constructed to explain the technical decision; they do not describe a customer implementation or measured results. We follow a practical principle: choose a configuration sufficient for the task and verifiable under real operating conditions.

What is an SLM, and how does it differ from an LLM?

Model size is one criterion, not the entire application

SLM stands for small language model. The term describes compact models relative to larger models and the resources needed to use them. We do not apply a universal numerical threshold across every family. A comparison should specify the model, parameter count, weight representation, architecture and environment in which the task is executed.

A large language model, or LLM, and an SLM can use mechanisms from the same technological family. The difference is not that one “understands” while the other merely finds words. We assess what each configuration produces on the company's examples, in the required language and format, with information available to the application.

As a historical reference, the Phi-3 technical report published by Microsoft in 2024 documents a compact model trained on selected data and adapted for dialogue. Such reports explain a configuration and its tests. For a project, we compare candidates available at implementation time rather than turning a historical result into a current ranking.

Which tasks make a small language model worth testing?

Repeated processes, bounded inputs and verifiable outputs

A useful testing candidate is a repeated activity with explicit rules and an output that can be checked. Examples include routing enquiries to departments, extracting data from messages, drafting reply options or rewriting text using an approved vocabulary. For each function, we separately define the input and the criterion for acceptance by the application.

Suppose the support team receives a message about an incorrectly delivered product. In this teaching example, the model proposes an enquiry category, identifies the mentioned product and indicates missing information. The application checks the product against the catalogue and prepares the case for an operator. The model does not automatically approve a replacement or refund.

In natural language processing projects for business, we also compare rules and specialised classifiers. If an identifier can be checked exactly against a catalogue, that check remains in software. A generative model is used where interpreting language provides a useful process advantage, not to replace every existing rule indiscriminately.

When should we retain a model performing better on the task?

Ambiguity, exceptions and analysis combining multiple facts

A compact candidate needs careful comparison when a request requires interpreting many conditions, following complex instructions or synthesising conflicting material. An answer can be fluent yet lose a decisive exception. The test set must include precisely these cases, rather than only short messages that are easy to classify and present in a demonstration.

Large size alone is not a certificate of competence. We retain the model meeting the requirements in the actual comparison. Romanian, domain terminology, codes, negations and source adherence are evaluated separately. An aggregate public score does not automatically describe behaviour on a particular company's conversations and documents or establish suitability for its operating rules.

If required information is missing, moving to a larger model does not complete the commercial record. The application must request clarification or consult an authorised source. If analysis exceeds the compact model's accepted criteria, the case can reach another model or an operator through an explicit routing rule.

SLMs, distillation and quantisation: three different concepts

Distillation describes a training method

In distillation, a student model is trained using signals or examples from a teacher model or an ensemble. The objective is to transfer useful behaviour into the student's configuration. Data, task and evaluation establish what is actually transferred. The SLM label does not by itself identify how a model was trained.

A compact model can also be trained through other processes, while a distilled model is not defined solely by its parameter count. We check the technical report and model card for provenance, modifications and intended uses. This information matters for reproducing the configuration and planning updates, not just for its commercial description.

Quantisation changes the numerical representation of values

Quantisation uses reduced-precision representations for weights and, in some configurations, other values involved in computation. It can reduce required memory, but its effect on speed and quality depends on the method, engine and hardware. Weight quantisation does not, by itself, mean reducing the architecture's parameter count or replacing its learned structure.

When comparing RAG, fine-tuning and prompt engineering, we separate supplied information from model behaviour. The same discipline applies here: size, training and numerical format are different axes. We test the exact delivered configuration, including the tokenizer and conversation template, rather than only the name of its model family.

Does a small model have to run locally?

Choose the execution location separately from model size

An SLM can run in a managed service, on company infrastructure or on a compatible device. Local execution describes location, not a capability category. For an internal application, we check connectivity, team access, model updates and the systems it needs to communicate with. An offline-working requirement must be designed for the entire workflow.

In a hybrid architecture, components have clear responsibilities. A compact model can prepare a classification or extraction, while the application validates the result and selects the next step. NVIDIA research on heterogeneous systems advocates investigating this direction; applying it in a business is justified through its own tests and conditions, not a general savings promise.

For business automation with AI integration, we establish where data resides and who can authorise operations. A message may be analysed locally, while querying the CRM or sending a reply requires a separate connection. Offline functionality is practically verified for every dependency needed to complete the task.

How much memory does a small language model need?

Weights are only part of runtime memory

Total memory includes weights, memory used during computation, the generation cache and execution-engine resources. Request lengths and the number processed simultaneously affect sizing. A file fitting on disk does not establish that the application can serve the required workload within available memory and response time, or sustain that workload under concurrent use.

To distinguish precision from parameter count, we use a hypothetical model with 4 billion parameters. Idealised weight storage at 16 bits is: 4 billion × 16 ÷ 8 = 8 billion bytes, or 8 decimal GB. This calculates the raw representation, not a measurement or a recommended minimum hardware configuration for an actual deployment.

The calculation excludes file-format and quantisation metadata, the KV cache, activations and workspaces. The KV cache retains information used when generating subsequent tokens; consumption depends on architecture, context and active requests. Some attention mechanisms limit growth in particular layers, so sizing must use the actual configuration rather than assuming every architecture behaves identically.

We also check the context window of the AI application, cache precision and inference engine. Technical compatibility, available memory and acceptable speed are distinct tests. Equipment is sized after measuring the workflow and concurrent load, not solely from the size of a downloaded model file.

How do we combine small and large models in an application?

The application can route tasks by type, complexity and data policy. In the support example, a bounded classification task can be evaluated on the compact model, while analysis exceeding its criteria can take another route. The routing decision belongs to software and is retained in the operational log alongside the relevant configuration.

We do not use the model's own claim of confidence as the only signal. Software can check the schema, identifiers, required information and compliance with a rule. Cases lacking a sufficient acceptance criterion are marked for clarification or review. An additional model does not replace checking facts that must come from the company's own system.

Data policy also applies during escalation. If a document cannot leave authorised infrastructure, the route to an external service is not used for that document. We retain the permitted alternative or operator intervention. Approval of an operation is also separate from selecting the model that prepared information for that operation.

How do we test an SLM before using it in production?

Compare quality, time and correctly rejected cases

We prepare representative messages, incomplete requests, Romanian wording with and without diacritics and cases involving closely related terms. Each receives an accepted output and conditions requiring an operator. The final set remains separate from examples used to adjust instructions or the model, so comparison also tests situations not used during development.

We measure the actual task: correct category, preserved fields, commercial conditions and the connection to a source. For performance, we distinguish time to the first result from complete duration, including queueing. Concurrent requests and different lengths are tested on the hardware and engine that will actually be used in the deployed workflow.

The comparison retains the same case set and scoring criteria while allowing suitable instructions for each configuration, documented explicitly. We record model version, precision, context, tools and execution settings. When a component changes, relevant tests are repeated to check that functions already accepted by the team still behave correctly after the update.

A useful acceptance record shows important errors, operator interventions and completed tasks, not just answers that look convincing in a demonstration. We also measure behaviour when a source is missing or permissions are insufficient. These situations belong to normal operation in a system connected to real business data and multiple users.

Is an SLM always cheaper for a business?

Compare accepted-operation cost and maintenance

A compact model can reduce resources needed for a particular operation. The application budget also includes data preparation, integration, hosting, updates, monitoring and human review. Self-hosting includes unused capacity and infrastructure administration; a managed service includes usage and the contractual conditions applying to the project. These are different operating models, not merely different model prices.

If some requests are retried or escalated, the calculation follows the whole route. A low-cost initial call may be followed by retrieval, checks and additional intervention. We compare the cost of an accepted task together with its duration and quality at the volume and concurrent load expected for the application.

An AI application costs and maintenance plan establishes who updates sources, investigates errors and validates a new version. Savings are calculated for the company's process. We do not transfer percentages from a public benchmark into a customer budget using different data, requests and operating conditions.

Does local execution also solve data security?

Running on company infrastructure provides control over the execution location, but the application still needs access controls, isolation and logging rules. We check where documents, messages and backups go. Telemetry, monitoring services and external integrations must be inventoried, including when the model itself runs locally without needing an external generation service.

An externally received document remains content to analyse. Instructions hidden inside it cannot grant access to another customer or change operational rules. Validation and authorisation are enforced on the server, while commercially consequential operations follow the approval established for each case. Model location does not replace these controls or the testing needed to verify them.

Data protection when integrating AI also includes model provenance, version control and access to the execution environment. The model card and licence are checked for the exact model, including adaptation and commercial use. Making weights available for download does not remove their conditions of use.

How do we select a compact model for a business application?

We start with a concrete task and clearly described operational routes. We establish permitted data, accepted outputs and situations requiring clarification or approval. Candidates are selected for compatibility with language, tools and hosting requirements, then compared before the product is sized around a single model or dependent on a particular deployment choice.

The project moves through preparation, testing, integration and operation. Its deliverable includes a verified configuration, routing rules, validations and maintenance responsibilities. The model is one component of this system, and replacing it should be assessable without losing history or access rules. This separation also makes ownership of changes clearer for the team.

In software application development for business, WebHat can integrate a compact model, a managed service or a hybrid architecture according to the process. We start with a function useful to the team and verifiable criteria. Choosing an SLM versus an LLM becomes a documented technical and commercial decision.

Frequently asked questions about small language models

Does a particular parameter count define every SLM?

We do not apply one threshold to every family. We specify the compared configuration, parameter count, numerical representation and resources required by the task. The term “small” is useful when these details are explicit and the comparison includes application outcomes, rather than merely the model's size or a category label.

Is a quantised model automatically an SLM?

Quantisation reduces the numerical precision of some values, not automatically the parameter count. A large model may have a quantised version, while a compact model can also be quantised. We evaluate the exact configuration because memory, speed and quality depend on its implementation, engine and operating conditions.

Can an SLM use company documents through RAG?

Yes, within an application supplying selected sources and the necessary instructions. We check document retrieval and correct use of the passage. Context capacity, request format and behaviour on the actual task are tested alongside permissions and source updates. The compact-model label alone does not establish whether those requirements are met.

Do we need our own server for a small model?

Hosting location is selected separately from model size. A compact model can be accessed through a managed service or run on compatible infrastructure. The choice includes permitted data, connectivity, workload and administration responsibility; capacity is established through testing, not solely through the size of the model file.

How do we check whether a model handles Romanian well?

We use permitted real examples or examples constructed around the process: diacritics, abbreviations, negations, specialised terminology and ambiguous enquiries. Important fields and conditions are checked, not just fluency. Outputs are compared with references accepted by the team and with cases retained separately for testing rather than instruction adjustment.

Can we start with one model and replace it later?

Yes, if the application separates model integration from rules, data and authorisation. We retain versions, the evaluation set and the previously accepted configuration. Replacement is validated against the same requirements and includes checking cost, duration and effects on workflows already used by the team, before adopting the new configuration.