Choose a local AI model according to the result it produces in business work. A distributor needs correct identification of products and commercial conditions. An agency needs Romanian copy that follows a brief. A technical team needs code changes that pass tests. A model shortlist becomes useful when it connects these tasks to the available computer.
Separate the model from the application using it
The model generates a result from the information it receives. An engine runs it, while the application manages users, documents and actions. The same interface can connect both local models and externally hosted services. A local deployment therefore checks the actual destination of each request, including document retrieval and image processing.
The selection record identifies the full model name, edition, weight source, licence, quantisation and engine. A short tag is convenient during installation, while the technical record retains the identity of the actual file used. Before an update changes the system used by employees, compare the old and new versions on the same task.
Three layers, three responsibilities
- Model
Interprets context and generates a proposal.
- Engine
Loads weights and executes inference on hardware.
- Application
Enforces access, retrieves documents and validates operations.
A shortlist for the initial assessment
Ministral 3 includes 3B, 8B and 14B variants documented by Mistral for local and edge use. The family offers base, instruct and reasoning editions, subject to the exact version's conditions. For conversations and employee requests, an instruct edition provides a practical starting point. A base edition is a foundation for further development and requires different preparation for direct use.
Qwen3.8-27B is a candidate for tasks justifying a larger model and text-and-image processing. Qwen publishes its model card under Apache 2.0. The concrete recommendation follows a trial on the chosen hardware using the actual quantised artefact and allocated context. Scores reported by the model's author remain separate from results obtained on company data.
What to test for Romanian-language work
Build a request set resembling daily work: answering a customer, summarising a procedure, identifying a condition in a quotation and rewriting a technical paragraph. Preserve Romanian diacritics, proper names and the original Romanian documents. The evaluator checks meaning, terminology and retained information rather than sentence fluency alone.
Include examples involving negation, exceptions, calendar dates and similar product names. A procedure promising delivery in five working days differs from one promising delivery in five days. A useful trial checks such distinctions and includes questions the document cannot answer. The intended behaviour is to identify missing information and pass the case to the responsible person.
Documents require more than fluent conversation
For a document assistant, test passage retrieval separately from answer generation. When the application retrieves the wrong fragment, changing the conversational model does not repair source selection. Record the document, version and page or section supporting each result. Documents containing tables, scans and footnotes receive separate test cases.
A vision model can process images, but extracted fields still require validation. For an invoice or order, the application checks data types, totals and correspondence with the document. A JSON schema defines output structure; business rules determine whether a value is acceptable and whether the operation can continue.
Three checks for a document
- Retrieval
Was the correct passage selected from the right version?
- Answer
Do the claims match the passage and question?
- Validation
Do fields and calculations follow business rules?
Evaluate coding against the right project and tests
A coding assistant receives a limited set of files and a clear requirement. Ask it to explain a function, propose a small correction and add a relevant check. Evaluate the code difference, introduced dependencies and test results. A persuasive explanation remains one part of the assessment; executable checks establish how the program behaves.
Context size affects how much code can be submitted in one request. Selecting useful files, searching locally and bounding the change reduce unnecessary input. Development teams also care about time to the first proposal, the corrections needed and editor compatibility. Keep these criteria together with the model and its version.
Quantisation and memory affect the practical choice
Quantisation reduces weight memory through lower-precision representations. Labels such as Q4 or Q8 describe representation families, and a file can contain components at different precisions. Check the specific artefact and its results on business tasks. On the same computer, a smaller model can leave room for more context or a more responsive team service.
In mixture-of-experts architectures, active parameters per token and total parameter count describe different things. Budget memory for the weights that must be available and the implementation selected. Compare effective memory use, speed and quality together rather than turning parameter count into a universal ranking.
A reproducible selection record
- Identity
Model, edition, licence and exact artefact.
- Execution
Engine, version, quantisation and hardware.
- Task
Context, data, users and acceptance criteria.
- Outcome
Quality, time and memory measured under matching conditions.
Check the licence of every component
Downloadable weights, inference-engine code and the user interface have separate licences. Apache 2.0 is the published licence for the candidates discussed here; the project retains notices and checks applicable distribution terms. An interface such as Open WebUI has its own licence and branding conditions. Access to code or a model file does not replace that review.
Document the final decision in a table covering tasks, accepted results, response time, memory and commercial conditions. WebHat uses a customised local AI system and provides consultancy on model selection and adaptation to business processes. Selection can lead into an application with AI integration, while the guide to small language models explains the role of compact variants in clearly defined tasks.
Questions about choosing a local model
Is a larger model the right choice for every business?
Choose according to the task and evaluation results. Memory, speed, context and concurrent users determine whether its advantages are useful in the particular application.
Which variant should handle the first conversation?
An instruct edition compatible with the selected engine and hardware. Record the exact tag and start with representative requests before connecting internal documents.
Can the conversational model search every document by itself?
The application must provide retrieval and the necessary permissions. It selects authorised documents and passages, then supplies them to the model for the answer.
How do we maintain a sound choice over time?
Keep the configuration and test set. Every relevant change goes through the same trials and includes a procedure for returning to the accepted version.
