Measuring Local AI Quality and Speed for Your Business

Local AI evaluation should establish whether the system completes business tasks correctly under accepted timing and access conditions. General benchmarks can inform a shortlist, while acceptance uses representative requests and the installed configuration. For an entrepreneur, the useful measure is an accepted working result together with the time and resources required.

Define an accepted result

A document assistant may need the correct passage, current version and preservation of conditions. Extraction requires field and calculation checks. Coding changes must meet requirements and pass project checks. Write criteria before comparing models so each candidate receives the same trial.

Separate critical errors from style observations. Wrong quantities, information retrieved from an unauthorised document and duplicated actions have different implications from an overly long sentence. The process owner defines exclusion criteria and reviewers apply the same rule to every evaluated output.

Build a representative Romanian test set

Include ordinary requests, alternative wording, internal terminology, numbers and exceptions. Add missing-information and restricted-access cases. For documents, retain reference answers and their supporting passages. Reflect business work, including less frequent cases capable of producing important errors.

A demonstration set can contain distinct sales, operations and support families. Report outcomes by family to identify strengths and areas needing intervention. An aggregate total can conceal a weak category. Keep separate cases for final verification after parameters have been selected.

Fix configuration before measurement

The test sheet identifies the computer, RAM, GPU and video memory, operating system, driver and engine. Add model, artefact, quantisation, context, generation settings and active requests. For RAG, retain collection version, embedding model and retrieval rules. These details help identify which change produced a different result.

Comparing models on one workstation differs from comparing computers. Choose the variable being changed and retain other relevant conditions. If quantisation also changes, state it explicitly. Compare configurations in their intended working form, including application resource requirements.

Measure the first response and complete operation

Time to first text helps assess interactivity. Time to a usable result includes data preparation, retrieval, generation and validation. Tool workflows add authorised execution and outcome verification. Measure these intervals from the user's perspective alongside internal engine timing.

Ollama API responses can include load_duration, prompt_eval_duration, eval_count and eval_duration. Documented durations are in nanoseconds. Generation speed can be calculated as eval_count / eval_duration × 1,000,000,000. This describes generation and remains separate from complete request duration.

Separate cold starts from loaded-model operation

After startup, the first request may include loading weights. A subsequent request may use the resident model and cached data. Record model state and trial order. A fast loaded-model result and a post-restart result answer different operational questions useful for service planning.

Choose a repeatable protocol, such as three runs per pilot condition, and retain every value. Report the median and observed variation alongside raw data. Avoid selecting only the fastest run. After system changes, repeat the same trial and identify changed versions or parameters.

Test context and concurrent users

Longer documents can change memory and processing duration even when final answers are short. Compare short- and long-document tasks under the same acceptance rule. Check inclusion of required information and any input truncation. Correctly answering from a document's opening does not establish complete handling of larger files.

For teams, start with one active request and introduce controlled concurrency levels. Record queue time, individual completion time, peak memory and errors. Select operating capacity at a level meeting business requirements. Account count and requests processed per minute remain distinct indicators.

Calculate time to an accepted outcome

Add review and correction time to technical duration. A faster-generating model can require more work before approval. Retain accepted cases, cases routed to review and critical errors. Document acceptance-rule changes and apply them consistently across the comparison.

Also test service interruption, denied access and job recovery. Acceptance records configuration, trials, observations and the decision for intended use. Those records become the reference for updates. New model or engine versions undergo the same assessment before changing the team system.

A practical comparison protocol

Prepare a sheet for every case containing the input, expected result and acceptance rule. Hide candidate names from reviewers when judging written output. Retain complete identifiers in the reproducibility record, separately from scoring forms. This reduces opportunities for brand preference to influence assessment.

Record run order, cache state and other applications using the workstation. If an unplanned activity interrupts a test, retain that trial and its explanation separately from the main series. When repeating, use the protocol's defined conditions. The report contains raw data alongside the criterion used to select the operating configuration. Another authorised reviewer should be able to apply the same rules and understand why a candidate was accepted, rejected or returned for further adjustment.

WebHat uses a customised local AI system and provides consultancy on test design and selecting a business-appropriate configuration. Include evaluation within a custom AI application project. For internal information, assess answer quality together with data-access controls.

Questions about evaluating local AI

Is tokens per second sufficient?

It measures one stage. Acceptance includes correctness, first response, completion, review and application behaviour.

How should configurations be compared?

Use the same tasks, criteria and relevant conditions. Identify differences in model, quantisation, context, hardware and versions.

Why retain weak results?

They expose variation and cases needing correction. Operating decisions must include those situations and their cost.

When should acceptance be repeated?

After relevant model, engine, data, access or hardware changes. Retain the previous reference and compare before activation.