EN
Română (Romanian)
Choosing a large language model should not begin with the question, "Which model is the smartest?" For a company, the useful question is: What is the smallest, most predictable, and easiest-to-manage model that meets the requirements of a specific task?
A model that excels at programming may be unnecessarily expensive for classifying messages. A fast model may miss exceptions in a contract. A model that achieves high scores on international tests may produce unnatural Romanian or misinterpret local dates, amounts, and names.
This is why making the right choice is not a contest between providers. It is a procurement and engineering process: we define the task, the risk, and the budget, build a test set that reflects real work, compare the options, and monitor the outcome after launch.
The central idea: we are not looking for a single LLM for the entire company. We choose a model profile for each category of tasks and retain the ability to replace it.
An LLM, short for "large language model," is a model trained to process and generate language. However, the application used by employees or customers contains more than the model: instructions, documents retrieved through RAG, tools, business rules, filters, approvals, and the user interface.
When evaluating a solution, we need to distinguish among three levels:
This distinction matters. A quality problem is not always solved by switching to a larger model. Sometimes the right documents are missing, the instructions are ambiguous, the data schema is weak, or the model has been allowed to make a decision that it should only have proposed.
Current commercial catalogs generally include several classes of models: frontier models for difficult tasks, balanced models for everyday use, compact models for high-volume workloads, and models specialized in code, voice, images, OCR, semantic search, or moderation. The official catalogs from OpenAI, Anthropic, Google, and Mistral AI illustrate this fragmentation.
This is a dated snapshot, not a ranking. Names, prices, limits, and availability change frequently. Provider documentation includes separate pages for deprecations and migrations, and catalogs already contain retired families and versions. A serious technical decision must include the exact identifier of the version tested, the evaluation date, and a migration procedure.
At the same time, "open-weight" and "open-source" are not synonymous. An open-weight model provides access to its learned parameters, but the license, training code, and information about the data may remain restrictive or incomplete. The Open Source AI Definition 1.0 requires the freedom to use, study, modify, and distribute the system, together with access to the preferred form for making modifications, including information about data, code, and parameters. For a company, the license must be reviewed separately from the availability of the model files.
Public benchmarks are useful for creating a shortlist. However, they cannot determine on their own which model a company should purchase.
The paper HELM, Holistic Evaluation of Language Models shows why evaluation must use multiple scenarios and multiple dimensions, not only accuracy. Among other factors, HELM measures robustness, calibration, fairness, toxicity, and efficiency. NIST AI 600-1 warns that laboratory tests and benchmarks may not reflect the real-world context of use and recommends evaluating performance in practical scenarios.
A public score can be misleading for several reasons:
For Romanian, internal testing is essential. The evaluation must include diacritics, names of people and places, Romanian date and number formats, lei and euros, forms of address, texts without diacritics, real typing errors, and company terminology.
LLMs are suitable when the input is ambiguous, primarily textual or visual, and the result requires interpretation, rewriting, or summarization. They are not the first choice for every automation.
A deterministic rule, a SQL query, a schema validator, or a traditional search engine is often cheaper and easier to verify for:
The LLM can interpret the request and propose an action, but business rules must validate the result before execution.
Before testing models, we describe the task on a single page. Without this specification, the comparison becomes a collection of impressions.
The minimum specification should answer the following questions:
Example: "Classify the customer's message into one of 12 categories, extract the order number, and return valid JSON within two seconds. If the number is absent, return null. Do not reply to the customer."
This is a testable task. "Use AI for support" is not.
Disqualifying criteria are checked before quality. If a provider cannot meet the data, licensing, or availability requirements, a better score does not make it suitable.
We inventory the data that may enter the system:
For managed services, we verify contractually and technically: whether data is used for training, the retention period, the processing region, subprocessors, the data processing agreement, deletion, logging, encryption, access control, and incident response. The purpose limitation and data minimization principles of the General Data Protection Regulation remain relevant regardless of a provider's popularity.
Local hosting can reduce exposure to an external API, but it does not eliminate risk. The company becomes responsible for configuration, updates, access, backups, vulnerabilities, and logs.
For downloadable models, we verify:
"Available for free" does not automatically mean "free of cost" or "free of restrictions."
We require answers regarding:
A healthy shortlist contains models from two or three classes, not ten nearly identical variants.
| Profile | Suitable for | Main trade-off |
| --- | --- | --- |
| Compact and fast | classification, simple extraction, repetitive responses, high volume | reaches its limits sooner with ambiguity and exceptions |
| Balanced generalist | internal support, drafting, summarization, RAG, tool calls | not optimal for every specialty |
| Advanced reasoning | multi-step analysis, difficult code, planning, and verification | higher cost and latency |
| Specialized | OCR, transcription, embeddings, code, moderation, or narrow domains | limited coverage outside its specialization |
| Internally hosted open-weight model | operational control, local processing, customization | infrastructure, maintenance, and in-house responsibility |
| Managed API | rapid launch, scaling, and access to new models | provider dependency and contractual terms |
These profiles are design categories, not claims that a specific model will succeed in a particular company. Confirmation comes from testing.
For an SME pilot, an initial set of 50 to 200 carefully selected cases is generally more useful than thousands of generic questions. The exact size depends on the process's variability and risk.
A practical distribution could be:
Where legally permitted, the set should be drawn from real work and anonymized. We keep a separate batch that we do not use to adjust the prompt. Otherwise, we optimize the system for the test and overestimate performance.
For each case, we define:
A single percentage is not enough. We track at least four groups of metrics.
NIST includes confabulation, privacy, information security, and component integration among the risks relevant to generative AI. The document recommends pre-deployment testing, source verification, and continuous monitoring.
The average can hide poor experiences. For an interactive assistant, p95 is often more important than the best demonstration.
The cost of an API is not just the price per million tokens. A simplified monthly estimate is:
number of requests × average input and output cost + tools + storage + observability + human review + cost of errors
For self-hosting, we also add:
A more useful metric than "cost per token" is cost per accepted result. A cheap model that requires two retries and ten minutes of correction may cost more than one with a higher price.
Weights must be established before seeing the results. Otherwise, there is a temptation to change the rules to favor the preferred model.
Example for an internal assistant that answers from company procedures:
| Criterion | Weight | Disqualifying threshold |
| --- | ---: | --- |
| Accuracy on the internal set | 30% | no critical errors |
| Citation and use of sources | 15% | at least 95% valid citations |
| Romanian language | 10% | no operational ambiguities |
| Privacy and contract | 15% | all mandatory requirements |
| p95 latency | 10% | below the process limit |
| Cost per accepted result | 10% | within the approved budget |
| Integration and observability | 5% | identifiable logs and versions |
| Continuity and portability | 5% | documented alternative |
The values are an illustrative scenario, not a universal standard. For marketing copy generation, the weight assigned to style may increase. For invoice extraction, structural validity and field accuracy become dominant. For an agent that can initiate payments, the security, authorization, and human approval thresholds must be much stricter.
The context window is the maximum amount of input and output that a model can process in a single request, usually measured in tokens. Tokens are fragments of text, not whole words.
A large window helps with long documents, but it does not prove that the model can find and correctly use every piece of information. The Lost in the Middle study observed performance degradation when relevant information appeared in the middle of a long context, even in models designed for extended context.
In practice, we test:
RAG can select relevant passages before generation and reduce unnecessary context. However, it does not automatically fix incorrect documents, weak permissions, or an unsuitable search method.
For the first pilot, one default model and one backup model are easier to manage. Once measurements are available, the company can introduce routing.
Three common architectures are:
The FrugalGPT and RouteLLM studies show that dynamic or cascade-based selection can improve the cost-quality tradeoff in the configurations they examined. These results do not guarantee the same savings in a local process. They support testing routing after we have an evaluation baseline and enough real requests.
A routing rule must be able to explain why it selected a model, enforce cost limits, and escalate when confidence is low. For important actions, the final escalation remains a human decision.
We start with a compact model. We ask it to choose labels from a closed list and return valid JSON. We measure accuracy by category, not only the overall average, because a rare category may be critical. If the confidence score is low or the message contains multiple intents, we escalate it.
We choose a general-purpose model that follows instructions and cites context reliably, connected to RAG. We test questions with no answer and conflicting documents. The model must say when the sources are insufficient instead of filling the gap with general knowledge.
We compare a multimodal model or specialized OCR with a combination of traditional OCR and a language model. We validate every field using deterministic rules: totals, currency, tax identification number (CUI), VAT, and dates. Low-confidence fields are sent for human review.
We use automated tests, static analysis, and review, not merely a developer's preference for a response. We measure the share of tasks solved, regressions, vulnerabilities introduced, total time, and cost. Access to the shell, repository, and production must be separated through permissions and approvals.
These are design scenarios. The winning model may differ for each company and even for two similar processes within the same organization.
A focused pilot can follow this schedule:
We define the expected result, users, data, risks, volume, budget, and acceptance thresholds.
We collect and anonymize cases, add exceptions, and establish the reference evaluation with the process experts.
We run the same inputs, instructions, and tools on the shortlisted models. We fix the version and parameters. Where possible, evaluators should not know which model produced each response.
The system receives copies of real requests, but its outputs have no effect. We compare them with human decisions and look for errors that did not appear in the initial set.
A small group uses the system with human approval. We establish budgets, alerts, rapid shutdown procedures, and an incident channel.
At the end, the decision is not limited to "launch" or "do not launch." We can launch for safe categories, retain review for exceptions, and postpone high-risk functions.
The record for a model used by the company should contain:
We reevaluate when the model, prompt, RAG sources, tools, or process change. We monitor the acceptance rate, corrections, incidents, latency, consumption, and distribution of request types. A model may remain unchanged while the data and user behavior evolve.
The right model is not the one that impresses in a demonstration, but the one that consistently produces acceptable results under the company's conditions, language requirements, budget, and risk limits.
For most SMEs, the realistic strategy is:
Choosing an LLM is reversible only if the application, data, and evaluations have been designed for portability. Without them, changing a model name can become a new project.
Would you like us to select and test the right models for your company's processes, using clear criteria for cost, quality, and risk? Talk to us about an audit or a controlled pilot.