EN
Română (Romanian)
A language model may know a great deal about the world, but it does not automatically know the latest version of your internal procedure, what was decided in a meeting, which clause applies to a customer, or which incident affected a particular server. Even a very capable model works with the information it received during training and the data provided in the current conversation.
RAG, short for retrieval-augmented generation, is one of the methods an AI application can use to search for relevant information before formulating an answer. Instead of asking the model to answer only from its general knowledge, we provide selected excerpts from documents, databases, or other authorized sources.
The idea seems simple: search, add the results to the context, then generate the answer. In practice, the difference between a convincing demonstration and a useful system comes down to document quality, how information is extracted and segmented, the search method, how permissions and updates are managed, and the quality of citations and evaluation.
As of September 24, 2026, no single RAG method is best for every organization. However, there is a set of practices mature enough for production and several advanced approaches that are worth using only when the problem justifies them.
RAG did not emerge from a single innovation. Earlier systems such as DrQA combined search with text comprehension, while kNN-LM and REALM explored the use of external memory. The paper published in 2020 by Patrick Lewis and his co-authors introduced the term RAG and a family of generative models that combined a sequence-to-sequence model with a dense index built from a version of Wikipedia.
The distinction remains useful:
RAG is therefore an architecture, not a product or a database. An implementation may use a traditional search engine, a vector database, PostgreSQL, a knowledge graph, managed cloud services, or a combination of these.
The goal is not to load all company information into the prompt. The goal is to find a small set of relevant, authorized, and sufficiently up-to-date evidence for the request at hand.
RAG is often confused with agent memory, the context window, or fine-tuning. These mechanisms solve different problems.
| Concept | Where the information resides | Primary role |
| :--- | :--- | :--- |
| Model knowledge | In the parameters produced by training | General knowledge and learned patterns, which are difficult to update and attribute to verifiable sources |
| Context window | In the current request sent to the model | Temporary workspace for instructions, messages, and evidence |
| Conversation history | In the messages and results retained by the application | Continuity between exchanges and, sometimes, between sessions |
| Agent memory | In persistent records of preferences, decisions, or events | Personalization and long-term continuity |
| RAG | In documents, indexes, and external knowledge bases | Brings relevant, updatable, and citable information into the context |
| Fine-tuning | In the model's modified parameters | Adjusts behavior, style, or performance on a task |
| Agent tools | In APIs, databases, and external applications | Read current state or perform actions |
RAG does not mean the model was "trained on the company's documents." The documents remain outside the model and are consulted when a question is asked.
RAG does not guarantee that an answer is true. If the system finds an outdated, incorrect, or compromised document, the answer may be faithful to that source and still be wrong. This is why groundedness, meaning that the answer is supported by the retrieved context, is not the same as truth.
RAG does not replace an API or a SQL query. For current inventory, an invoice amount, or an order status, the appropriate source is often the operational system. A document index can explain the applicable policy, but it should not invent the state of a transaction.
Procedures, decisions, manuals, and lessons learned can become searchable through natural language. The answer should identify the document, version, and date, and users should see only the information they are allowed to access.
Documentation, policies, and guides can be accessed through RAG. Customer, subscription, and ticket data are read through an API. Changes, refunds, and external communications remain separate actions that require authorization.
An agent can find relevant services, case studies, and technical answers, then prepare a first draft of a proposal. Current prices and terms must come from the valid commercial source, not from an outdated document returned by semantic search.
A system can identify clauses, compare versions, and point to applicable policies. The final legal decision should not be delegated to the model, and the sources must remain verifiable.
Code, documentation, tickets, incidents, and runbooks can be searched together. Lexical search remains important for errors and identifiers, while semantic retrieval helps when the same problem is described in different ways.
Course materials, textbooks, bibliographies, and activities can be queried with page citations. Tables, maps, diagrams, and scanned pages require a multimodal strategy or good structural extraction.
RAG can explain products and policies, find alternatives or documentation, and support comparisons. Current prices, availability, compatibility, and commercial terms must be verified in the catalog and operational systems.
An agent can search reports, internal documents, and public sources, group the evidence, and generate a report with citations. Important conclusions require human verification of both sources and claims.
A complete system has two flows: preparing knowledge and answering questions.
Sources may include PDFs, Office documents, web pages, emails, meeting notes, wikis, support tickets, contracts, manuals, files from Nextcloud, SharePoint, or Google Drive, and, when appropriate, data from CRM, ERP, or relational databases.
A list of files uploaded once is not yet a knowledge base. The system must know the original source, who owns it, which version is valid, and when it should be synchronized or removed.
Text must be extracted without losing the structure that gives it meaning. In a real document, headings, sections, footnotes, columns, tables, images, and the relationship between pages can change how the information should be interpreted.
A scanned PDF will usually require OCR or a multimodal workflow that processes page images directly. A textbook or a complex report may require analysis of the page's visual structure, separate extraction of tables, and preservation of page coordinates. In many projects, the quality of this stage can affect the outcome more than changing the language model.
Documents are divided into excerpts, commonly called chunks. A fixed chunk size and overlap between chunks are a starting point, not a universal rule.
For company documents, it is more useful to preserve:
Azure documentation describes both semantic chunking and general document chunking strategies for vector search. There is no universally optimal chunk size that can simply be copied from one vendor and applied to every corpus.
Each chunk must be associated with its own metadata, or with a secure reference to the parent document's metadata, so that filtering and auditing can use:
Permissions are applied before the text is sent to the model. A prompt that tells the model not to disclose another customer's documents is not an access control. The search engine must exclude chunks, documents, or sources the user is not entitled to see.
Chunks can be indexed in several ways:
In a conversation, the question "but what about the 2025 contracts?" cannot be searched correctly without the previous message. The application can rewrite the request as a standalone query and identify the language, entities, time range, and likely source.
Complex questions can be split into subquestions. This stage helps in research and multi-hop scenarios, but it adds latency, cost, and the possibility that the system will drift away from the user's intent.
The system retrieves a set of candidates, may combine results from several methods, and can use a reranking model to select the chunks that best answer the question.
The first stage aims to avoid missing important evidence. Reranking aims to remove noise before information reaches the model's context.
The model receives the question, instructions, and selected chunks. The application can require it to answer only from the evidence, cite its sources, and state explicitly when the information is insufficient.
In a verifiable implementation, citations should lead to the most precise location the platform allows: a chunk, page, or at least the document used, rather than merely the home page of a website. For internal documents, the version, date, page, and source status are useful.
An observable system should retain the technical trace of the question: generated queries, applied filters, retrieved documents, scores, answer, citations, cost, and latency. Sensitive data must be masked before logging or retained according to a clear policy, rather than recorded in full by default.
The expression second brain is a useful metaphor for a personal or organizational knowledge base, but it does not refer to a standard technology.
A useful "second brain" is not a chatbot into which every file has been uploaded. It is a managed system made up of:
For example, "the user prefers concise reports" is suitable information for the agent's personal memory. "Backup procedure, version 4.2" is documentary information that must be retrieved from the official source. "Current free disk space on the server" must be read through a monitoring tool.
RAG can be the retrieval engine of a second brain system, but it is not the entire system. Without permissions, versions, and update rules, it can turn disorganized documents into a convincing answer, not into a trustworthy knowledge source.
Products such as NotebookLM illustrate a workspace grounded in sources selected by the user. Self-managed implementations include projects such as AnythingLLM, Khoj, and RAGFlow. These can accelerate a prototype, but choosing a product does not automatically solve document quality, permissions, or evaluation.
Methods such as BM25 assign importance to terms that occur in both the question and the document. They are highly useful for:
Lexical search should not be treated as obsolete technology. The BEIR benchmark showed that BM25 remains a robust baseline across heterogeneous domains.
A semantic embedding model transforms the question and chunks into numerical representations. The engine searches for nearby vectors, making it possible to find an idea even when the question uses different words from the document.
Dense Passage Retrieval was one of the important papers in this direction. Dense search is effective for paraphrases and semantic similarity, but it can miss exact identifiers and return chunks that are topically similar without containing the required answer.
Hybrid search combines lexical and vector search, then merges the result lists. A common method is Reciprocal Rank Fusion, which combines document positions without assuming that scores from the two systems are directly comparable.
For many production projects, hybrid search is a safer starting point than using vectors alone. The documentation for Azure AI Search, Qdrant, Weaviate, and OpenSearch describes implementations of this model.
Models such as ColBERT retain token-level representations and compare a question with a passage in greater detail. Compared with methods that use a single vector for each chunk, this family can improve precision, but it requires more storage and scoring many more representations at query time. ColBERTv2 significantly reduced the space requirements compared with the method's first generation.
The same multiple-representation principle can be used for different fields, different languages, or combinations of text and images.
Contextual Retrieval, described by Anthropic in 2024, adds a short explanation derived from the complete document to each chunk before indexing. In this way, a passage stating that "company revenue grew by 3%" can retain information about the company, period, and report.
In its own evaluation, Anthropic reported that contextual embeddings combined with contextual BM25 reduced the retrieval failure rate within the top 20 chunks from 5.7% to 2.9%. After reranking 150 candidates and retaining the top 20, the rate reached 1.9%, a relative reduction of 67%. These results come from the vendor's evaluation and do not guarantee the same performance on every corpus.
More advanced systems can:
HyDE is a zero-shot dense retrieval method that does not require relevance labels. It generates a hypothetical document, then uses its representation only to find similar real documents. The hypothetical text may contain false details and must not be used as evidence. Multiple queries and question decomposition can improve coverage, but they may increase cost and introduce off-topic results.
These techniques should be enabled after evaluation. Not every question requires five reformulations and several rounds of search.
An agent should not send every request to the same vector database. It can select the appropriate tool for the type of information requested.
| Tool | Best suited to | Example |
| :--- | :--- | :--- |
| Lexical search | Identifiers and exact wording | Finding an error code in a runbook |
| Vector search | Concepts and paraphrases | Finding a procedure described in different terms |
| Hybrid search and reranking | Mixed corpora and real-world questions | Technical documentation, policies, and support |
| SQL or operational API | Exact values and current state | Inventory, invoices, orders, or metrics |
| Knowledge graph | Relationships and questions requiring multiple steps | Connections among incidents, vendors, and components |
| Web search | Recent public information | Rules, documentation, and market information |
| Document system or object storage | Documents and their original permissions | Nextcloud, SharePoint, Drive, or S3 |
| Multimodal tool | Scanned pages, tables, and diagrams | Manuals, invoices, and complex reports |
| Persistent memory | Selected preferences and decisions | Preferred report format |
| Action tool | Changing a system | Creating a ticket or updating a CRM |
When an agent looks for the return policy in a manual, it uses retrieval. When it checks an order status, it uses the store's API. When it initiates a refund, it performs an action that requires permissions, limits, and, depending on the risk, human approval.
Model Context Protocol can expose sources and tools in a common format. However, MCP is not a RAG engine. It can provide the agent with a search tool, a fetch tool, a database query, or a business action, and the application decides how they are used.
Increasingly large context windows do not automatically eliminate retrieval. Including a small number of complete documents may be the simplest solution when the information fits into the context at a reasonable cost and relationships between sections matter.
For large collections that change frequently or have different permissions, RAG retains clear advantages: it selects information, reduces the volume sent to the model, enables filtering, and can link the answer to its source.
The Lost in the Middle study showed, for the tasks and models evaluated, that performance often declines when relevant information is located in the middle of a long context. This finding should not be generalized automatically to every current model. LaRA, published at ICML 2025, found that the choice between long context and RAG depends on model capability, context length, task type, and retrieval quality, which supports evaluating and routing between the two approaches.
The pragmatic approach is to measure three variants: direct context, RAG, and a hybrid solution that routes the question to the appropriate method.
An agentic system can decide whether it needs retrieval, select the source, decompose the question, run searches in parallel, assess result relevance, and repeat the search.
This flexibility is useful for research and questions that require several steps. However, it adds steps, latency, cost, and new points where errors can occur. The system needs limits for the number of steps, permitted sources, budget, and time.
According to the Azure AI Search documentation for agentic retrieval, the extractive component is generally available through the stable 2026-04-01 API, while LLM-based planning, synthesis, and some multi-turn conversation capabilities continue to use 2026-08-01-preview. This distinction is a good example of the uneven maturity of individual components.
Self-RAG trains the model to decide when to retrieve and uses reflection tokens to assess the evidence and its own generation. Corrective RAG uses a result evaluator and can trigger corrective steps, including web search. Adaptive-RAG uses a complexity classifier to choose among answering without retrieval, a single retrieval step, and iterative retrieval. These are important research directions, but implementations described in papers should not be confused with a universal option that can simply be enabled in a product.
Classic RAG finds chunks similar to the question. Some questions, however, require a view of the entire corpus: recurring themes, relationships among organizations and events, or connections among incidents, components, and vendors.
GraphRAG, developed by Microsoft Research, extracts entities and relationships, builds communities, and generates summaries of them. The original paper evaluates primarily global questions about themes and patterns across the entire corpus. Microsoft's implementation also documents local search methods separately. RAPTOR builds a hierarchy of chunks and recursive summaries.
These methods may suit investigations and cross-cutting analyses, especially in corpora where relationships are essential. Building and updating the structure adds cost and complexity. GraphRAG is not an automatic recommendation for an FAQ, a catalog, or finding an exact clause.
Real documents contain more than text: tables, charts, technical drawings, images, formulas, and complex visual structures. A mature approach combines OCR, structural parsing, and descriptions of visual elements while preserving the connection to the original page.
One recent direction indexes page images directly. ColPali is a page-level visual retrieval model based on multiple representations. VisRAG is a complete workflow for visual indexing, retrieval, and generation that does not first convert all content into text. Both papers were published at ICLR 2025.
Commercial services have begun to include multimodal parsing and retrieval. Amazon Bedrock Knowledge Bases documents multimodal workflows for text, images, audio, and video, while Gemini API File Search announced multimodal processing and page-level citations in 2026.
For a business, the prudent approach is to preserve the text, document structure, and page provenance, then add visual retrieval where tables and images change the answer.
"Local" and "cloud" do not automatically determine security or quality. They describe how control and responsibility are distributed.
| Deployment model | Advantages | Responsibilities and trade-offs |
| :--- | :--- | :--- |
| On-premises | Direct control over documents, indexes, and logs; option to use local models | The team manages authentication, updates, backups, monitoring, scaling, and security for the entire technical stack |
| Self-managed in a private cloud | Architectural control and strong integration with existing infrastructure | Requires operational expertise and a clear cost and availability model |
| External managed service | Faster pilot implementation and scaling; less infrastructure to operate internally | Retention, data use, region, export, deletion, costs, and vendor dependency must be reviewed |
| Hybrid | Documents and permissions can remain on your own infrastructure, while only the necessary chunks are sent to the external model | More complex architecture; the data crossing each trust boundary must be tracked |
A local vector engine does not necessarily require a GPU. Resource requirements depend on index size, desired latency, and the models used for vectorization and reranking. Local generation with a large model is a separate problem from storing and searching vectors.
Products should be compared within the same category:
| Layer | Role | Examples |
| :--- | :--- | :--- |
| Search and storage engine | Lexical and vector indexes, filtering, and ranking | pgvector, Qdrant, Weaviate, Milvus, Vespa, OpenSearch, Elasticsearch |
| Managed RAG service | Vendor-managed ingestion, indexing, and retrieval | OpenAI File Search, Azure AI Search, Bedrock Knowledge Bases, Google Agent Search, and RAG Engine |
| Embedding and reranking models | Semantic representation and reranking of candidates | OpenAI, Cohere, Voyage, Jina, and open-source models |
| Orchestration framework | Connectors, workflows, retrieval mechanisms, agents, and evaluation | LlamaIndex, LangChain and LangGraph, Haystack |
| User interface or complete application | Document, chat, and agent experience | AnythingLLM, RAGFlow, Open WebUI, Dify, Khoj |
| Generative model | Formulating the answer from the context | Local models or services from external providers |
PostgreSQL with pgvector is a pragmatic choice when the company already uses PostgreSQL and the corpus is moderate in size. Data, metadata, permissions, and vectors can remain in the same database. Ingestion, hybrid search result fusion, and reranking need to be assembled around the extension, in SQL and/or in the application.
Qdrant is a dedicated engine for dense vectors, sparse vectors, and multiple representations, with filters and hybrid queries. It can run locally or as a managed service and is suitable when retrieval becomes a separate architectural component.
Weaviate combines vector search, BM25F, and hybrid search, with options for multitenancy and integration modules. It is more integrated, but requires careful management of collections and resources.
Milvus offers Lite, Standalone, and Distributed variants, as well as the Zilliz Cloud service. The distributed version is generally justified by large volumes or availability and scaling requirements, not as the default choice for an SME's first project.
OpenSearch and Elasticsearch are natural options when the organization already uses these ecosystems for search and analytics. Both combine lexical search with vector capabilities, but operating the cluster and tuning relevance require experience.
Vespa is suitable when multi-stage ranking and business signals are central product capabilities. Its flexibility comes with a steeper learning curve.
OpenAI File Search is a hosted tool for the Responses API, built on Vector Stores. It manages files, chunking, indexing, semantic and lexical search, filters on file attributes, and file-level citations. It offers a short path to a pilot when the application already uses the OpenAI ecosystem, with less control over the internal mechanism than a custom architecture.
Azure AI Search provides full-text, vector, and hybrid search, semantic ranking, OCR, and integrated vectorization. It is especially relevant in the Azure and Microsoft Entra ecosystem. Direct SharePoint integration and access control list propagation need to be assessed separately because some capabilities remain in preview. The same applies to agentic features that have not reached general availability.
Amazon Bedrock Knowledge Bases offers a Managed Knowledge Base, in which the service manages ingestion, indexing, storage, and retrieval, as well as a Customer-managed Knowledge Base, in which the customer controls the ingestion workflow and vector database. Some capabilities, including certain third-party connectors and access control list filtering, are available only in the managed option. The RetrieveAndGenerate workflow can produce answers with citations, while Retrieve returns the retrieved results. Model and feature availability depends on the region.
Agent Search on Gemini Enterprise Agent Platform, the current name for the product previously known as Vertex AI Search, is designed for search across websites and documents. RAG Engine on Gemini Enterprise Agent Platform is intended for custom applications and agents. Some modes and features still have preview or regional limitations.
Pinecone is a managed engine for semantic and hybrid search, with filtering, namespaces, and hosted embedding and reranking models. It reduces operational work, but does not replace application ingestion, authorization, and evaluation.
Cohere provides models and APIs for semantic vectorization, reranking, and document parsing, but it is not primarily a vector database. A Cohere reranking model can be used on results retrieved from pgvector, Qdrant, OpenSearch, or other engines.
LlamaIndex provides connectors, ingestion, indexes, retrieval mechanisms, query engines, workflows, and agents. The LlamaParse platform adds hosted document parsing and processing services.
LangChain and LangGraph provide components and control for custom applications, including agents that decide when and where to search. The large number of integrations is useful, but can produce an architecture that is difficult to follow unless clear boundaries are maintained.
Haystack builds modular workflows from components, document stores, retrieval and reranking mechanisms, agents, and tools. It is suitable when the team wants explicit control over the stages and the option to change vendors.
A software framework accelerates implementation. It does not decide which documents are authoritative, which permissions apply, or what level of quality is acceptable.
A RAG system brings company documents into an operational workflow where a model interprets natural language. This integration must be treated as a new security boundary.
A document, email, or website may contain malicious instructions addressed to the model. OWASP describes indirect prompt injection as a situation in which the model receives external content that can alter its behavior.
Useful measures include:
Delimiters and system prompts reduce risk, but are not sufficient security controls on their own.
Authorization must be enforced at retrieval time, at the level of a chunk, document, or authorized source. Caches must be isolated or keyed by the complete relevant authorization context, including tenant, user, groups, roles, and access control list version. The system must undergo deliberate cross-access testing.
Deleting the original document must propagate to chunks, vector representations, secondary indexes, caches, and pre-generated results, according to the retention policy. Every source needs a stable identifier, version, effective date, and status.
Vector representations should not be treated as a guaranteed form of anonymization. The research paper Text Embeddings Reveal (Almost) As Much As Text shows why simply transforming text into a vector does not automatically eliminate exposure risk. Only necessary data should be indexed, and personal information and secrets should be removed or pseudonymized where the intended purpose permits it.
Contracts with external providers must address retention, the use of data for training, processing regions, deletion, export, and subprocessors. "Data is not used for training" does not automatically mean "zero retention."
A hybrid architecture can retain documents, access control lists, and retrieval on the organization's own infrastructure, sending only strictly necessary and, where possible, masked chunks to the external model.
The OWASP RAG Security Cheat Sheet provides a practical checklist for ingestion, embeddings, access, provenance, caching, monitoring, and deletion. The dedicated article on prompt injection will examine these risks in detail.
A demonstration is not an evaluation. A pilot needs a set of real questions, expected sources, and clear conditions for cases in which information is missing.
Evaluation should be broken down by layer:
| Layer | Question being evaluated | Example metrics |
| :--- | :--- | :--- |
| Retrieval | Does it find the necessary information? | Recall@k, Hit Rate@k |
| Ranking | Does it place useful evidence before noise? | Precision@k, MRR, nDCG@k |
| Generation | Does it answer correctly and completely enough? | relevance, correctness, completeness |
| Faithfulness | Are claims supported by the context? | faithfulness, groundedness, claim-level verification |
| Citations | Do sources support and cover important claims? | citation correctness, citation completeness |
| Abstention | Does it refrain from inventing when evidence is missing? | correct refusal rate and unsupported answer rate |
| Operations | Is it sustainable in production? | p50 and p95 latency, cost, tokens, error rate |
| Security | Does it respect access and withstand hostile content? | cross-organization isolation, prompt injection resistance, and deletion propagation tests |
Ragas and ARES provide methods for evaluating context relevance, faithfulness, and answer quality. RAGChecker breaks answers into claims and attempts to diagnose retrieval and generation separately.
Definitions of faithfulness, groundedness, and citation correctness vary across evaluation frameworks. Scores are not directly comparable unless we use the same rubric, test set, and type of evaluator.
Evaluators based on other models are useful for rapid comparisons, but they must be calibrated against human-evaluated examples. An automated score does not replace review of important cases.
An SME's test set should include:
For each case, retain the question, role, expected sources, mandatory claims, acceptable abstention behavior, and the severity of an error.
RAG is not a mandatory stage for every AI application.
It may be unnecessary or disproportionate when:
Sometimes the first useful project is not a RAG chatbot, but organizing documents, eliminating duplicates, establishing the official version, and implementing access controls.
A good first pilot answers a clear set of questions from a bounded corpus. Examples include approved internal procedures, documentation for a product family, or the runbooks for a service.
Identify the owner, version, and update frequency. Documents with no known authority or status are not included in the index automatically.
Start with structural parsing, metadata, lexical and vector search, filters, result reranking, and citations. Do not introduce GraphRAG or agentic loops until measurements identify a limitation in the baseline.
Collect real questions, expected answers, and cases with no answer. Measure retrieval separately from final answer formulation.
Verify role and tenant isolation, expired documents, deletion, prompt injection, conflicting sources, and the refusal to invent.
Evaluate at least one local or self-managed option and one managed option, using the same documents and questions. Base the decision on quality, total cost, latency, control, operations, and risk, not on the number of features in a presentation.
The first users can see the sources, flag incorrect answers, and have a clear escalation path. Expand the corpus and autonomy only after measurements show that the system remains useful and controllable.
The first step is not choosing a vector database, but defining a real problem and the sources that can solve it. Together, we can assess which data is worth connecting, which search method is appropriate, and whether the solution should run on your own infrastructure, through cloud services, or in a hybrid configuration. We can then define a measurable pilot project with controlled access and clear quality criteria.