How AI can cite company knowledge

RAG research combines knowledge stored in model parameters with an external retrievable source. For your business, this means finding suitable approved information first and then drafting an answer from it.

If retrieval returns an outdated or unsuitable document, even a well-written answer may be wrong. A correct source can also be represented inaccurately. The system needs to make both steps traceable.

Not every accessible folder belongs in the knowledge base. Define which content is approved, who owns it and which groups may read it. Drafts and superseded versions need an unambiguous status.

Indexing must account for changes, new versions, deletions and permission updates. Otherwise zombie documents remain available after they have been deleted or restricted at the source. Synchronisation failures need a visible status and a defined response.

• Keep the original system and a stable document identifier.
• Record version or hash, modification date and validity status.
• Document successful synchronisation, failures and deletion status.
• Synchronise permissions and their changes with the content.
• Provide links back to the authoritative original.

PDFs, tables and multi-column documents are not uniform blocks of text. Check real examples: Do values still belong to their columns, are headings preserved and do repeated headers appear as factual content?

Smaller retrieval sections, called chunks, should form understandable units: a process step, a question with its answer or a table row with column names. A fixed character count alone is not a complete strategy. Very small sections lose context; large ones can contain irrelevant information.

Preserve titles, sections or pages, versions, language, product context and permissions on each chunk. Without this metadata, results are difficult to check and changed documents are difficult to update.

A prompt saying “Do not disclose secrets” is not access control. The application must reliably identify the person and restrict retrieval to authorised documents. Permissions must also apply to individual chunks and derived content.

The security chain is authentication → authorisation → filtered retrieval → answer. Microsoft documents query-time permission filtering, for example. Such mechanisms depend on correctly synchronised permissions and trusted user context; their availability does not automatically make an application secure.

Also examine caches, logs and source links. When permissions change, previously generated or cached answers must not bypass access restrictions. An unavailable source is not a reason to relax permissions.

Semantic search helps with different wording. Keyword search can matter for product numbers, versions and names. Test whether combining the two improves results for your questions. Metadata and permission filters remain separate rules.

Fictional example: “Which seal does pump XZ-410 need?” requires the exact product identifier. A similar-sounding pump is not sufficient evidence. A reranker can assess candidates against the question more closely, but does not replace validity checks or permissions.

The context should contain relevant, preferably consistent passages with clear source identifiers. A 2023 study found that the position of relevant information in long contexts affected performance for the models and tasks studied. It does not establish one universal chunk size or limit for all current models.

A source list alone is insufficient: Readers need to see which passage supports an important claim. Show the title, section or page, version or date and a link to the original where access is permitted.

A real source can still fail to support a statement. Check faithfulness: Can the statement be derived from the supplied material? NIST describes how generative systems can produce false content and misleading reasoning or citations.

The system must distinguish missing results, conflicting evidence, insufficient permissions and technical errors. Document content is evidence, not trusted instructions for the system. Clear uncertainty is more useful than invented additions when sources are insufficient.

First test whether the right documents are found. Then check that passages are sufficient, valid and understandable. Finally evaluate answer relevance, faithfulness and the appropriate handling of uncertainty.

Ragas studies multiple evaluation dimensions for these systems. Automated assessments can help compare versions; they do not replace expert acceptance of important results. One successful demonstration does not validate every use.

• Known questions with expected supporting passages
• Questions with no approved answer
• Conflicting and outdated sources
• Restricted content queried by unauthorised roles
• Changed or deleted documents
• Product identifiers, ambiguous terms and faulty text extraction

Record response times, failures, feedback and the versions of the index, model and rules used. Decide which changes require another test. Logs should minimise data collection and have appropriate access protection.

A manageable architecture includes approved originals, controlled ingestion, an index with metadata and permissions, checked retrieval, answer rules and ongoing evaluation. The measure is whether a person can verify the answer, not how many documents were imported without review.