Making Your Company's Knowledge Actually Searchable

Every established business has the same hidden asset and the same hidden cost. Somewhere across a shared drive, an email archive, a project management tool and three generations of intranet sit the answers to most of the questions your team asks each week. Policies, past quotes, scope documents, technical specifications, meeting notes, the reasoning behind a decision made four years ago.

Almost none of it is findable. So people ask a colleague instead, and the colleague — usually one of your most experienced and most expensive staff — stops what they're doing and answers the same question they answered last month.

This is the problem retrieval-based AI search solves, and it's the one we'd recommend most businesses tackle before anything flashier. It's also the foundation nearly every other useful AI feature is built on.

How it actually works

The technique usually goes by the acronym RAG — retrieval-augmented generation. The name is unhelpful; the idea is simple.

A language model on its own knows general things about the world but nothing about your business. Ask it about your refund policy and it will produce something plausible and wrong. That's not a flaw you can prompt your way out of — it genuinely doesn't have the information.

So you give it the information at the moment of the question:

1. Index your content. Documents are split into passages and converted into a numerical representation that captures meaning, then stored in a searchable index.
2. Someone asks a question. That question gets converted the same way.
3. Retrieve the relevant passages. The system finds the handful of passages most likely to contain the answer — matching on meaning rather than exact words, so "how long do we keep client files" finds a policy that says "records are retained for seven years".
4. Answer from those passages. The model receives the question and the retrieved text, with instructions to answer only from what it's been given and to say so when the answer isn't there.
5. Cite the sources. Every answer links back to the documents it came from.

That last step is what makes the difference between a system people trust and one they abandon. When a user can click through and verify, an occasional imperfect answer is a minor annoyance. Without citations, one confident fabrication destroys confidence in the whole thing.

The retrieval is the hard part

There's a persistent misconception that these systems live or die on the model. They don't. If the right passage isn't retrieved, no model can answer correctly — it will either decline or invent something. Most of the engineering effort, and most of the quality difference between a good system and a poor one, sits in retrieval.

What that work involves:

Splitting documents sensibly. Chop a document into arbitrary 500-word blocks and you'll cut tables in half and separate headings from the text they introduce. Splitting along the document's actual structure — sections, clauses, headings — makes a dramatic difference.

Keeping context with each passage. A passage reading "this must be approved by two directors" is useless without knowing which policy and which section it came from. Carrying that context into the index is cheap and hugely effective.

Combining search methods. Semantic search is excellent at meaning and surprisingly poor at exact identifiers — part numbers, client codes, acronyms. Traditional keyword search is the reverse. Running both and merging the results handles far more real questions than either alone.

Re-ranking. Retrieve twenty candidate passages, then use a more precise model to reorder them and keep the best five. A meaningful accuracy gain for modest cost.

Handling documents that supersede each other. Your 2024 policy and your 2026 policy will both match a query. The system needs to know which one is current, which usually means capturing dates and status as metadata during indexing.

Permissions are not optional

This deserves its own section because it's where these projects create real risk.

If your documents have access restrictions — HR files, salary information, client-confidential material, board papers — the search system must enforce them at retrieval time, filtered to the asking user. Not afterwards. Not by asking the model nicely to avoid sensitive topics.

The failure mode is severe and entirely predictable: someone asks an innocuous question, and the system helpfully surfaces a passage from a document about redundancies that they were never meant to see. There's no recovering from that, either technically or organisationally.

Practically, this means your index carries the same permission metadata as the source system, and every query is filtered by the current user's entitlements before retrieval runs. If your source systems have messy or undocumented permissions, sort that out first — and be aware that this is frequently the largest piece of work in the whole project.

What to index first

Don't attempt everything. Start where the questions actually are.

Good first candidates:

• Policies and procedures — high question volume, clear authority, relatively stable.
• Product and technical documentation — the questions are specific and the answers are verifiable.
• Past proposals and scope documents — enormous value for anyone writing a new one.
• Support and troubleshooting history — turns your resolved-ticket archive into a resource.

Be cautious with:

• Email archives. Vast, redundant, full of superseded thinking and off-hand remarks. High noise, difficult permissions.
• Meeting notes. Often ambiguous about what was decided versus merely discussed.
• Anything you can't establish authority for. If nobody can say whether a document is current, its answers will be wrong in ways nobody can catch.

A good way to choose: look at what people ask in your internal chat channels. The recurring questions tell you exactly which corpus to index.

Measuring whether it works

"It seems good" is not a standard. Build an evaluation set before you build the system — 50 to 100 real questions with known correct answers and known source documents. Then you can measure:

• Retrieval hit rate. How often does the correct source document appear in the retrieved passages? If this is below about 85%, fix retrieval before touching anything else.
• Answer accuracy against the known answers.
• Refusal behaviour. When the answer genuinely isn't in the corpus, does the system say so? Test this deliberately with questions you know it can't answer.
• Citation correctness. Does the cited source actually support the claim? An answer with a wrong citation is worse than no citation.

Run this suite whenever you change anything. Without it, you'll make a prompt tweak that improves one case and quietly degrades ten others — a trap we describe in more detail in our post on adding AI to an existing product.

Keeping it current

A knowledge system that answers from last year's documents is worse than no system, because people will act on it.

• Re-index on change, ideally triggered by the source system rather than on a nightly sweep.
• Remove deleted content promptly. Orphaned passages in an index are a genuine liability.
• Surface document dates in answers. Let users see they're reading something from 2023.
• Watch the refusal rate. A rise usually means people are asking about something you haven't indexed — useful product feedback.

What it costs

• A focused internal knowledge search over a defined document set typically runs $25k–$60k to build, with permissions complexity the main variable.
• Running costs are modest — indexing is cheap, and query costs are cents.
• Ongoing work is mostly keeping the index synchronised and expanding the corpus.

The return is easier to estimate than most AI projects. Ask a few senior people how long they spend each week answering questions others could self-serve, and how long new starters take to become productive. Those two numbers usually make the case on their own.

---

Sitting on a decade of documents nobody can find? Tell us what your team keeps asking and we'll scope what indexing it would involve — including an honest read on whether your permissions are in a fit state to build on.