AI: RAG and chunking - how the assistant finds what to read

What RAG is, why text must be cut into chunks, the main ways to cut it with a worked example of each, how pieces are scored, and how {{brand}} applies them - step by step, with pictures.

In short

A language model knows nothing about your workspace. To answer a question about your cards or test runs, {{brand}} first finds the facts, then hands them to the model together with the question. This pattern is called RAG - retrieval-augmented generation. The facts travel as small pieces of text called chunks; in {{brand}} they are called snippets.

After this lesson you can Explain RAG in one minute, choose a chunking strategy for any kind of text, explain why {{brand}} uses full-text search instead of a vector database, and build a tiny RAG pipeline yourself.

Words you need

RAG Retrieval-augmented generation: find the facts first, then let the model write an answer from them.
Token The unit a model reads: a piece of a word, about 4 characters of English. Prices and limits are counted in tokens.
Context window How many tokens a model can read at once - rules, question, facts and answer together.
Chunk (snippet) A small, self-contained piece of material, cut so it fits the window and means something on its own.
Retrieval and ranking Retrieval finds the chunks that might answer the question, before the model is asked anything. Ranking gives each a score against the question and keeps the best.
Embedding A list of numbers that stands for the meaning of a text. Similar meanings get similar numbers.
Vector database A database that finds the embeddings closest to the question's embedding, for example pgvector in PostgreSQL.
Full-text search Search by words, with stemming (cutting a word to its root, so failing finds failed) and a relevance score. Built into PostgreSQL.
Grounding Telling the model to answer only from the chunks, and to say it does not know otherwise.
Top-k How many of the best-scoring chunks you keep, for example the top 5.
Recall and precision Recall: did the right chunk make it into the candidates. Precision: how many of the kept chunks are actually useful.
Hallucination A fluent but invented answer. It is more likely when the facts the model needs are missing.

How RAG works, step by step

Most RAG systems follow these five stages, sometimes with extra steps such as re-ranking. Learn them and you can read almost any RAG design.

1. Prepare the material Cut documents and records into chunks. Many systems do this once, in advance, and store the chunks in an index; others cut at question time. Both work - the choice depends on how often the data changes and who may see it.
2. Retrieve candidates Collect chunks that might answer the question - by words (full-text search), by meaning (embeddings), or both.
3. Rank and cut Score every candidate against the question and keep the best ones that fit the budget.
4. Ground the prompt Put rules first, then the chosen chunks, then the question: answer only from this material.
5. Answer with sources The model writes the answer and names the chunks it used, so a person can check it.

Why text must be cut at all

Room The context window is limited. A workspace with 400 cards may hold 200,000 words; a question needs a few hundred of them.
Cost Every token read is paid for, on every question. Sending everything costs far more than the answer is worth.
Focus Models answer better from a few relevant chunks than from many loosely related ones. Less noise means fewer invented answers.
Permissions A chunk comes from one record, so it is easy to leave out what the person asking may not see. That is a permissions rule you can check chunk by chunk.

The ways to cut text - with an example of each

Take one Help guide as the example: a title, an In short section of two paragraphs, a Steps section with five numbered steps, and a Good to know section with a table. Here is what each chunking strategy does to it, and what a chunk ends up holding.

A length limit is fixed-size cutting applied to only the opening of one long text. {{brand}} therefore uses four cuts: by structure, one record per piece, a summary and a length limit.

How big should a chunk be? A common starting point is 200-500 tokens per chunk and 10-20% overlap for plain prose. Always test on your own questions: too small loses context, too large adds noise and cost.

The options in the market

Fixed size (overlap optional) Every N tokens, with or without a few tokens repeated at each cut.; Simplest to build; Predictable piece sizes; Overlap keeps the text near a cut; Can cut a list of steps in half; A piece may start mid-sentence; Overlap means more pieces and the same text twice
Sentence or paragraph Cut at the end of a sentence or paragraph.; Pieces read naturally; A one-line step becomes a tiny piece with no context
By structure One piece per heading. Used for documentation guides.; Keeps the author's own units; The steps stay together; Needs text that has headings
Semantic An embedding model finds where the topic changes and cuts there.; Best for long prose without headings; A model call per document; Needs an embedding model; results change when the model changes
One record, one piece Used for cards, notes, designs and conversations.; Perfect for structured data; Labels tell the model what each fact is; Only for records, not for documents
Summary of many records Used for boards, lists of tests and runs.; Answers how-is-it-going questions without reading every row; The database does the counting; Detail is lost - a question about one row needs a piece of its own

Full-text search Used today; Exact on names, statuses and dates; Built into PostgreSQL - nothing extra to run; Permissions checked live; Matches words, so synonyms need extra help (a synonym list or embeddings)
Embeddings + vector database Not used today; Finds meaning, not just words; Great for long notes and documents; Costs a model call per piece; A second copy of customer text to secure and keep in step; Stale until re-embedded
Hybrid Both searches, results merged; Best recall; Most moving parts

What {{brand}} chose, and why

{{brand}} cuts text by material and finds the pieces with full-text search rather than embeddings. The reasons, the price paid and the cases where you should choose differently are below.

Most questions are about rows - which cards are overdue, which runs failed - and an exact word search answers those better than similarity.
There is no second copy of customer text: pieces are built when the question is asked, so there is nothing to keep in step.
Permissions are checked live, as the person asking, every time.

Synonyms are missed: money owed will not find unpaid invoices unless a word matches.

You have thousands of long documents with no headings.
Users phrase questions very differently from the text, so words alone do not match.
You can afford to keep an index in step with permissions.

The four ways {{brand}} uses, and where

The same four techniques apply to any product: documents, rows, groups of rows and long text.

Why labelled lines matter A model reading Due: 2026-10-24 knows it is a date the card is due. The same date with no label could be a creation date or a deadline. Labels make chunks unambiguous, and the number in front lets the answer cite the piece it used.

How the pieces are scored and chosen

Help sections are searched first Written guides are the same for everyone, so a simple keyword score finds them: a word in a guide or section title counts far more than one in the text.
Give every source a fair share One source with thousands of rows would otherwise crowd out the rest, so each source gets part of the budget.
Fetch more candidates than you will keep Ranking needs a real choice. If a source returns only what you will keep, ranking has nothing to decide.
Score by words PostgreSQL full-text search scores the pieces: it turns each into stems (failing, failed, fail all become fail), weighs the title above the body, lets any word of the question match, and counts how close the question's words stand together.
Always keep the record on screen If the question is asked while a record is open, what its source says about that record goes first and is never cut, whatever it scores.
Keep the best that fit Count pieces, not words, when pieces are similar in size. The best-scoring pieces fill the budget, then go back in the order their source gave them, because that order - failing first, newest first - means something too. The prompt reads: the record on screen, then the guides, then the ranked rows.

The | sign means any word may match, and the where line drops pieces that match nothing. In a real index you store the tsvector in a column and add a GIN index so this stays fast. BM25 is the best-known classic word-scoring formula; PostgreSQL's ranking is a related idea. {{brand}} applies this pattern.

Nothing is indexed in advance {{brand}} builds pieces at question time, as the person asking, inside their organization, and does not keep a separate searchable copy of their data. So there is nothing to keep in step with permissions.

Reusing answers safely

Asking a model costs money, so identical questions should not be paid for twice. The safe way is to key a stored answer on the pieces that were sent, not on the question text alone. Answers can be reused only when the person's own retrieval returns exactly the same pieces, so a reused answer never crosses a permission.

Retrieve as the person asking Retrieval runs every time, with that person's rights.
Fingerprint the pieces Compute a hash over the exact pieces chosen.
Look up a stored answer The key is the question plus the fingerprint. Someone who sees different records gets a different fingerprint, so a different key.
Found: skip the model Only the model call is saved. Not found: ask the model, then store the answer under that key.

Embeddings explained - and why they are not used today

An embedding model turns text into a list of numbers - often 768 to 3,072 of them - so that texts with similar meaning sit close together. The question is turned into numbers too, and a vector database returns the closest pieces. It finds meaning even when words differ: money owed finds unpaid invoices.

Embed every chunk once Turn each chunk into a vector and store it.
Embed the question The question becomes a vector with the same model.
Measure closeness Use cosine similarity: 1 means the vectors point the same way, so the meaning is the same.
Find the nearest ones fast An approximate index such as HNSW finds the nearest vectors without comparing against every one.
Optionally rerank A stronger model re-reads the top 20 or so and puts the best first.

Hybrid search runs both searches and merges the two ranked lists, for example with reciprocal rank fusion. It gets the best recall, with the most moving parts.

Most questions people ask {{brand}} are about rows: which runs failed, which cards are overdue, which endpoint is slowest. An exact query answers those better than similarity. A meaning-based retriever could be added next to the word-based one, but the ranking step would also have to merge the two result lists, for example by rank fusion.

How to know it works

A RAG system that looks right can still miss the right chunk half the time. You find out with a golden set: questions whose right answers you already know.

Write 20 real questions For each, note the chunk that should answer it.
Test retrieval alone Count how often the right chunk is in the top 5. This number is recall@5.
Test the full pipeline Check that each answer cites a chunk, and that questions with no answer get an I do not know.
Change one thing Chunk size or top-k, never both at once, then run the set again.
Keep the set It becomes a regression test that catches the next change that makes answers worse.

recall@k The share of questions whose right chunk is among the top k results.
Golden set A fixed list of questions with known right chunks, run again after every change.

Try it yourself: a mini RAG in six steps

You can build the core of RAG in an afternoon, with free tools only: SQLite, any language, and a local model such as Ollama or any API free tier. This is the exercise interviewers love to see you understand.

Pick 20 documents For example 20 Markdown pages with headings.
Chunk by heading Split each page at ## headings; keep the page title in every chunk so it stands alone.
Store with full-text search Put the chunks in SQLite FTS5 (free, no server) or in PostgreSQL with a tsvector column.
Retrieve the top 3 For a question, run the ranked search and take the three best chunks.
Ground and ask Send: rules, the three chunks with their titles, then the question. Ask for the titles it used.
Check Ask one question the documents do not answer and confirm the model says it does not know. Ask one they do answer and confirm the cited piece is the right one.

Common mistakes

Chunks that do not stand alone: a piece that says Click Save without saying where. Keep the title or record name in every chunk.
Too many chunks, or the key fact buried: fifty loosely related pieces make the model guess, and models often attend best to the start and end of a long prompt. Keep few chunks, put the rules first and the question last.
Forgetting permissions: an index built for everyone will one day answer someone with a record they may not see.
No sources and no way to say I don't know: an answer that cites nothing cannot be checked, and without that rule a model fills the gap with a confident invention.
Never testing retrieval: change the chunk size or the ranker without a question set and you will not know whether it got worse.
One size fits all: cutting tables, code and prose the same way. Choose the cut by material.

Interview questions

What is RAG? Retrieve the facts first, then ask the model to answer only from them, with sources. It gives a model knowledge it was never trained on.
What is a chunk, and why cut text at all? A small, self-contained piece. The window is limited, every token costs money, and three relevant pieces beat fifty vague ones.
How do you choose a chunking strategy? Match the cut to the material: headings for documents, one chunk per record for data, summaries for groups, a length limit for very long text. Then test on real questions, because chunk size changes recall. Start around 200-500 tokens.
Why might you choose full-text search over embeddings? When questions are mostly about rows, names and dates, exact search wins, and a vector store is a second copy of the data whose permissions can drift. Embeddings pay off when users phrase questions very differently from the text.
How do you keep a RAG system from leaking data between users? Build the pieces as the person asking, so anything they may not see is never retrieved. Do not index first and filter later.
How do you stop one source crowding out the rest? Split the budget per source before ranking, then rank all candidates against the question.
What is the difference between recall and precision in retrieval? Recall asks whether the right chunk made it into the candidates; precision asks how many of the kept chunks are useful. Fetching more candidates raises recall, and ranking and a small top-k raise precision.
When would you use hybrid search? When users mix exact terms (names, codes) with loosely worded questions. Run a word search and a meaning search, then merge the two ranked lists, for example with reciprocal rank fusion.
How do you reduce hallucinations in RAG? Tell the model to answer only from the chunks and to say it does not know otherwise. Cite sources so answers can be checked, and send fewer, better chunks.
How do you evaluate a RAG system? Keep a golden set of questions with known right chunks. Measure recall@k for retrieval, then check each answer cites a chunk and refuses when nothing covers the question.

/help/technology/ai-models
/help/technology/ai-safety
/help/technology/search-reports