Sources, documents, and source chunks in Knowledge Bases

How the material you add to a Knowledge Base becomes documents and then source chunks, the units a build reads and your plan counts.

Everything you add to a Knowledge Base is processed into three nested units. A source is what you add, a document is one page, Sanity document, or file inside it, and a source chunk is a passage of about 1,500 characters cut from a document. A build reads and cites source chunks, and your organization's plan counts them.

A source is one thing you add to a Knowledge Base:

• Website source: one crawl, starting from a URL.
• Dataset source: one GROQ query against a Sanity dataset.
• File source: one upload, which can hold many files or an archive. Markdown or plain text you send to the Context API is also a file source.

You add, refresh, and remove material one source at a time. For what each kind of source accepts, see Knowledge Base source types.

A document is one item inside a source:

• Website source: one document per crawled page.
• Dataset source: one document per published Sanity document the query matches.
• File source: one document per file. Context expands an archive first, so each file inside it is its own document.

A long item becomes several documents. When a page, Sanity document, or file converts to more than about 40,000 characters of text, Context splits it at its headings into at most 12 sections, and each section counts as one document in place of the original. Most web pages and Sanity documents are far shorter than that, so for them one item is one document. Long PDFs and manuals are where the difference shows: a 120-page manual with chapter headings counts as about 8 documents, not 1.

Each Knowledge Base holds at most 5,000 documents across all its sources.

A source chunk is a passage of a document, about 1,500 characters long and usually no more than 3,000. Source chunks are what a build reads when it writes entries, what an entry's citations point back to, and what counts toward your organization's source chunk limit.

A short product description is usually one source chunk. A long reference page can be 20 or more.

Context processes a source in four stages when you add it, and processes anything that changed again when the source refreshes.

• Collection. Context crawls the website, runs the dataset query, or unpacks the upload, and records one document per page, Sanity document, or file.
• Conversion to Markdown. Context converts every document to Markdown text, so every kind of source is read the same way:
  • Web pages keep their headings and text.
  • Sanity documents become a card: the title first, then every field the query returns, with Portable Text rendered as Markdown. Sanity system fields, such as _rev and _updatedAt, are left out.
  • PDF, DOCX, PPTX, XLSX, HTML, and AsciiDoc files have their text and layout extracted. Images have their text extracted, except images under 50 KB.
  • Markdown and plain text are kept as they are. Source code, JSON, CSV, YAML, and other text formats are kept verbatim in a code block.
  • A file Context can't read becomes a short card with its name, type, and size, so the build still knows the file exists.
• Splitting long documents. A converted document longer than about 40,000 characters is split at its headings into at most 12 sections. From then on, each section counts as a document. A long document with no headings stays whole.
• Chunking. Context cuts the Markdown at its headings, joins short neighboring sections until a chunk reaches about 1,500 characters, and splits sections longer than 3,000 characters at paragraph breaks. A single line longer than 3,000 characters, such as a long table row, stays whole.

Chunking is done by code, not a model, so the same text always produces the same source chunks.

A source's source chunk count follows the amount of text it holds, so the most direct way to use fewer source chunks is to send less text:

• Narrow a dataset query's projection. *[_type == "product"]{title, description} sends two fields per Sanity document. A projection that follows references with -> copies the referenced documents' fields into every document that points at them, which adds source chunks.
• Scope a crawl to the pages agents need. A documentation section usually beats a whole domain.

Two limits apply to documents and source chunks:

• Documents per Knowledge Base: at most 5,000, across all of the Knowledge Base's sources. An import that would go past it fails with a message that starts This import would add and says how much room the Knowledge Base has left.
• Source chunks per organization: one pool, set by your organization's plan and shared by all its Knowledge Bases. You decide how to spread it across them.

Only source chunks a build has taken in count toward the organization's source chunk limit. A Knowledge Base that has never been built, and documents added since a Knowledge Base's last build, count nothing until the next build.

Each build decides what fits when it starts. Documents the Knowledge Base already took in stay in, even past the limit, so a document that grows or a plan downgrade never drops a page from a built Knowledge Base. New documents go in the order they were created, each one if it fits in the room left, and a document that doesn't fit is excluded from the build. Two errors mean there's no room at all:

• When nothing fits, the build fails with None of this knowledge base's documents fit under the organization's plan limit for source chunks.
• A website crawl that starts with no room fails with The organization's plan limit for source chunks is reached.

To make room, remove sources or upgrade the plan. For each plan's limits and everything that happens when the organization reaches them, see Context plans and limits.

A Knowledge Base for a product company has three sources, which produce the following documents and source chunks.

This Knowledge Base holds three sources, 1,608 of its 5,000 documents, and about 5,040 source chunks, which count toward the organization's source chunk limit once the Knowledge Base is built.

These numbers are approximate. A page's count depends on how much text it holds, and the Context app shows the exact count for every source once it's processed.

• Knowledge Base source types. What each kind of source accepts and how it's scoped.
• Context plans and limits. Each plan's source chunk limit, how builds decide which documents count, and what happens when the organization reaches the limit.
• Quick start: build a Knowledge Base and ask it a question. Add a source and run the first build.
• Keep a Knowledge Base current. How sources refresh and when changed material is processed again.