Back to Introducing the Vidbyte SDK
July 27th, 2026

Document Q&A Agent

Documents

Citations

Managed context

Sessions

8 min read

The problem

The answer is already in a folder somewhere, but finding it means opening nine files, and asking a chat model means getting an answer that sounds right and cites nothing.

How it goes today

  • You keyword-search the folder, open the near-misses, and read until you find the clause.
  • Pasting a document into a chat model works once, then the next question needs the same paste.
  • When the documents do not cover the question, the model answers anyway from general knowledge.

What you'll build

  • An agent with read and search tools locked to your documents folder.
  • A typed answer carrying the file paths it used and how confident it is.
  • A durable session, so follow-up questions continue the same thread instead of starting over.

Done looks like

  • A supported question comes back with the file paths the answer was taken from.
  • An unsupported question comes back as a refusal with an empty sources list.
  • A forty-page policy file never sits raw in the context window.
  • Asking a follow-up does not require re-supplying anything.

Paste into Claude, Codex, Cursor, or another coding agent to scaffold this harness.

SDK example

Markdown
# Context: Vidbyte SDK Vidbyte SDK is a Python package (import as `vidbyte`) for building agent workflows yourself. Use it when you want to own the agent loop: system prompts, model/provider choice, tools, managed context, middleware, pipelines, tracing, evals, and multi-agent composition. Core mental model: 1. Create an Agent or BaseAgent. 2. Attach a system prompt, provider/model, optional tools, context, middleware, and runtime choices. 3. Call run() or arun(). 4. Let the SDK handle message assembly, tool schemas/calls, iteration limits, and optional pipelines. Public package boundary: reusable local agent infrastructure. It does not ship private Vidbyte learning models or proprietary platform internals.

Build it step by step

Every step adds to the same file. By the end you have one runnable doc_qa.py.

  1. 01

    Decide what an answer is

    output_schema

    Putting sources in the schema is what makes a refusal legible: an empty list is a fact the caller can check, not a tone the model chose.

    + an answer that carries its own citations

    SDK example

    Python
    from typing import Literal from pydantic import BaseModel, Field class Answer(BaseModel): answer: str = Field(description="The answer, or a plain statement that the docs do not cover it.") sources: list[str] = Field(description="File paths used. Empty when nothing supported the answer.") confidence: Literal["high", "medium", "low"] = Field(description="How well the docs covered it.")
  2. 02

    Scope the tools to your documents

    FileSystemToolConfig + GrepTool

    The config leaves allow_write off, so these tools can read the folder and nothing else. Grep finds the candidate files; read_text opens them.

    + read-only search over one folder

    SDK example

    Python
    from pathlib import Path from vidbyte.tools.builtins.code_search import GrepTool from vidbyte.tools.filesystem import FileSystemToolConfig, ListDirTool, ReadTextTool DOCS = Path("./docs").resolve() docs_fs = FileSystemToolConfig(root=DOCS) # allow_write stays False doc_tools = [ReadTextTool(docs_fs), ListDirTool(docs_fs), GrepTool(DOCS)]
  3. 03

    Supply the context it always needs

    ContextManager

    Some things belong in every run: your index file and your house rules. A ContextManager holds them as structured items instead of you pasting them into the prompt each time.

    + context present on every run

    SDK example

    Python
    from vidbyte import ContextManager, FileContextItem, TextContextItem context = ContextManager([ FileContextItem.from_path(DOCS / "index.md", include_content=True), TextContextItem( title="House rules", content=( "Answer only from text the tools returned. " "Cite the file path for every claim. " "If nothing in the docs answers the question, say so and leave sources empty." ), ), ])
  4. 04

    Keep big files from flooding the window

    ContextWindow preset

    A forty-page policy read in full will crowd out everything else. This preset keeps the fact that a tool ran and what it concluded, without pinning the raw text in the window.

    + large files kept out of the window

    SDK example

    Python
    from vidbyte import ContextWindow window = ContextWindow.preset.no_raw_tool_outputs
  5. 05

    Assemble the agent

    Agent

    The schema, the tools, the managed context, and the window strategy all arrive in one constructor call.

    + the agent, wired from the four pieces above

    SDK example

    Python
    from vidbyte import Agent agent = Agent( name="document-qa", system_prompt=( "Answer questions about the user's documents. " "Search before answering, and quote only what the tools returned." ), provider="openai", model_name="gpt-5.5", tools=doc_tools, output_schema=Answer, context_manager=context, algorithm=window, )
  6. 06

    Make follow-up questions cheap

    FileSessionStore

    One call attaches the agent to a durable session that checkpoints after every turn. The agent itself stays stateless; persistence lives in the session wrapper.

    + a thread that survives the process

    SDK example

    Python
    from vidbyte import FileSessionStore store = FileSessionStore(root="./.vidbyte/sessions") session = agent.persist(store=store)
  7. 07

    Ask something covered, then something not

    reply.structured

    Both paths matter. The second question is the one that tells you whether the grounding rule actually holds.

    + the supported case and the refusal

    SDK example

    Python
    first = await session.arun("What does our policy say about data retention?") print(first.structured.answer) print("sources:", first.structured.sources) # Same session, so the agent still has the thread. follow_up = await session.arun("Does that apply to backups too?") print(follow_up.structured.confidence) missing = await session.arun("What is our office parking policy?") assert missing.structured.sources == [] # nothing in the docs supported it print(missing.structured.answer)
  8. 08

    The finished harness

    The seven steps above, in one file.

    doc_qa.py — complete

    SDK example

    Python
    """Document Q&A Agent - answers from your files, cites them, and remembers the thread.""" import asyncio from pathlib import Path from typing import Literal from pydantic import BaseModel, Field from vidbyte import ( Agent, ContextManager, ContextWindow, FileContextItem, FileSessionStore, TextContextItem, ) from vidbyte.tools.builtins.code_search import GrepTool from vidbyte.tools.filesystem import FileSystemToolConfig, ListDirTool, ReadTextTool DOCS = Path("./docs").resolve() class Answer(BaseModel): answer: str = Field(description="The answer, or a plain statement that the docs do not cover it.") sources: list[str] = Field(description="File paths used. Empty when nothing supported the answer.") confidence: Literal["high", "medium", "low"] = Field(description="How well the docs covered it.") docs_fs = FileSystemToolConfig(root=DOCS) doc_tools = [ReadTextTool(docs_fs), ListDirTool(docs_fs), GrepTool(DOCS)] context = ContextManager([ FileContextItem.from_path(DOCS / "index.md", include_content=True), TextContextItem( title="House rules", content=( "Answer only from text the tools returned. " "Cite the file path for every claim. " "If nothing in the docs answers the question, say so and leave sources empty." ), ), ]) agent = Agent( name="document-qa", system_prompt=( "Answer questions about the user's documents. " "Search before answering, and quote only what the tools returned." ), provider="openai", model_name="gpt-5.5", tools=doc_tools, output_schema=Answer, context_manager=context, algorithm=ContextWindow.preset.no_raw_tool_outputs, ) session = agent.persist(store=FileSessionStore(root="./.vidbyte/sessions")) async def main() -> None: first = await session.arun("What does our policy say about data retention?") print(first.structured.answer) print("sources:", first.structured.sources) follow_up = await session.arun("Does that apply to backups too?") print(follow_up.structured.confidence) missing = await session.arun("What is our office parking policy?") assert missing.structured.sources == [] print(missing.structured.answer) if __name__ == "__main__": asyncio.run(main())