July 27th, 2026

Document Q&A Agent

Documents

Citations

Retrieval

Grounding

6 min read

The problem

People already have the answers in folders, wikis, and export dumps — they just cannot find them. A Document Q&A Agent makes those files queryable: ask a question, retrieve passages, answer with citations. The hard rule is simple: if it is not in the docs, the agent says so.

Success looks like a short answer, a list of source files, and a clear refusal when nothing relevant was found — not a confident hallucination.

Paste into Claude, Codex, Cursor, or another coding agent to scaffold this harness.

# Context: Vidbyte SDK

Vidbyte SDK is a Python package (import as `vidbyte`) for building agent workflows yourself.

Use it when you want to own the agent loop: system prompts, model/provider choice, tools, managed context, middleware, pipelines, tracing, evals, and multi-agent composition.

Core mental model:
1. Create an Agent or BaseAgent.
2. Attach a system prompt, provider/model, optional tools, context, middleware, and runtime choices.
3. Call run() or arun().
4. Let the SDK handle message assembly, tool schemas/calls, iteration limits, and optional pipelines.

Public package boundary: reusable local agent infrastructure. It does not ship private Vidbyte learning models or proprietary platform internals.

GitHub: https://github.com/cerredz/Vidbyte-SDK
Install: pip install vidbyte-sdk
Verify: python -c "from vidbyte import Agent, BaseAgent, tool; print(Agent, BaseAgent, callable(tool))"

# Install

Primary:
```bash
pip install vidbyte-sdk
```

From a checkout (pre-release / local development):
```bash
git clone https://github.com/cerredz/Vidbyte-SDK.git
cd Vidbyte-SDK
pip install -e .
```

# Task: Build a Document Q&A Agent

Build a Vidbyte SDK agent that answers questions using the user's own documents.

Scenario: Someone drops a folder of PDFs, notes, or markdown specs and asks "What does our policy say about data retention?" The agent should retrieve relevant passages and answer only from that material.

The harness should:
- Accept a question plus a documents root
- Use tools to list/read (or search) files
- Answer only from retrieved text
- Cite which file (and roughly where) the answer came from
- Refuse politely when the documents do not contain the answer

Constraints:
- Ground answers in tool-retrieved document text
- Cite file paths used
- Do not invent policy clauses that are not in the files
- Keep the first version file-based (markdown/text) for simplicity

# Deliverable

Produce a small, runnable Python harness that uses the Vidbyte SDK (`vidbyte` package) to implement this agent. Prefer clear modules, a main entrypoint, and short comments that explain the control flow. Do not invent private Vidbyte backend APIs.

Build it step by step

Create the Q&A agent

The system prompt must forbid answering from general knowledge when the user asked about their documents.

from vidbyte import Agent

agent = Agent(
    name="document-qa",
    system_prompt=(
        "Answer only using text returned by document tools. "
        "Cite file paths. If the documents do not contain the answer, say you could not find it."
    ),
    provider="openai",
    model_name="gpt-4.1",
)

Add list and read tools

A minimal retrieval stack is often enough to start: list files, read file contents, optionally search by keyword.

from pathlib import Path
from vidbyte import tool

DOCS = Path("docs").resolve()

@tool
def list_documents() -> list[str]:
    """List document files under the docs root."""
    return [str(p.relative_to(DOCS)) for p in DOCS.rglob("*") if p.is_file()]

@tool
def read_document(relative_path: str) -> str:
    """Read one document as text."""
    path = (DOCS / relative_path).resolve()
    if not str(path).startswith(str(DOCS)):
        return "error: path escapes docs root"
    return path.read_text(encoding="utf-8", errors="replace")

Optional keyword search tool

Keyword search helps when the folder is large. Return file path + matching snippets so the model can cite them.

@tool
def search_documents(query: str, limit: int = 5) -> list[dict]:
    """Keyword search across docs; return path + snippet hits."""
    hits = []
    needle = query.lower()
    for path in DOCS.rglob("*"):
        if not path.is_file():
            continue
        text = path.read_text(encoding="utf-8", errors="replace")
        if needle in text.lower():
            idx = text.lower().index(needle)
            hits.append({
                "path": str(path.relative_to(DOCS)),
                "snippet": text[max(0, idx - 80): idx + 120],
            })
        if len(hits) >= limit:
            break
    return hits

Wire tools and ask a question

Run a question that should hit a known file first, then one that should refuse — both paths matter.

agent = Agent(
    name="document-qa",
    system_prompt="Use tools. Cite paths. Refuse when unsupported.",
    provider="openai",
    model_name="gpt-4.1",
    tools=[list_documents, read_document, search_documents],
)

reply = await agent.arun("What is our data retention policy?")
print(reply.content)

Standardize the answer format

Ask for Answer / Sources / Confidence every time so UI and humans can trust the shape of the output.

system_prompt = """
Use document tools before answering.

Respond with:
## Answer
## Sources (file paths)
## Confidence (high/medium/low)

If no document supports the answer, say so under Answer and leave Sources empty.
"""