Overview
The Problem
Architecture
Document Processing & Enhanced OCR
Document Classification & Entity Extraction
- Pass 1: Type-specific structured metadata extraction (dates, amounts, identifiers stored as document properties)
- Pass 2: Entity typing with few-shot examples. The system identifies people, organizations, locations, products, medical conditions, named systems, documents, and events.
- Pass 3 (Verification): An independent LLM pass critiques the extracted entities and removes junk, duplicates, and over-extractions
- Pass 4 (Relationships): Relationship inference between verified entities
LLM Query Routing via LiteLLM Proxy
Agent-Assisted Query Pipeline
- Fast entity/graph search (~2 seconds): Neo4j name matching for structured lookups, e.g. "find all documents mentioning Allstate" or "what entities are connected to Duke Energy?" Results render in the graph explorer where you can traverse relationships visually.
- Deep AI synthesis (~60-120 seconds): A hybrid RAG pipeline that retrieves relevant document chunks via vector similarity (pgvector, 3072-dimensional indexes), trigram keyword search (pg_trgm), and entity graph traversal. Strands Agents build a bounded retrieval plan, review evidence gaps, verify the drafted answer, and produce a claim ledger alongside citations.
- Timeline mode: Strands extracts dated events from retrieved evidence, then the deterministic query engine keeps source attribution and ordering explicit.
- Strict mode: The system fails closed when evidence is weak instead of forcing a confident-sounding answer.
Interactive Frontend
- 2D/3D graph explorer using react-force-graph with physics simulation for navigating entity relationships across the entire document corpus
- Natural language query interface with streaming SSE responses and real-time status updates as the system retrieves, analyzes, and synthesizes
- Conversation history for multi-turn Q&A with persistent sessions
- Document browser filterable by type, with per-document reindex support
- Model selector for choosing between available LLMs via LiteLLM per query
- Trace metadata for inspecting retrieval plans, evidence grades, verifier feedback, and claim support
Tech Stack
- Paperless-ngx: document ingestion, storage, and base OCR
- Google Gemini Flash: document classification, entity extraction, query planning, verification, and synthesis
- Strands Agents: bounded planner, verifier, timeline extractor, answer repair, and entity review support
- Neo4j 5 + APOC: graph database for entity relationships and multi-hop traversal
- PostgreSQL + pgvector + pg_trgm: vector similarity search (3072-dim indexes) and trigram keyword search
- FastAPI: async Python backend and REST API
- Next.js 16 + react-force-graph: frontend with 2D/3D graph visualization, query UI, and conversation history
- LiteLLM: LLM proxy for model routing, key management, and cost tracking
- Redis: query, vector, and graph result caching with TTL-based expiration
- RapidFuzz: fuzzy string matching for entity resolution
- Kubernetes + container images: current app deployment for the API, frontend, and Redis, with Docker Compose still available for local development
Key Design Decisions
- Graph over vector-only: Pure vector search finds similar documents but can't answer relationship questions ("who is my insurance agent and what policies do they manage?"). The graph layer captures structure that embeddings lose.
- Multiple query modes: Not every question needs a 90-second synthesis. Fast graph search handles most lookups instantly, while deep, timeline, and strict modes are reserved for questions that genuinely need multi-document reasoning and evidence review.
- Verification as a first-class step: The system does not just draft an answer. It verifies claims against retrieved sources, repairs weak answers when possible, and exposes trace data so the result can be inspected.
- Exact freshness checks: The system compares Paperless document IDs against Neo4j document nodes, pgvector chunks, and stored document hashes. Count-only checks were not enough. Exact ID-set comparison catches partial indexing drift before it shows up as a missing or stale answer.
- LLM proxy abstraction: Decoupling the application from any single LLM provider means the system survives API changes, rate limits, and pricing shifts without code changes.
- Enhanced OCR as a separate layer: Running AI-enhanced OCR independently from Paperless-ngx means the base system stays vanilla and upgradeable, while the intelligence layer can evolve independently.
Results
- Queryable knowledge base spanning 800+ personal and business documents
- Roughly 7,000 graph nodes, 25,000 relationships, and 6,000+ searchable document chunks
- Sub-2-second answers for entity and relationship lookups
- Natural language queries across tax records, contracts, insurance policies, medical records, and financial documents
- Zero manual data entry, fully automated from document scan to queryable graph