What it is

A self-hosted AI assistant for GRC (Governance, Risk & Compliance) and cybersecurity. You ask a plain-language question ("What are the DPDPA breach-notification timelines?") and it answers from a curated corpus of 2,681 real controls across 48 frameworks (GDPR, HIPAA, ISO 27001, NIST, PCI DSS, SOC 2, RBI, DPDPA, DORA, and more) with inline citations back to the specific control and its official source URL, plus a verification-tier badge so you know how trustworthy each source is.

Design goals: runs entirely on your own hardware, near-zero infra cost, no data ever leaves the machine (it's compliance data that's the whole point), and a clean integrable API so it's not locked to the bundled UI.

The stack

Backend (Rust)

• Axum 0.8 + Tokio async API, SSE streaming
• LanceDB (embedded vector DB) for dense retrieval + Tantivy (BM25) for sparse — merged with Reciprocal Rank Fusion.
• No external vector-DB service.
• Ollama for local inference: bge-m3 embeddings (1024-dim) + qwen2.5:7b-instruct generation
• SQLite (rusqlite) for users, hashed API keys, audit log, and persistent chat history — everything embedded, one file
• JWT + API-key dual auth with RBAC (viewer/admin); RFC 7807 error envelopes; OpenAPI/Swagger auto-generated via utoipa
• SearXNG (self-hosted metasearch) as a live-web-search fallback when the corpus is thin or the question is time-sensitive

Frontend: React + TypeScript + Vite + Tailwind v4, streaming chat UI, collapsible "thinking" reasoning trace, citation panel, admin console (coverage/audit/API-key issuance)

Ops

• Multi-stage Docker with cargo-chef layer caching, single-binary that also serves the built frontend (one port, no separate frontend host)
• Eval harness (retrieval precision@k + a grounding/hallucination check) and a drift monitor that re-checks high-stakes facts (penalties, deadlines, circular numbers) against their source URLs on a schedule

Features that actually work right now

• Hybrid retrieval -> grounded, cited answers (token-streamed)
• Persistent chat -> generation runs as a detached backend task, so an in-flight answer survives a reload/tab-close and gets saved regardless
• Thinking mode toggle -> runs a deeper pipeline (query rewriting → rerank → self critique) and streams the reasoning steps
• Live-web-search toggle, per-framework filtering, thumbs feedback logging
• Full auth/RBAC/audit trail

Being honest about the rough edges

• It's CPU-bound on my dev box (no GPU) a 7B model answer takes ~1-2 min; streaming hides most of it but a GPU is the real fix
• Small local models (1.5B/0.5B) drop citations under long RAG context instruction following degrades; 7B is the floor for reliable citing. Interesting problem I'd love input on.
• Single-node, self-hosted MVP not horizontally scaled yet
• The corpus went through two automated audit/fix passes but hasn't had a formal expert review

Why open source

Compliance tooling is either eye-wateringly expensive SaaS or spreadsheets. A self hostable, private, auditable RAG assistant over open framework data feels like something the community should own. Planning to open it up once the core is battle-tested.

Happy to go deep on any part the RRF hybrid retrieval, the detached-generation persistence model, the cargo-chef Docker setup, the citation-grounding eval, whatever. What are you all building in this space?

​

I've been wondering about this lately. Many organizations are building in-house LLMs for GRC to answer security questionnaires, map compliance controls, generate policies, and support audit preparation. Keeping everything on-premises helps with data privacy, but it doesn't solve the biggest problem: a confident hallucination can still end up in an audit report or customer response if no one catches it.

The more capable these systems become, the more people are likely to trust them. That creates an interesting trade-off. If an AI-generated answer helps close deals faster but occasionally invents evidence or misinterprets a control, the financial and compliance impact could outweigh the productivity gains. At that point, is the technology truly ready for critical GRC work, or are we becoming overconfident because the responses sound convincing?

For those using an in-house LLM in GRC today, where do you draw the line between AI assistance and human accountability—and do you think we'll ever reach a point where an LLM can be trusted to answer security questionnaires without manual verification?