r/netsec • u/BordairAPI • 2h ago
I ran a paid bug-bounty-style game against my own multimodal prompt firewall, it didn't make money, so here's the code, the model and 13k real bypass attempts
huggingface.coQuick bit of context for why this dataset exists. I spent the best part of a year on a startup where the product was an AI guard sitting in front of an LLM, and the pitch to users was "try and break it." Leaderboard, tiered levels, a bit of prize money for anyone who got through. Never turned into a business. What it left behind was a year of logged, motivated, unpaid red-teaming against a detector that knew it was being attacked, which is a more interesting asset than the business ever was. Rather than let it sit in an RDS instance going stale, I've open-sourced the lot.
What it is: a two-stage filter that sits between user input and an LLM. Stage one is a regex gate, 119 patterns covering direct-override phrasing in about eleven languages, DAN/persona-style jailbreak framing, template-injection sigils ({{, ${), agentic chain-of-thought injection, homoglyph and zero-width-character stripping, with a decode-then-scan pass ahead of it that catches base64, ROT13 and leetspeak payloads before either the regex or the model sees them. Whatever the gate can't resolve falls through to a fine-tuned DeBERTa-v3-large, exported to ONNX and quantised to INT8. One thing I'll own upfront: the training script and config.json still describe a four-label head (benign/direct/jailbreak/indirect), but every training sample is actually labelled 0 or 1 and the exported graph only emits two logits. So the shipped model is a plain binary classifier. I caught this re-reading my own code a couple of weeks ago and corrected the docs rather than quietly leave the four-way claim standing.
Multimodal side pulls text out of images (OCR plus EXIF/PNG/XMP metadata fields, with a lossy re-encode first to kill LSB steganography and adversarial pixel perturbation), out of PDF/DOCX/XLSX/PPTX, and out of audio via ASR. Same two-stage pipeline runs over whatever comes out, tagged by modality so the model treats OCR and ASR text as noisier than typed text. Most open prompt-injection tooling I've come across only looks at the text field; I haven't seen another open-source detector that also scans images, documents and audio for injected payloads, so as far as I know this is the first one that does.
The bit worth your time here is the game logs. 13,230 attack strings, hand-written by real people trying to beat a live detector. No templates, no synthetic generation. Anonymised before publishing: identifiers and payment data dropped at the table level, emails/phone numbers/card-shaped digit runs redacted in the text itself, anything still risky after that pulled for manual review instead of auto-published. What ships is attack text plus a few coarse labels. Public injection datasets are almost all generated; this one is scraped from people actually trying.
Code: https://github.com/Josh-blythe/bordair-detector Dataset (synthetic + the real split): https://github.com/Josh-blythe/bordair-multimodal Weights: https://huggingface.co/Bordair/bordair-detector
Licensing: Apache-2.0 for the code and weights. Base model is microsoft/deberta-v3-large, MIT, attribution kept in NOTICE.
Threat model, so nobody's disappointed: this reads the prompt at inference time. Training-time and weight-level attacks are out of scope entirely. Treat it as a mitigation layer, one input among several a real deployment should have, not something you point at your LLM and forget about. I'd rather someone here find a trivial regex bypass now than have it sit unexamined, so have at it.