TL;DR: Moderation catches harm and many injection attempts. It does not enforce domain or operational policy. A policy reasoning layer (LLM-as-a-judge) closes that gap, especially in multi-turn conversations.

Abstract

Moderation APIs are widely used to filter harmful content in LLM applications, yet they are not designed to enforce domain-specific operational policies. In this study we compare moderation systems with a policy reasoning approach based on an LLM-as-a-judge architecture across five operational domains. Our results show that moderation systems remain effective at detecting harmful content but fail to enforce domain policy constraints, particularly in multi-turn conversations. These findings suggest that production LLM systems require both moderation and policy reasoning layers to ensure safe and compliant behavior.

Introduction

Large language models are increasingly deployed in real-world applications across regulated domains such as finance, healthcare, insurance, and legal services. Ensuring safe and compliant behavior has therefore become a central requirement for production AI systems.

Most deployments rely on moderation systems to filter unsafe prompts. Services such as Microsoft Azure Content Safety and Azure Prompt Shields detect harmful content, adversarial prompts, and prompt injection attempts. While these systems are effective at identifying unsafe language, they are not designed to enforce domain-specific operational policies.

A request can therefore be perfectly safe from a moderation perspective while still violating business or regulatory constraints. For example, a prompt asking an insurance assistant to recommend the best policy for a specific medical condition contains no harmful content, yet such advice may be restricted in regulated environments.

Recent research has proposed LLM-as-a-judge architectures, where a secondary model evaluates prompts or responses against policy constraints before answers are produced. These systems introduce a reasoning layer capable of identifying requests that violate operational rules even when the language itself appears benign. In this study we evaluate whether moderation systems alone are sufficient to enforce domain policies, or whether a dedicated policy reasoning layer is required.

The Two Dimensions of LLM Safety

Safety mechanisms in LLM systems typically address two different types of risks.

Moderation (Harm / Injection): This is the foundational layer. Moderation systems operate primarily in the lower layer of this structure, filtering harmful or adversarial prompts.

Domain Policy (Business / Compliance): This is the operational layer. Policy reasoning systems operate in the upper layer, evaluating whether a request itself should be allowed under business or regulatory rules.

Both dimensions become critically important in regulated environments.

Evaluation Methodology

To examine the difference between moderation-based safety mechanisms and policy reasoning systems, we conducted a cross-domain evaluation comparing two independent approaches to LLM safety enforcement.

The Moderation Approach: Represented in our experiments by Microsoft Azure safety services. Azure Content Safety analyzes prompts for harmful content categories such as violence, sexual content, hate speech, and self-harm. Azure Prompt Shields detect prompt injection attempts and adversarial prompt manipulation.

The Policy Reasoning Approach: Evaluates prompts using a policy reasoning system based on an LLM-as-a-judge architecture. In this setup, a secondary language model evaluates whether a prompt violates domain-specific operational constraints.

Evaluation Domains and Safety Layers

The evaluation spans five operational domains: finance, healthcare, insurance, legal services, and retail. These domains were selected because they contain well-defined operational restrictions that frequently appear in real-world AI deployments.

Five prompt categories were evaluated:

• L1, Generic Harmful Content: Prompts containing violence, hate speech, sexual content, or self-harm.
• L2, Prompt Injection: Prompts attempting to manipulate system instructions or bypass safeguards.
• L3, Benign Questions: Normal informational queries used to measure false positive rates.
• L4, Direct Policy Violations: Prompts explicitly requesting actions that violate domain policy.
• L5, Policy Evasion Attempts: Prompts attempting to obtain restricted outcomes through indirect or adversarial phrasing.

Single-Prompt Performance

Each system was evaluated on 500 prompts per layer per domain, with results reported as cross-domain averages. Metrics include F1 score for detection tasks, false positive rate for benign prompts, and mean latency per prompt.

• L1 (Generic harmful content): Both systems achieved an F1 of 73.1%. Moderation works as intended for generic harm detection. Latency: Judge 1095ms, Azure 427ms.
• L2 (Prompt injection): LLM-as-Judge F1 67.8%, Azure APIs F1 53.5%. Both moderate, with the judge somewhat better. Latency: Judge 1068ms, Azure 463ms.
• L3 (Benign questions): LLM-as-Judge false positive rate 86.4%, Azure APIs false positive rate 0.8%. Moderation is far less prone to overblocking. The judge is very conservative in this experimental setup. Latency: Judge 1068ms, Azure 532ms.
• L4 (Direct policy violations): LLM-as-Judge F1 98.2%, Azure APIs F1 5.3%. Moderation almost never catches domain policy violations. This is the core finding. Latency: Judge 1121ms, Azure 489ms.
• L5 (Policy evasion attempts): LLM-as-Judge F1 83.7%, Azure APIs F1 0.0%. Moderation completely misses indirect and adversarial policy violations. Latency: Judge 1134ms, Azure 509ms.

The most significant differences appear in the policy layers. The LLM-as-a-judge system achieves high detection accuracy for both direct policy violations and evasion attempts. Moderation APIs detect almost none of these cases, reflecting the fact that they are not designed to encode domain-specific operational constraints.

Multi-Turn Conversation Evaluation

Because many safety failures occur within conversational context, we also evaluated multi-turn interactions. Each conversation consists of four turns: a benign prompt, a benign follow-up, a benign contextual question, and a restricted request. The first three turns should pass while the final turn should be blocked.

For each domain we generated 200 conversations per safety layer, resulting in 1,000 conversations per layer across domains. Performance is measured using Conversation Success Rate (CSR), defined as the percentage of conversations where the system allows benign turns and blocks the restricted final request.

LLM-as-Judge results:

• L4 CSR 94.1%
• L5 CSR 83.6%
• L4 Block Rate 100.0%
• L5 Block Rate 88.8%
• Clean Pass 96.9%
• Mean Latency 3960ms

Azure Safety APIs results:

• L4 CSR 0.0%
• L5 CSR 0.6%
• L4 Block Rate 0.0%
• L5 Block Rate 0.6%
• Clean Pass 100.0%
• Mean Latency 1924ms

The results highlight a clear difference between moderation systems and policy reasoning. Moderation APIs maintain a perfect clean-pass rate, meaning they rarely block benign prompts. However, they almost never block policy-violating requests when they appear in conversational context.

The LLM-as-a-judge system demonstrates the opposite pattern. It successfully blocks most restricted requests and achieves high conversation-level correctness, though at the cost of slightly higher false positive rates and increased latency. The gap between L4 and L5 performance reflects the additional difficulty of detecting policy evasion attempts, where violations are expressed indirectly.

Today an AI agent trying to browse the web is like a thief in a balaclava sneaking around a police academy. Site protections block it, challenge it, turn it away.

browser-search flips the script: your agent stops being the thief and becomes the chief of police. No more clumsy access attempts. It walks through every door because it has the right tools. SearXNG for search, Camofox for browsing, CloakBrowser when things get tough.

100% self-hosted, free, no limits, no API keys.

I just released v2.0, whose core logic enforces the exclusive use of deterministic scripts. This eliminates model hallucinations, even with the cheapest models. The skill describes the 3 tools in natural language, but execution is rigid: the model can neither get the command wrong nor misinterpret the output. The result is guaranteed success on every query — the skill and deterministic scripts guide the model to scour the web until it finds the answer.

No more excuses. Your agent has the badge now.

https://github.com/Johell1NS/browser-search

My 2016 box ran GLM-4.5-Air (110B, 7x its RAM) streamed from a SATA drive:
pre-registered prediction 0.2-0.3 tok/s, measured 0.19 The same equation (tok/s = eta(tier) x bandwidth / active-bytes) runs Qwen3-30B at 19.3 tok/s on the GTX 1060 and prices any memory upgrade in tok/s before you buy. And the cleanest proof it's placement, not hardware: two Q2_K GGUFs of Gemma-4-12B, same 5.22 GB on disk, differing by 2.25 perplexity purely from which 12 layers got the protected bits, placement is worth roughly 2x the byte budget.

Below: the head-to-head table, the four laws it falls out of, the pre-registered predictions that test them (including a model I predicted within 1% without ever touching the hardware), and quantprobe, the tool that runs the whole loop. Every number measured on one box: i5-7600K (4c/4t), GTX 1060 6GB (Pascal, \~$150 used), 16GB DDR4, Crucial MX500 SATA. Solo project, AI-supported.

1. The head-to-head: same bytes, different layers

Stock llama.cpp \`--tensor-type\`, Gemma-4-12B, FFN at Q2_K. Same quantizer, same bit budget — the only change is which 12 blocks stay at a higher type:

| Recipe | PPL (WikiText-2) | File |

|---|---|---|

| Uniform Q2_K FFN | 14.41 | — |

| Protect first 12 layers | 12.27 | 5.22 GB |

| Protect last 12 layers | 10.02 | 5.22 GB |

The last two rows are byte-identical. That's the cleanest control I know how to build for a placement effect

Where the method lands against baselines — same box, same eval windows:

| At parity | Baseline | This work | Delta |

|---|---|---|---|

| llama.cpp naive-best placement (Qwen3-30B, same GGUF) | 12.6 tok/s | 19.3 | +53%, zero cost |

| imatrix-calibrated community Q2 (Qwen3-30B) | 11.27 PPL | 11.08 | data-free edges calibrated (+15% size) |

| Calibrated SOTA MxMoE (DeepSeek-V2-Lite 16B, 2-bit) | 1.18x gap | 1.10x (6.31→6.96) | data-free, resident on the 6GB card |

2. Why it works: four laws

The recipe isn't a trick — it falls out of four falsification-tested findings:

The recipe isn't a trick — it falls out of four falsification-tested findings:

1. Rotation is rank-conditional. Incoherence rotation (QuIP#/QTIP/QuaRot) costs +0.006 PPL on a full-rank MLP and +1623 PPL on a low-rank bottleneck — a ~270,000x swing on effective rank alone.

2. Trained networks are dense everywhere. Experts sit at the rate-distortion floor, routing is domain-flat (prose vs code expert sets: Jaccard 1.00), 1-bit collapses (+253 PPL) under every codec. ~2-bit is the data-free floor. No free sparsity.

3. Fragility is measurable, not predictable. Gemma-4-12B late-fragile ~4x, Qwen2.5-7B late ~2-3x, Qwen3-30B-MoE late ~2.3x, Mistral-7B early-fragile ~25x — Qwen's architectural near-twin, fragile in the opposite direction. You must probe, not guess. [attach: charts/x_chart_C_depthcurve.png]

4. The tiered decode law:
   tok/s = eta(tier) x bandwidth / active-bytes-per-token
   with eta = 0.56 (VRAM), ~0.62 (RAM dense), ~0.38 (RAM MoE), 0.88–1.0 (disk). One equation, 7B to 744B.

What's mine vs. what I build on

Not mine: llama.cpp + k-quants; the QuIP#/QTIP/QuaRot incoherence-codec line; colibri's tier-streaming engine (github.com/JustVugg/colibri) as inspiration — its published tiers land inside my eta bands too.

Mine: the four laws, probe-then-quantize + this tool, the byte-identical placement control, pre-registration as methodology, the depth-aware recipes and placement solver.

Honest limitations

- WikiText-2 PPL is my only quality metric so far — no MMLU/HellaSwag yet.

- Fragility atlas covers four families: enough to disprove universality, not chart the world.

- 0.19 tok/s on the 110B is a capacity demo, not usable inference.

- Single-box research; speeds are single-stream decode, ±25% across environments; eta values are fitted, not derived.

- Machine presets beyond my own box (Mac, newer GPUs) are falsifiable predictions from the law, not measurements. Validated on llama.cpp b9596.

I wanted to know what actually happens under loss.backward(), so I ported Karpathy's llm.c to Mojo and wrote the whole training stack by hand: every kernel (attention, layernorm, softmax, the fused classifier, AdamW) and the full backward pass, derived on paper first (the derivation is a ~1,470 line LaTeX writeup in the repo). It extends dorjeduck's CPU-only llm.mojo with CUDA and Metal GPU backends. No PyTorch and no CPython at train time; PyTorch only appears offline, as the reference my gradients are tested against.

Things I learned that I never got from reading papers:

• Training is mostly one operation. The matmul is ~70% of a step, and it's the one thing everyone (me included) delegates to a vendor library. The other 30% is where the project actually lived: attention, normalization, the optimizer, and the backward wiring.
• A model can train with badly broken gradients. For weeks my position-embedding gradient was about a million times too small because one skip-connection gradient was never seeded, and the loss still went down. The other parameters compensated. Loss going down proves almost nothing; my tests only started catching real bugs once they compared every gradient tensor against PyTorch with per-tensor tolerances.
• The same math runs 8 to 10x faster or slower depending on how you shape it for the hardware. Scalar flash-attention kernels tuned for NVIDIA ran at under 1% of FLOP peak on Apple GPUs; rewriting attention as plain GEMMs made it 8 to 10x faster there. Nothing about the model changed, only the layout of the work.
• You can reproduce GPT-2 at home. The 124M trained to completion on FineWeb and scores 29.53% on HellaSwag, statistically indistinguishable from Karpathy's own llm.c reproduction at 29.9%. The checkpoint is on HuggingFace (ulmentflam/gpt2-124m-fineweb-mojo).

Where it landed on speed: parity with llm.c's CUDA path in bf16 on a GB10 (0.999x), 1.71x faster than PyTorch MPS on an M4 Max but still behind Apple's MLX (MLX is 1.24x faster than me), and about 4x faster than llm.c's 20-thread OpenMP build on CPU. The benchmark harnesses are in the repo, so you can rerun the comparisons on your own machine.

Disclosure: the kernels and trainer are hand-written without LSPs or LLMs; that was the point of the project. Tests and a later optimization pass were AI-assisted, and both are documented in the repo.

Repo: https://github.com/ulmentflam/llm.mojo

I was able to ran Deepseek V4 Flash local with my rig with tokens per seconds ranging from 1.3-2.0 t/s depends on topic and what experts are already hot and in ram/vram. TTFT after getting warm is around 10-11s still trying to optimize it further. But I think that I can run this 143GB (int4) model with my spec is not to shabby. Do some guys of you experienced with llm on nvme and got some tips to further improve it?

Hi, so I have 8 H100 cards, on which I tried multiple experiments to fine tune Gemma 4 26B MOE model.

Have tried full sft, Lora-peft, and even cpt+sft, cpt + Lora-Peft. And after all these experiments, the model becomes worse, in each case, and hallucinating.

Is it because of sensitivity of MOE models, also should I try experiments with dense models here. Or should I focus more on correcting my dataset.

Built a Python lib that does repo-scale context selection for LLMs (AST parsing + dependency graphs + relevance scoring, instead of dumping whole files). Retrieval-side tests are solid (103 passing), but proving it actually helps downstream task performance is where I'm stuck:

Free local models aren't strong enough to give trustworthy task-success signal

API rate limits kill any real benchmark run

Paid API costs balloon fast once you're running many files/tasks

Anyone found a good middle ground — specific local models that work for this, cheaper eval strategies, sampling tricks? Would appreciate any pointers.

I'm currently in college studying engineering, and I've done so many different project that I want to create a portfolio. I have experience in doing so many different things, but I never finished a project such that it looks like a usable app or a downloadable GitHub repo. I have Gemini Pro for free as a college student, but honestly Claude is so much more powerful, clever, and I personally think has less memory loss. So I was considering using claude to help me polish all the projects and help me create my portfolio such that it looks nice, but from what I hear, the 20$ plan could be limiting, and the 100$ seems better for this. However, I'm in college and this will not generate me direct cash in return.
Thus, is it interesting for me to put that 100$ in claude for the rest of the summer, or should I cleverly use the free plan and continue with Gemini Pro?

Question for people building fintech support / KYC / collections / dispute workflows.

If you transcribe calls, where do you actually handle PII/PCI redaction?

I keep seeing three possible patterns:

1. STT-level redaction
The transcript comes back already masked.

2. Post-processing redaction
Transcript is generated first, then another rules/model layer masks sensitive data.

3. Tokenized workflow
Sensitive values are captured into secure fields and never live in the general transcript.

My worry is that "redaction supported" sounds clean on a vendor page, but real calls are messy:

• card number spoken in chunks
• account numbers corrected mid-sentence
• addresses mixed with local landmarks
• emails spelled badly
• user says phone number twice
• agents repeat sensitive info
• background person says something
• partial transcript briefly contains sensitive info before final redaction

I noticed Smallest AI Pulse talks in the direction of real-time STT with PII/PCI redaction, which is exactly the kind of feature fintech teams would ask about.

But would you trust STT-layer redacttion alone?

or would your architecture still

transcribe/redact
→ second pass check
→ tokenize critical fields
→ restrict storage
→ audit logs
→ retention policy

Especially for real-time voice agents, there's a weird tension:

The system may need the sensitive value for 5 seconds to complete the t ask.

But the transcript should not become a permanent sensitive-data landfill.

How are teams solving this?

Not a gimmick. Not a demo with a cherry-picked prompt and a loading screen that took 4 minutes. A real language model, generating coherent intelligent text, running entirely on a Snapdragon 855 from 2019 - a chip that was already considered “last-gen” when Biden was inaugurated.

No API calls. No Wi-Fi. No subscription. Just silicon, RAM, and math.

If that doesn’t make you stop and think - keep reading, because it gets more interesting.

https://github.com/m4vic/TinyMobileLLM

Hi everyone,

I’ve spent the last few months compiling everything I know about Large Language Models into a structured, open-source book. My goal was to create the resource I wish I had when I started: something that bridges the gap between high-level tutorials and complex academic papers.

https://github.com/Drobiazkin/ai-agent-architecture