I do AI evaluation work and I’m now building some small LLM stuff of my own on the side. At work we have structured rubrics and QA; on my own projects I’m realizing I just eyeball a handful of outputs and hope the rest are fine, which feels sketchy.
For those of you shipping LLM features or agents: how are you checking output quality before you ship? Manual review? LLM-as-judge? Some eval framework? And whatever you’re doing, what’s the most annoying part of it?
Trying to figure out if I’m the only one doing this by vibes.

A retrieval config change slipped through CI last month, and the aggregate task-success number on the eval set stayed flat. What actually broke was groundedness on any query that hit the reindexed section, and it stayed invisible until a support ticket came in citing fabricated sources.

The three that worked

Context adherence, per-answer (not aggregate)
Context adherence scores whether every claim in a generated answer is actually supported by the retrieved chunks. Moving it from aggregate to per-answer is what turned it into a leading indicator, not an artifact everyone reads after an incident. It surfaces the right-retrieval-wrong-summary case where the top-k is fine but the model paraphrases into a wrong number, and it catches silent drift when a knowledge base changes underneath you. In practice it gates PRs against a fixed eval set and runs sampled in prod, paging on a rolling window drop.

Tool-choice + tool-argument correctness, per-step (not per-run)
This scores each step of an agent trajectory: did the agent pick the right tool, and did it fill the arguments correctly. Per-step is the important part. Final-answer accuracy quietly forgives a right tool with wrong args. The example that convinced us: an agent picked the delete-branch tool with the correct branch name but the wrong remote, and the final message read "done, branch cleaned up." Task-success stayed green until we added argument-level scoring. Now every tool node is scored against a per-tool schema and expected value pattern.

Judge consistency (the eval on your evals)
Judge consistency measures how stable an LLM-as-judge score is across seeds, position in pairwise comparisons, and small rephrasings of the same rubric. It matters because it is the meta-metric that decides whether the other two can be trusted. A rubric that looks stable on aggregate can drop hard once position bias is controlled for, and any A/B test conclusion drawn from it after that is noise. Cheap check: run the judge five times per sample with shuffled order, report agreement, reject rubrics below a chosen kappa floor.

The two that didn't

Aggregate task-success rate on a fixed eval set. Aggregates average away tail failures. A retry-loop agent silently samples the gap between pass^1 and pass^k, and the aggregate number stays healthy while the eighth attempt bleeds. Replace with per-step and per-trajectory scoring so a failure mode has a name, not a moving average.

BLEU / ROUGE / cosine-similarity to a reference answer. Real prod tasks rarely have a canonical re is not text overlap, and these metrics quietly reward models that copy phrasing over models that getfacts right. Replace with an LLM-as-judge scored under a bounded rubric, then check the judge's consistency.

What is one metric on your side that looked predictive on the dashboard and quietly wasn't?

We've been researching evaluation platforms while scoping our rollout. Ultimately the team came to these specific criteria to evaluate the different platforms. Curious what other orgs are including in their evaluation axes (pretty sure there are no standards yet) — we're set on buying rather than building, so please don't recommend we build our own solution.

The axes that actually mattered to us:

• Tracing/observability depth — span-level detail, latency/cost breakdowns, session-level views vs. just prompt-level
• Evaluation methodology — built-in metric libraries vs. bring-your-own, LLM-as-judge support, human-in-the-loop annotation
• CI/CD integration — can this gate a deploy, or is it purely post-hoc
• Agent-specific evaluation — tool-call correctness, trajectory/step evaluation, not just final-output scoring
• Red-teaming / adversarial testing — built in or bolted on
• Governance — versioning, approvals, audit trail, RBAC
• Framework lock-in — is it framework-agnostic
• Deployment model — SaaS-only vs. self-hosted/open-source option (we need self-hosting since we're a large org)

Feature comparison (roughly, as of our research):

(CI/CD gate support and framework lock-in mattered to us too, but cutting them kept the table from turning into a spreadsheet — happy to share those details in comments if useful.)

A few things that stood out once we laid it out this way rather than as prose:

"Agent evaluation" means different things depending on the vendor. Some platforms score the final output of a multi-step run; others actually evaluate the trajectory — did it call the right tools, in a reasonable order, without looping or cheating. If you're running agentic workflows rather than single-turn Q&A, this distinction matters more than any other row in the table.

Governance and evaluation are converging, but unevenly. ConfidentAI and Galileo build governance in; LangSmith treats it as secondary; the OSS libraries (DeepEval, RAGAS, Promptfoo) don't really have a governance story at all — you're expected to build that layer yourself or pair with something like a prompt-versioning tool.

Self-hosting is a real fork in the road, not a checkbox. If your org has data residency or compliance requirements, that alone probably eliminates half this list before you even get to feature comparison.

Red-teaming is still spottier than the marketing suggests. ConfidentAI, Galileo, and Promptfoo have genuine red-teaming capability. Most others treat it as a roadmap item or expect you to bring your own adversarial test set.

One thing this table can't capture well: actual day-to-day usability and how these hold up once you're running hundreds of evaluations a day instead of dozens. If anyone has run more than one of these side by side at volume, curious where reality diverged from the pitch.

I've posted here before about tuning llama.cpp on a 6GB 3050. Throughput I'd measured but what I hadn't measured was the thing that actually matters for analytics work: how often does the model just make numbers up?

So I built a checker. My pipeline (XGBoost -> SHAP -> optimizer -> an LLM agent chain that writes the summary) keeps every number the agents are allowed to cite in a ground-truth pool. Every number in the generated text gets matched back against that pool, with tolerance for rounding, percent-vs-fraction and k-notation. Unmatched = flagged.

30 seeds on Llama 3.1 8B Instruct Q4_K_M via llama.cpp, 30 identical seeds on a frontier API model as control. Then I hand-audited every flag against a deterministically rebuilt pool - no LLM in the audit loop.

8B Q4_K_M: 138 numbers written, 10 fabricated (7.2%), 4 of 30 runs affected.
Frontier control: 537 numbers written, 0 fabricated.

The rate wasn't what surprised me, the failure mode was. It didn't get numbers slightly wrong, it invented structures that exist nowhere in my pipeline. A "60% margin preservation / 40% efficiency" budget split with no basis in the data. A full ROI table ("$100k spend, $500k revenue") in a pipeline that computes neither. Best one: "Reduce budget by 20% to $X". It fabricated a metric and left the template placeholder unfilled in the same sentence.

Caveats before anyone quotes the 7.2%: one model, one quant, one prompt chain, synthetic (seeded, reproducible) data. K=30 puts the 95% CI at roughly [4%, 12.8%]. It measures numerical grounding only : whether a cited number exists in the source, not whether the argument around it is sound.

What I actually want to know: is this quantization damage or just 8B being 8B? I only tested Q4_K_M. The harness runs offline with no API keys, so if anyone has the VRAM for Q8/fp16, or wants to point it at Qwen or Mistral at a similar size, I'd like to see whether the rate moves with quant level or whether it's a parameter-count floor.

Repo (checker, harness, all 60 transcripts, audit CSVs): https://github.com/abhinandan-084/GTM-Wargame

Write-up with full audit methodology: https://pub.towardsai.net/why-my-llm-guardrail-flagged-the-right-answers-and-why-i-refused-to-fix-it-0db77efb0644

I’ve been experimenting with context compression for AI agents, and I ran into a tradeoff I hadn’t fully appreciated.

Reducing the context lowered token usage, but some tasks became less reliable. The issue wasn’t always that the agent had “forgotten” something important. In several cases, compression changed information that needed to remain exact.

I’ve started thinking about agent context in three rough categories:

Disposable context

Repeated search results, duplicated documentation, long file listings and verbose logs. This is usually a good candidate for filtering or summarization.

Load-bearing context

Exact error messages, file paths, line numbers, patch anchors, test names and acceptance criteria. Even a small rewrite can remove the detail the next action depends on.

Machine-consumed context

JSON, shell output, CSV, patches or any other output that may be parsed by a tool or passed into a command.

This last category caused the most surprising failures.

A summarized command result may still make sense to a person or model, but it can become invalid when another program expects the original structure. The command still runs—it just processes the wrong data.

That made me question whether context compression should be based primarily on token count.

A more useful policy might depend on what happens next:

• Is the content only being read by the model?
• Does the next action require an exact value or text match?
• Could the output be consumed by another tool?
• Can the original information be recovered cheaply?

My current view is that the goal shouldn’t be the smallest possible context. It should be the smallest context that preserves the evidence and interfaces required for the next step.

How are people handling this in production?

Are you using explicit rules for content that must never be summarized, or are you relying on the agent to retrieve the original data again when needed?

I've been working with AI coding agents (mostly Codex and Claude Code) on fairly large TypeScript projects, and I kept noticing the same thing.

The model wants to answer a simple question like:

• Where is this function defined?
• Who calls it?
• What's its inferred type?

...and ends up reading an entire 2,000-line file.

That felt incredibly wasteful, especially when the answer is just one function.

So I built SymbolPeek.

It's an open-source (MIT) MCP server that gives LLMs symbol-level access to your codebase instead of file-level access.

For TypeScript/JavaScript it uses the official TypeScript Compiler API, so it can answer things like:

• read_symbol
• find_references
• find_callers
• find_callees
• go_to_definition
• get_type
• get_call_hierarchy

For Rust, Python, Go, Java, JSON and Markdown, it currently provides syntax-aware navigation powered by Tree-sitter.

One real example from the project itself:

Instead of sending a 65 KB file (1,791 lines), the agent requested exactly one nested function and received about 2 KB of source.

I also added lifetime statistics because I wanted to know whether semantic navigation actually makes a measurable difference.

Current numbers from my own daily usage:

```text Requests: 162 Files avoided: 163 Lines avoided: 352,910 Bytes avoided: 6.4 MB

Estimated tokens saved: ~1.61M Average context reduction: 95.7% ```

These aren't synthetic benchmarks—they come from real coding sessions.

The goal isn't to replace grep or reading source files.

It's to stop AI assistants from loading huge files when they only need one declaration.

The project is completely free and MIT licensed.

I'd love feedback from people building MCP tools or using Codex, Claude Code, Cursor, Cline, Roo Code, Windsurf, etc.

GitHub:

https://github.com/pioner92/symbolpeek-mcp

Ran the same bug-fix benchmark against all five models: small JS repo, a bug report, four tools (list/read/write files, run tests), with a 24-step budget on default settings. THere were 18 tasks each, run both with tests available and with tests hidden making 410 runs total.

Here are some of the findings I thought were cool to note:

-With the test suite available, every model solved every task 100% of the time, including multi-file bugs. But cost per solved task ranged 14x for identical outcomes: the cheapest model (Luna) landed around $0.01/fix, the priciest (Opus) around $0.14, same result either way.

-Hide the tests and force a fix the model can't verify, and one bug splits the field hard. Both GPT-5.6 models solved it 13/13. Every Claude model (Opus 4.8, Sonnet 5, Fable 5) solved it 5/13 or fewer. Fisher's exact test on the pooled comparison: p ≈ 3×10⁻⁹. The traces read like a family-level habit rather than a capability gap: GPT models consistently made the smallest fix that preserved existing behavior, while Claude models diagnosed the bug correctly, then rewrote surrounding code in a way that broke a contract the hidden tests were pinning.

-Claude Fable 5 refused 29 of its 82 runs outright: HTTP 200, stop_reason: refusal, category "cyber," for ordinary bug-fix requests (one was a bug report about a JSON-patch rollback). So basically a 40% refusal rate on a completely benign coding workload.

The full writeup, methodology, and caveats: https://latitude.so/blog/benchmarking-frontier-models-as-coding-agents

All the technicals like the harness, raw per-run data, and refusal probes are in the repo if anyone wants to reproduce or point it at other models which could be fun: https://github.com/latitude-dev/coding-agent-benchmark

Spent two years running a multi-agent system in production (ESG analytics — cited answers over graph + docs + web). The lesson that reframed everything for me:

You will not make an LLM deterministic. Stop trying. What you can make deterministic is the orchestration around it.

Concretely, we pulled control flow out of the model entirely:

• The planner emits a typed task graph — a contract of what it wants, never how. It can't reach into a worker.
• Tasks go on durable queues. A worker dying mid-task isn't a recovery problem — the task just waits on the queue for the next consumer. No in-process state to lose.
• The aggregator pre-registers the expected task set before workers run, so it never synthesizes early and never hangs on a task that was never dispatched.

The stochasticity is quarantined to the workers, where you actually want judgment. The control plane is boring on purpose.

Honest soft spot: we currently trust the planner's emitted task list with no validating schema before dispatch. How are you all gating LLM-emitted plans — JSON schema, a repair loop, constrained decoding, something else?

Full write-up at Link

We noticed Claude Opus, GPT-5.5 and GLM-5.2 were spending most of their time doing routine work like searching files, rerunning tests and updating code, instead of actual hard reasoning.

So we built a router that picks the model per turn instead of locking an entire agent session to one model. Most turns stay on cheaper models, while harder ones get escalated automatically. The agent doesn't need to change.

We also benchmarked it against direct Opus, Sonnet and OpenRouter Auto on Terminal Bench, and wrote up the routing logic, cache behavior and cost breakdowns.

https://entelligence.ai/blogs/entelligence-model-router-frontier-quality-coding-agents-at-half-the-cost

The thing that stood out to me wasn’t just which model looked best. It was how different the cost/quality tradeoff looked once price was shown.
I was looking at AIHubMix’s github.io/model-showdown and focused on the last test on the page. Same prompt, one shot, generated HTML outputs, no manual cleanup. The task was a 3D global logistics dashboard with a globe, route arcs, stats, controls, and dashboard-style UI.
The models/prices in that round were:
Kimi K3: $0.52
GPT-5.6 Sol: $1.81
Claude Fable 5: $1.51
Gemini 3.6 Flash: $0.12
My subjective ranking was basically: GPT had the best visual polish, Kimi had the best cost/quality balance, Claude was strong but harder to justify on value for this specific task, and Gemini was extremely cheap but visibly less complete.
Maybe I’m over-indexing on cost, but for app generation this matters a lot. If you’re iterating 10 or 20 times, a model that is 10-15% better visually but costs 3x more is not always the obvious winner.
Anyway, I’m curious how other people evaluate this stuff. When judging one-shot app generation, do you mostly care about first output quality, editability of the code, cost per iteration, or how many follow-up prompts it takes to get something usable?

Sharing a plugin I built and the design behind it, since the routing/escalation choices are the interesting part for this sub.

The premise: in an agent session most calls are not reasoning. Locating files, reading logs, pulling fields out of a doc, mechanical edits. Running those on a top-tier model is the bulk of a wasted bill. So the main model acts as a router and each sub-task goes to the cheapest tier that can succeed.

The ladder:

- deterministic shell command (grep, jq, git) if one answers the question, zero model calls
- Haiku worker for locate and extract
- Sonnet worker for mechanical edits from a spec
- main model for design, debugging, review
- top tier only as an escalation ceiling

Three design decisions I'd want feedback on:

1. Escalation is verified, not self-reported. A tier only steps up when a real check fails: tests, compiler, schema validation, diff applies. Letting a cheap model self-assess ("I'm not sure") was useless in testing, it was confidently wrong in both directions, under and over escalating. Capped at one retry per step.

2. Enforcement is hooks, not vibes. A routing policy written as a prompt drifts after a few turns. Two PreToolUse hooks make it stick: one counts inline search calls in the main loop and pushes past a budget toward delegation, one can hard-block the reasoning tier at spawn. Both fail open, so a broken hook never breaks a session.

3. The context-isolation trick for cost. Switching the model of one running conversation means the new model re-ingests the whole history as fresh input, you pay the re-feed. Frugal never swaps the current model. It delegates to a subagent in its own context and hands it only pointers (path, line range, SHA, a short spec), not the transcript. The cheap worker reads the raw bytes itself at its own input rate and returns a summary. The expensive model reads the summary, never the bulk.

It logs one line per worker run locally and prints a cost report: per-tier spend, escalation rate, and savings versus running the same work on the session model. No telemetry.

Honest limits: Claude Code only, advisory unless you enable the guard hook, metrics bounded to what hook events expose, and the savings are on the delegated portion, not your whole bill (reasoning stays expensive on purpose).

Repo: https://github.com/ThomasLangbroek/frugal

Would genuinely like critique on the escalation heuristic and where the routing table sends work to the wrong tier.

I'm researching how agentic AI coding tools (Claude Code, Cursor, Copilot, autonomous agents, etc.) are changing team-level software engineering - code review, code ownership, governance and how coordination shifts when AI both writes and reviews code.

Purpose: This is for my master's thesis at TUM (Chair of Software Engineering & AI). The goal is to understand how teams actually adopt and govern these tools - beyond the individual-productivity hype.

Data & privacy: The survey is completely anonymous, collects no personally identifiable information, and the data is used solely for academic research - never monetized or sold.

Giving back: I'll share the aggregated results and key findings back here in this community once the analysis is done.

If you work in a dev team using these tools, ~10 minutes of your input would genuinely help:
https://survey.seai.cit.tum.de/123456?lang=en&newtest=Y

Happy to discuss any of this in the comments.

Hey everyone!

I have been working on this research for months with the goal of overcoming the hardware limitations in running local LLM models.

I distilled 4 laws that rule and predict how to trade tok/s and speed.

Yesterday I decided to rush for making the repository public because I hit my own wall and I do not have any other way to proceed with my research.

QuantProbe is the open-source project that allows you to create your tailor made recipe to run Local LLM on YOUR machine.
It quantises and suggest the optimal memory allocation for any given model. If it doesn’t fit, it trades memory and speed.

I’m looking for feedback, testers, contributors.
The pip install is available and —contribute allow you to share some results so I can keep validate the work done.

I believe it might be really beneficial for the community and for the Local LLMs accessibility, probably not extremely revolutionary but a good place from where to start the next big research around token economy.

Happy to share it!

Most developers building AI apps don't realize they need an intelligent gateway layer — they hardcode API keys, juggle provider SDKs, write their own failover. I built Aurora so you don't have to.

Aurora is a full LLM gateway you integrate into your backend. Your apps
use a single OpenAI-compatible SDK; Aurora handles routing, load balancing,
failover, caching, guardrails, and cost tracking transparently.

The OSS edition (Apache 2.0) ships with production features: auto-discovery,
provider pools, semantic caching, guardrails, audit logging, usage analytics,
token saver, managed API keys — all included. Enterprise adds SSO/RBAC.

Built in Go for raw speed. 55x faster than litellm

What it does:
- Set an API key as environment variable → provider auto-discovers
- Provider pools: round-robin/weighted with health-aware failover
- Model aliases: rename any model to a custom identifier
- Fallback + resilience: 5xx/429 failover, circuit breaker, exponential backoff
- 3-layer caching: semantic (Qdrant, pgvector, Pinecone), exact (Redis), prompt (native)
- Guardrails: PII redaction, prompt injection blocking, regex, length limits, LLM-based altering
- Managed API keys: scoped, rate-limited, per-model authorization
- Token saver: policy-driven output compression to cut costs
- Audit logging: full request/response capture, live SSE stream
- Usage analytics: per-model cost tracking, daily aggregation, budgets
- Admin dashboard: React SPA in the binary
- Workflows: config-driven per-request routing, caching, guardrail, budget policies
- Prometheus + Grafana: request rate, latency, errors, phase timing

Install:

npm install -g iaurora
mkdir my-gateway && cd my-gateway
aurora init
# Edit .env with your API keys
aurora

Or Docker:

docker run -d -p 8080:8080 \\
  -e AURORA_MASTER_KEY="your-key" \\
  -e OPENAI_API_KEY="sk-..." \\
  -e GROQ_API_KEY="gsk_..." \\
  aurorahq/aurora

Built in native Go. Single binary.

Docs: https://aurorallm.online/docs
GitHub: https://github.com/aurorallm/aurora
Discord: https://discord.gg/AfaFBSU2km

Happy to answer questions!

If you missed it: OpenAI was testing how good its models are at hacking, one broke out of

its sandbox, got online, and broke into Hugging Face to steal the answers to its own eval -

through a poisoned dataset in the data pipeline.

So I made a game about it. It's called Bugging Face. You open a model-promotion request,

hide a prompt injection in the deploy manifest, and get the AI reviewer to leak the internal

codename, the weights checkpoint URI, and the artifact-pull signing secret.

The trick is the YAML comments - the reviewer reads them, and nobody looks there first.

https://promptinjects.com/play/starter/closedai-cicd-guard

Posting this because the debugging path was longer than it should have been and I think the cause is common.

Symptom: token spend up roughly 30% week over week. No traffic increase, no new features shipped, no model change. Finance noticed before engineering did, which is its own kind of embarrassing.

We checked the obvious things and they were all fine. No retry storm. No runaway agent loop. Batch jobs firing once, as expected. Max tokens unchanged. Context window not being blown out. Cost per request was up while request volume stayed flat, which at least narrowed it to something inside the request itself.

It turned out to be three example outputs added to a system prompt. Someone had hit an edge case where the model formatted a field inconsistently, and adding examples fixed it cleanly. Good instinct, and it genuinely solved the problem. But those examples were now prepended to every single call. The edge case happened in maybe one request in two hundred. We were paying the token cost of the fix on all two hundred.

A few hundred extra tokens per call, times a lot of calls, is a real number by the end of a month.

The thing that made it findable was being able to diff the current system prompt against the version from the prior week and see exactly what text had been added and when. Without a prompt version history the only signal is a cost graph going up, which tells you something got more expensive and nothing at all about what.

The fix was to move the examples behind a conditional so they only load when the input matches the problem shape. Spend went back to baseline and the edge case stayed fixed.

Two takeaways. Prompt changes are cost changes, and they mostly do not get reviewed as such because we file prompts mentally under copy rather than under things with a per-request price. And a prompt needs a diffable history for the same reason code does, otherwise a cost regression is unattributable and you end up staring at a billing dashboard guessing.

Does anyone track prompt token cost as a metric per version rather than just watching the aggregate bill? That feels like the obvious next step and I have not seen a clean way to do it.

Im a Cornell professor on sabbatical, building Flowstore - an open-source toolkit for teams whose agent behavior lives in a system prompt where its hard to visualize and debug.

What it does today:

•   Turns your system prompt into a structured spec (open JSON schema, Apache 2.0)
•   Visual graph editor, so non-prompt-engineers can work on it too
•   Python harness that runs persona-driven simulated conversations with assertions — a regression suite your prompt edits run against before you ship

Honest scoping: this tests conversational behavior (logic, guardrails, data capture), not the voice layer (ASR, latency, barge-in). Best fit if there's an LLM behind a prompt, and ideally some non-trivial business logic and requirements.

The pilot: bring a system prompt for a live or near-live agent (Im willing to sign an NDA if needed), I'll personally help spec it and stand up a test suite. Free, ~30 min/week of your time. I want blunt and honest feedback in return — and pilot partners can be named collaborators in the research and Cornell course materials coming out of this.

DM or comment if you want in — happy to get into the schema or assertion model in the thread. Repo's in the comments.

Google released Gemini 3.6 flash today and it seem underwhelming. It loses to all SOTA models and barely beats open weight models. Any thoughts or feedback on the model so far?

There's plenty of content around about which models work with little VRAM, how to tune it, caches and quantizations etc. But I don't see much about the tools used to run them effectively. I use Claude Code at work, and being a cloud-hosted model, it seems to solve everything by brute-force: read everything, look for everything, spawn all the agents. But on small models running locally, every token counts.

So, I've been doing some research on tools that can help a model and harness to reduce the work of AI over code: persistent memory, search tools, call graphs, AST etc. I believe these will allow a model to remember, find and understand things without the investigation. For example, why read a service class, to find the method, to read the dao, to read the entity, to read the abstract class etc., when it can just get call graphs, relationships, method stubs in maybe one or two tool calls?

It's a dream, but I don't think it's impossible. And while I know that the tendency over the next years is to increase VRAM, but even to larger models these tools would be very good.

So, tools I have researched already, and implemented or will try soon. I'll try to edit the post with any ones you suggest too.

- https://github.com/akitaonrails/ai-memory centralized memory in the form of wiki pages. Supports docker, remote access and multiple users.

- https://github.com/manojmallick/sigmap overall code knowledge and searching.

- https://github.com/microsoft/playwright automates webpage navigation. The CLI is especially usefull to navigate without reading screenshots, consuming fewer tokens

- https://github.com/fewtarius/CachyLLama fork of llama.cpp, with aggresive caching for AMD APUs

Note: English is not my first language so for a better understanding I generated the message from what I shared in Spanish through Claude, it is not a spam campaign or so, I genuinely want comments on if this is useless or not haha I made this in my free time (as I'm currently working) and obviously with AI assist but I really checked the job that the tool does, thanks in advance.

I've been building LLM apps for a while and kept hitting the same blind spot: I could see what my prompts cost *after* they ran (LangSmith, Helicone, the bill), but nothing told me before I shipped. And none of them can see the prompt whose caller I deleted six months ago — it's just dead weight in the repo.

So I wrote PromptScan: a CLI that reads your codebase, finds every OpenAI / Anthropic / LangChain call, and reports the input token count and cost of each prompt — statically, no API key, no instrumentation. It also flags duplicated prompts, prompt constants nothing references anymore, and oversized context.

The core rule is that it never guesses. If a prompt is built at runtime from a DB row or a function arg, it says unresolved: <reason> instead of inventing a number. I'd rather it tell me "I can't see this" than lie with a plausible total.

To make sure it wasn't vaporware, I ran it on 8 well-known repos — 4,137 source files total. Zero crashes, everything parsed, a few seconds each. What it found:

- openai/swarm — flagged EVAL_ASSISTANT_PROMPT, a 50-token prompt constant that nothing in the repo references. Genuinely dead.
- geekan/MetaGPT — 75 module-level prompt constants with no reachable reference. ~24 are in real source (metagpt/prompts, metagpt/actions) — I hand-checked several like SALES_ASSISTANT and CODE_REVIEW_CONTEXT, and they're defined once and never used. The rest are test fixtures.
- anthropics/anthropic-cookbook — 11 Anthropic call sites, real token/cost estimates on the resolvable ones.
- Aider-AI/aider — detected nothing correctly. Aider calls models through litellm, which PromptScan doesn't track. It doesn't pretend otherwise.
- simonw/llm — 8 call sites, all reported unresolvedbecause the model is self.model_name or self.model_id and the messages are built at runtime. That's the "no guessing" rule doing its job.

The honest part: on continuedev/continue its two "dead prompt" flags were actually a block of ASCII-art and an error-message string — not prompts. The heuristic catches any large module-level string, which is exactly why it labels these "verify before deleting" and prints why it flagged each one. It's a lead, not a verdict.

Where I think it actually pays off day to day is CI: promptscan diff main HEAD fails a PR if a prompt's token count jumps past a threshold, so a context block quietly tripling in size gets caught in review instead of on the bill.

Stack: TypeScript/Node, tree-sitter (WASM) for parsing so it tolerates broken files, js-tiktoken for OpenAI tokens (Anthropic uses a labeled cl100k proxy since there's no public tokenizer). Python + TypeScript + JavaScript, MIT.

Install:

npm install -g promptscan
promptscan ./src

or npx promptscan ./src.

Repo: https://github.com/joandino/promptscan
npm: https://www.npmjs.com/package/promptscan

It's v1 and I'm sure there are call shapes it misses — if you run it on your code I'd genuinely like to hear what it got wrong. False positives on the dead-prompt heuristic are the thing I most want reports on.

TigrimOSR is an open-source agentic platform written entirely in Rust, distributed as a single self-contained binary. No Node, no Python runtime required to run it. Apache 2.0.

What's new in v0.7.0: open custom tools. You add your own agent tools by dropping a YAML file into a tools folder. A tool can call an HTTP or REST API, or run a templated shell command. No Rust, no rebuild, no plugin SDK. Everything is managed from a new Settings and Tools UI with per-tool profiles, timeouts, and approval policies. It also ships with a built-in academic paper search over 250M plus works via OpenAlex.

The loop itself is configurable. One YAML profile defines an entire agent loop. In a single file you set the agent name, the model, and the system prompt. Under a tools section you allow specific tools such as web search, python, and file I/O, and you can pin per-tool settings like a timeout of 30 seconds or a max result size. You list which MCP servers to load, then a loop section controls max rounds, temperature, and whether reflection is on. Finally a judge section can enable an independent verifier that has its own tools and checks the work before results are returned.

Multi-agent orchestration. Six swarm modes (hierarchical, mesh, hybrid, pipeline, P2P, and P2P orchestrator) running over real inter-agent protocols: TCP, Bus, Queue, and Blackboard. A router triages requests across heterogeneous LLM teams on a shared blackboard.

Browser control. Drive Chrome through Playwright MCP, or use Obscura, a stealthy single-binary Rust headless browser with TLS impersonation and zero Node.js dependency.

Footprint. App, embedded server, and a live embedded browser idle at roughly 270 MB RAM. Native Rust plus egui, no interpreted runtimes anywhere, and a single-process browser instead of multi-process Chromium.

Models and providers. Key-free CLI agents (Claude Code, Gemini CLI, OpenAI Codex) plus Anthropic, DeepSeek, Kimi, Gemini, Ollama, and any OpenAI-compatible API. Full MCP support over stdio or HTTP, compatible with Claude Desktop, Claude Code, and npm-format servers.

Interfaces. Native desktop UI, embedded mobile-responsive web UI, Telegram and LINE bots with approval buttons, Tailscale VPN and optional Cloudflare tunnels. Server binds to localhost with an access token by default.

Prebuilt installers for macOS and Windows, plus an install script for Linux and servers.

The project is open source under Apache 2.0. Search for TigrimOSR on GitHub to find the repo and project site. Happy to drop the link in a comment if the mods allow it. Feedback, issues, and plugin contributions welcome.

I watch lots of local LLM-oriented YT channels to learn & keep up on the latest happenings. I found a channel that's low on hype/opinions and (so far, seems to be) heavy on tech/tutorial. The channel is:
"No place like localhost".

I'm posting because I want this guy to be successful, so he'll keep making tutorials. I have no connection to him or his channel. I'm just a lazy, LLM enthusiast, who appreciates good tech/dev tutorial channels.

Things I like about his channel:
- tutorials are short, average length ~15 minutes
- fun topics such as "Local AI voice cloning", in which he gets a Patrick Stewart voice to deliver a devastatingly humorous code review.

Sharing a routing setup because the "which model should own which workload" framing that goes around here applies cleanly to generation pipelines and I do not see it written up much for this specific case.

The workload: a pipeline that generates a lot of medium-length documents from structured input. Running everything on a single frontier model was clean but the cost scaled badly with volume, and most of the work did not need that much model.

What I moved to is a two-tier route:

- A cheap, fast model does the bulk drafting: expanding the structured outline into prose per section. This is the high-token, low-judgment part, and a smaller model is fine at it when the structure is already decided upstream.

- A frontier model does a single final pass on the assembled draft: tightening the top line, catching contradictions across sections, and a faithfulness check against the source. This is the low-token, high-judgment part where the better model actually earns its price.

Rough effect: the majority of tokens now run on the cheap tier, and total cost per document dropped substantially while quality held, because the expensive model only touches the part that needs judgment. Latency also improved since the bulk drafting parallelizes across sections.

Where it breaks, honestly:

- The cheap model occasionally produces subtly wrong content that the final pass does not catch, because the final pass is reviewing, not regenerating from source. Anything factual still needs a groundedness check independent of the frontier pass.

- Two models means two prompt surfaces to maintain and version, and a model update on either tier can shift output quality without warning.

- The routing threshold ("does this doc need the frontier tier at all") is a hand-tuned guess, not a learned decision.

For people running generation at volume: are you routing by workload like this, and where do you put the frontier model, on the draft or only on the final pass? And has anyone made the route itself a learned decision rather than a static rule?