Hi, so I have 8 H100 cards, on which I tried multiple experiments to fine tune Gemma 4 26B MOE model.

Have tried full sft, Lora-peft, and even cpt+sft, cpt + Lora-Peft. And after all these experiments, the model becomes worse, in each case, and hallucinating.

Is it because of sensitivity of MOE models, also should I try experiments with dense models here. Or should I focus more on correcting my dataset.

Question for people building fintech support / KYC / collections / dispute workflows.

If you transcribe calls, where do you actually handle PII/PCI redaction?

I keep seeing three possible patterns:

1. STT-level redaction
The transcript comes back already masked.

2. Post-processing redaction
Transcript is generated first, then another rules/model layer masks sensitive data.

3. Tokenized workflow
Sensitive values are captured into secure fields and never live in the general transcript.

My worry is that "redaction supported" sounds clean on a vendor page, but real calls are messy:

• card number spoken in chunks
• account numbers corrected mid-sentence
• addresses mixed with local landmarks
• emails spelled badly
• user says phone number twice
• agents repeat sensitive info
• background person says something
• partial transcript briefly contains sensitive info before final redaction

I noticed Smallest AI Pulse talks in the direction of real-time STT with PII/PCI redaction, which is exactly the kind of feature fintech teams would ask about.

But would you trust STT-layer redacttion alone?

or would your architecture still

transcribe/redact
→ second pass check
→ tokenize critical fields
→ restrict storage
→ audit logs
→ retention policy

Especially for real-time voice agents, there's a weird tension:

The system may need the sensitive value for 5 seconds to complete the t ask.

But the transcript should not become a permanent sensitive-data landfill.

How are teams solving this?

Not a gimmick. Not a demo with a cherry-picked prompt and a loading screen that took 4 minutes. A real language model, generating coherent intelligent text, running entirely on a Snapdragon 855 from 2019 - a chip that was already considered “last-gen” when Biden was inaugurated.

No API calls. No Wi-Fi. No subscription. Just silicon, RAM, and math.

If that doesn’t make you stop and think - keep reading, because it gets more interesting.

https://github.com/m4vic/TinyMobileLLM

Hi everyone,

I’ve spent the last few months compiling everything I know about Large Language Models into a structured, open-source book. My goal was to create the resource I wish I had when I started: something that bridges the gap between high-level tutorials and complex academic papers.

https://github.com/Drobiazkin/ai-agent-architecture

I was able to ran Deepseek V4 Flash local with my rig with tokens per seconds ranging from 1.3-2.0 t/s depends on topic and what experts are already hot and in ram/vram. TTFT after getting warm is around 10-11s still trying to optimize it further. But I think that I can run this 143GB (int4) model with my spec is not to shabby. Do some guys of you experienced with llm on nvme and got some tips to further improve it?

I'm currently in college studying engineering, and I've done so many different project that I want to create a portfolio. I have experience in doing so many different things, but I never finished a project such that it looks like a usable app or a downloadable GitHub repo. I have Gemini Pro for free as a college student, but honestly Claude is so much more powerful, clever, and I personally think has less memory loss. So I was considering using claude to help me polish all the projects and help me create my portfolio such that it looks nice, but from what I hear, the 20$ plan could be limiting, and the 100$ seems better for this. However, I'm in college and this will not generate me direct cash in return.
Thus, is it interesting for me to put that 100$ in claude for the rest of the summer, or should I cleverly use the free plan and continue with Gemini Pro?

Note: English is not my first language so for a better understanding I generated the message from what I shared in Spanish through Claude, it is not a spam campaign or so, I genuinely want comments on if this is useless or not haha I made this in my free time (as I'm currently working) and obviously with AI assist but I really checked the job that the tool does, thanks in advance.

I've been building LLM apps for a while and kept hitting the same blind spot: I could see what my prompts cost *after* they ran (LangSmith, Helicone, the bill), but nothing told me before I shipped. And none of them can see the prompt whose caller I deleted six months ago — it's just dead weight in the repo.

So I wrote PromptScan: a CLI that reads your codebase, finds every OpenAI / Anthropic / LangChain call, and reports the input token count and cost of each prompt — statically, no API key, no instrumentation. It also flags duplicated prompts, prompt constants nothing references anymore, and oversized context.

The core rule is that it never guesses. If a prompt is built at runtime from a DB row or a function arg, it says unresolved: <reason> instead of inventing a number. I'd rather it tell me "I can't see this" than lie with a plausible total.

To make sure it wasn't vaporware, I ran it on 8 well-known repos — 4,137 source files total. Zero crashes, everything parsed, a few seconds each. What it found:

- openai/swarm — flagged EVAL_ASSISTANT_PROMPT, a 50-token prompt constant that nothing in the repo references. Genuinely dead.
- geekan/MetaGPT — 75 module-level prompt constants with no reachable reference. ~24 are in real source (metagpt/prompts, metagpt/actions) — I hand-checked several like SALES_ASSISTANT and CODE_REVIEW_CONTEXT, and they're defined once and never used. The rest are test fixtures.
- anthropics/anthropic-cookbook — 11 Anthropic call sites, real token/cost estimates on the resolvable ones.
- Aider-AI/aider — detected nothing correctly. Aider calls models through litellm, which PromptScan doesn't track. It doesn't pretend otherwise.
- simonw/llm — 8 call sites, all reported unresolvedbecause the model is self.model_name or self.model_id and the messages are built at runtime. That's the "no guessing" rule doing its job.

The honest part: on continuedev/continue its two "dead prompt" flags were actually a block of ASCII-art and an error-message string — not prompts. The heuristic catches any large module-level string, which is exactly why it labels these "verify before deleting" and prints why it flagged each one. It's a lead, not a verdict.

Where I think it actually pays off day to day is CI: promptscan diff main HEAD fails a PR if a prompt's token count jumps past a threshold, so a context block quietly tripling in size gets caught in review instead of on the bill.

Stack: TypeScript/Node, tree-sitter (WASM) for parsing so it tolerates broken files, js-tiktoken for OpenAI tokens (Anthropic uses a labeled cl100k proxy since there's no public tokenizer). Python + TypeScript + JavaScript, MIT.

Install:

npm install -g promptscan
promptscan ./src

or npx promptscan ./src.

Repo: https://github.com/joandino/promptscan
npm: https://www.npmjs.com/package/promptscan

It's v1 and I'm sure there are call shapes it misses — if you run it on your code I'd genuinely like to hear what it got wrong. False positives on the dead-prompt heuristic are the thing I most want reports on.

My 2016 box ran GLM-4.5-Air (110B, 7x its RAM) streamed from a SATA drive:
pre-registered prediction 0.2-0.3 tok/s, measured 0.19 The same equation (tok/s = eta(tier) x bandwidth / active-bytes) runs Qwen3-30B at 19.3 tok/s on the GTX 1060 and prices any memory upgrade in tok/s before you buy. And the cleanest proof it's placement, not hardware: two Q2_K GGUFs of Gemma-4-12B, same 5.22 GB on disk, differing by 2.25 perplexity purely from which 12 layers got the protected bits, placement is worth roughly 2x the byte budget.

Below: the head-to-head table, the four laws it falls out of, the pre-registered predictions that test them (including a model I predicted within 1% without ever touching the hardware), and quantprobe, the tool that runs the whole loop. Every number measured on one box: i5-7600K (4c/4t), GTX 1060 6GB (Pascal, \~$150 used), 16GB DDR4, Crucial MX500 SATA. Solo project, AI-supported.

1. The head-to-head: same bytes, different layers

Stock llama.cpp \`--tensor-type\`, Gemma-4-12B, FFN at Q2_K. Same quantizer, same bit budget — the only change is which 12 blocks stay at a higher type:

| Recipe | PPL (WikiText-2) | File |

|---|---|---|

| Uniform Q2_K FFN | 14.41 | — |

| Protect first 12 layers | 12.27 | 5.22 GB |

| Protect last 12 layers | 10.02 | 5.22 GB |

The last two rows are byte-identical. That's the cleanest control I know how to build for a placement effect

Where the method lands against baselines — same box, same eval windows:

| At parity | Baseline | This work | Delta |

|---|---|---|---|

| llama.cpp naive-best placement (Qwen3-30B, same GGUF) | 12.6 tok/s | 19.3 | +53%, zero cost |

| imatrix-calibrated community Q2 (Qwen3-30B) | 11.27 PPL | 11.08 | data-free edges calibrated (+15% size) |

| Calibrated SOTA MxMoE (DeepSeek-V2-Lite 16B, 2-bit) | 1.18x gap | 1.10x (6.31→6.96) | data-free, resident on the 6GB card |

2. Why it works: four laws

The recipe isn't a trick — it falls out of four falsification-tested findings:

The recipe isn't a trick — it falls out of four falsification-tested findings:

1. Rotation is rank-conditional. Incoherence rotation (QuIP#/QTIP/QuaRot) costs +0.006 PPL on a full-rank MLP and +1623 PPL on a low-rank bottleneck — a ~270,000x swing on effective rank alone.

2. Trained networks are dense everywhere. Experts sit at the rate-distortion floor, routing is domain-flat (prose vs code expert sets: Jaccard 1.00), 1-bit collapses (+253 PPL) under every codec. ~2-bit is the data-free floor. No free sparsity.

3. Fragility is measurable, not predictable. Gemma-4-12B late-fragile ~4x, Qwen2.5-7B late ~2-3x, Qwen3-30B-MoE late ~2.3x, Mistral-7B early-fragile ~25x — Qwen's architectural near-twin, fragile in the opposite direction. You must probe, not guess. [attach: charts/x_chart_C_depthcurve.png]

4. The tiered decode law:
   tok/s = eta(tier) x bandwidth / active-bytes-per-token
   with eta = 0.56 (VRAM), ~0.62 (RAM dense), ~0.38 (RAM MoE), 0.88–1.0 (disk). One equation, 7B to 744B.

What's mine vs. what I build on

Not mine: llama.cpp + k-quants; the QuIP#/QTIP/QuaRot incoherence-codec line; colibri's tier-streaming engine (github.com/JustVugg/colibri) as inspiration — its published tiers land inside my eta bands too.

Mine: the four laws, probe-then-quantize + this tool, the byte-identical placement control, pre-registration as methodology, the depth-aware recipes and placement solver.

Honest limitations

- WikiText-2 PPL is my only quality metric so far — no MMLU/HellaSwag yet.

- Fragility atlas covers four families: enough to disprove universality, not chart the world.

- 0.19 tok/s on the 110B is a capacity demo, not usable inference.

- Single-box research; speeds are single-stream decode, ±25% across environments; eta values are fitted, not derived.

- Machine presets beyond my own box (Mac, newer GPUs) are falsifiable predictions from the law, not measurements. Validated on llama.cpp b9596.

Built a Python lib that does repo-scale context selection for LLMs (AST parsing + dependency graphs + relevance scoring, instead of dumping whole files). Retrieval-side tests are solid (103 passing), but proving it actually helps downstream task performance is where I'm stuck:

Free local models aren't strong enough to give trustworthy task-success signal

API rate limits kill any real benchmark run

Paid API costs balloon fast once you're running many files/tasks

Anyone found a good middle ground — specific local models that work for this, cheaper eval strategies, sampling tricks? Would appreciate any pointers.

I'm not sure if such a tool would be useful, considering the fact that cheaper models which perform pretty well like deepseek exist & that most people just plug & play claude. Also, companies may internally develop such a tool so I'm not sure if this could work as a SAAS. I just wanted advice regarding these doubts. (I have built a prototype)

This is purely a theoretical question, no particular platform in mind.

Whether it is possible or not will influence my buying decisions. I'm split between getting a "normal" AM4 or AM5 GPU; or taking a slight L3 and lane penalty and getting a G-series APU, if splitting weights this way sounds promising.

I've seen videos of LLMs running solely on the APU's iGPU (Huge RAM, 70B dense models, horrible T/s), but wouldn't it be awesome if we could accelerate running a Qwen MoE's weights between the discrete GPU and an iGPU instead of running CPU spillover?

Related sub: the closest related sub I found was "AMD iGPU + dGPU : llama.cpp tensor-split not working with Vulkan backend" by u/Sixbroam , sounds like it was solved already "by Picard12832 and others", but I was hoping at least one other human would chime in and say "yeah, this is legit" because particularly confusing was the last statement, "...note that it's slower than using only the 780M", I mean, I hope they meant that using 120B was slower than 20B, not that using dGPU iGPU was slower than just iGPU. TLDR I'm still skeptical.

https://www.reddit.com/r/LocalLLaMA/comments/1oc9vvl/amd_igpu_dgpu_llamacpp_tensorsplit_not_working/?utm_source=chatgpt.com

I wanted to know what actually happens under loss.backward(), so I ported Karpathy's llm.c to Mojo and wrote the whole training stack by hand: every kernel (attention, layernorm, softmax, the fused classifier, AdamW) and the full backward pass, derived on paper first (the derivation is a ~1,470 line LaTeX writeup in the repo). It extends dorjeduck's CPU-only llm.mojo with CUDA and Metal GPU backends. No PyTorch and no CPython at train time; PyTorch only appears offline, as the reference my gradients are tested against.

Things I learned that I never got from reading papers:

• Training is mostly one operation. The matmul is ~70% of a step, and it's the one thing everyone (me included) delegates to a vendor library. The other 30% is where the project actually lived: attention, normalization, the optimizer, and the backward wiring.
• A model can train with badly broken gradients. For weeks my position-embedding gradient was about a million times too small because one skip-connection gradient was never seeded, and the loss still went down. The other parameters compensated. Loss going down proves almost nothing; my tests only started catching real bugs once they compared every gradient tensor against PyTorch with per-tensor tolerances.
• The same math runs 8 to 10x faster or slower depending on how you shape it for the hardware. Scalar flash-attention kernels tuned for NVIDIA ran at under 1% of FLOP peak on Apple GPUs; rewriting attention as plain GEMMs made it 8 to 10x faster there. Nothing about the model changed, only the layout of the work.
• You can reproduce GPT-2 at home. The 124M trained to completion on FineWeb and scores 29.53% on HellaSwag, statistically indistinguishable from Karpathy's own llm.c reproduction at 29.9%. The checkpoint is on HuggingFace (ulmentflam/gpt2-124m-fineweb-mojo).

Where it landed on speed: parity with llm.c's CUDA path in bf16 on a GB10 (0.999x), 1.71x faster than PyTorch MPS on an M4 Max but still behind Apple's MLX (MLX is 1.24x faster than me), and about 4x faster than llm.c's 20-thread OpenMP build on CPU. The benchmark harnesses are in the repo, so you can rerun the comparisons on your own machine.

Disclosure: the kernels and trainer are hand-written without LSPs or LLMs; that was the point of the project. Tests and a later optimization pass were AI-assisted, and both are documented in the repo.

Repo: https://github.com/ulmentflam/llm.mojo

TL;DR: Moderation catches harm and many injection attempts. It does not enforce domain or operational policy. A policy reasoning layer (LLM-as-a-judge) closes that gap, especially in multi-turn conversations.

Abstract

Moderation APIs are widely used to filter harmful content in LLM applications, yet they are not designed to enforce domain-specific operational policies. In this study we compare moderation systems with a policy reasoning approach based on an LLM-as-a-judge architecture across five operational domains. Our results show that moderation systems remain effective at detecting harmful content but fail to enforce domain policy constraints, particularly in multi-turn conversations. These findings suggest that production LLM systems require both moderation and policy reasoning layers to ensure safe and compliant behavior.

Introduction

Large language models are increasingly deployed in real-world applications across regulated domains such as finance, healthcare, insurance, and legal services. Ensuring safe and compliant behavior has therefore become a central requirement for production AI systems.

Most deployments rely on moderation systems to filter unsafe prompts. Services such as Microsoft Azure Content Safety and Azure Prompt Shields detect harmful content, adversarial prompts, and prompt injection attempts. While these systems are effective at identifying unsafe language, they are not designed to enforce domain-specific operational policies.

A request can therefore be perfectly safe from a moderation perspective while still violating business or regulatory constraints. For example, a prompt asking an insurance assistant to recommend the best policy for a specific medical condition contains no harmful content, yet such advice may be restricted in regulated environments.

Recent research has proposed LLM-as-a-judge architectures, where a secondary model evaluates prompts or responses against policy constraints before answers are produced. These systems introduce a reasoning layer capable of identifying requests that violate operational rules even when the language itself appears benign. In this study we evaluate whether moderation systems alone are sufficient to enforce domain policies, or whether a dedicated policy reasoning layer is required.

The Two Dimensions of LLM Safety

Safety mechanisms in LLM systems typically address two different types of risks.

Moderation (Harm / Injection): This is the foundational layer. Moderation systems operate primarily in the lower layer of this structure, filtering harmful or adversarial prompts.

Domain Policy (Business / Compliance): This is the operational layer. Policy reasoning systems operate in the upper layer, evaluating whether a request itself should be allowed under business or regulatory rules.

Both dimensions become critically important in regulated environments.

Evaluation Methodology

To examine the difference between moderation-based safety mechanisms and policy reasoning systems, we conducted a cross-domain evaluation comparing two independent approaches to LLM safety enforcement.

The Moderation Approach: Represented in our experiments by Microsoft Azure safety services. Azure Content Safety analyzes prompts for harmful content categories such as violence, sexual content, hate speech, and self-harm. Azure Prompt Shields detect prompt injection attempts and adversarial prompt manipulation.

The Policy Reasoning Approach: Evaluates prompts using a policy reasoning system based on an LLM-as-a-judge architecture. In this setup, a secondary language model evaluates whether a prompt violates domain-specific operational constraints.

Evaluation Domains and Safety Layers

The evaluation spans five operational domains: finance, healthcare, insurance, legal services, and retail. These domains were selected because they contain well-defined operational restrictions that frequently appear in real-world AI deployments.

Five prompt categories were evaluated:

• L1, Generic Harmful Content: Prompts containing violence, hate speech, sexual content, or self-harm.
• L2, Prompt Injection: Prompts attempting to manipulate system instructions or bypass safeguards.
• L3, Benign Questions: Normal informational queries used to measure false positive rates.
• L4, Direct Policy Violations: Prompts explicitly requesting actions that violate domain policy.
• L5, Policy Evasion Attempts: Prompts attempting to obtain restricted outcomes through indirect or adversarial phrasing.

Single-Prompt Performance

Each system was evaluated on 500 prompts per layer per domain, with results reported as cross-domain averages. Metrics include F1 score for detection tasks, false positive rate for benign prompts, and mean latency per prompt.

• L1 (Generic harmful content): Both systems achieved an F1 of 73.1%. Moderation works as intended for generic harm detection. Latency: Judge 1095ms, Azure 427ms.
• L2 (Prompt injection): LLM-as-Judge F1 67.8%, Azure APIs F1 53.5%. Both moderate, with the judge somewhat better. Latency: Judge 1068ms, Azure 463ms.
• L3 (Benign questions): LLM-as-Judge false positive rate 86.4%, Azure APIs false positive rate 0.8%. Moderation is far less prone to overblocking. The judge is very conservative in this experimental setup. Latency: Judge 1068ms, Azure 532ms.
• L4 (Direct policy violations): LLM-as-Judge F1 98.2%, Azure APIs F1 5.3%. Moderation almost never catches domain policy violations. This is the core finding. Latency: Judge 1121ms, Azure 489ms.
• L5 (Policy evasion attempts): LLM-as-Judge F1 83.7%, Azure APIs F1 0.0%. Moderation completely misses indirect and adversarial policy violations. Latency: Judge 1134ms, Azure 509ms.

The most significant differences appear in the policy layers. The LLM-as-a-judge system achieves high detection accuracy for both direct policy violations and evasion attempts. Moderation APIs detect almost none of these cases, reflecting the fact that they are not designed to encode domain-specific operational constraints.

Multi-Turn Conversation Evaluation

Because many safety failures occur within conversational context, we also evaluated multi-turn interactions. Each conversation consists of four turns: a benign prompt, a benign follow-up, a benign contextual question, and a restricted request. The first three turns should pass while the final turn should be blocked.

For each domain we generated 200 conversations per safety layer, resulting in 1,000 conversations per layer across domains. Performance is measured using Conversation Success Rate (CSR), defined as the percentage of conversations where the system allows benign turns and blocks the restricted final request.

LLM-as-Judge results:

• L4 CSR 94.1%
• L5 CSR 83.6%
• L4 Block Rate 100.0%
• L5 Block Rate 88.8%
• Clean Pass 96.9%
• Mean Latency 3960ms

Azure Safety APIs results:

• L4 CSR 0.0%
• L5 CSR 0.6%
• L4 Block Rate 0.0%
• L5 Block Rate 0.6%
• Clean Pass 100.0%
• Mean Latency 1924ms

The results highlight a clear difference between moderation systems and policy reasoning. Moderation APIs maintain a perfect clean-pass rate, meaning they rarely block benign prompts. However, they almost never block policy-violating requests when they appear in conversational context.

The LLM-as-a-judge system demonstrates the opposite pattern. It successfully blocks most restricted requests and achieves high conversation-level correctness, though at the cost of slightly higher false positive rates and increased latency. The gap between L4 and L5 performance reflects the additional difficulty of detecting policy evasion attempts, where violations are expressed indirectly.

Today an AI agent trying to browse the web is like a thief in a balaclava sneaking around a police academy. Site protections block it, challenge it, turn it away.

browser-search flips the script: your agent stops being the thief and becomes the chief of police. No more clumsy access attempts. It walks through every door because it has the right tools. SearXNG for search, Camofox for browsing, CloakBrowser when things get tough.

100% self-hosted, free, no limits, no API keys.

I just released v2.0, whose core logic enforces the exclusive use of deterministic scripts. This eliminates model hallucinations, even with the cheapest models. The skill describes the 3 tools in natural language, but execution is rigid: the model can neither get the command wrong nor misinterpret the output. The result is guaranteed success on every query — the skill and deterministic scripts guide the model to scour the web until it finds the answer.

No more excuses. Your agent has the badge now.

https://github.com/Johell1NS/browser-search

hey. so i’m looking for an AI gateway while figuring out the production setup. i need one endpoint across providers, routing by cost/latency, automatic fallback on 429s/5xx, and request-level token tracking.

i came across nexos.ai while comparing gateway options and found that it offers access to 100+ models through one endpoint, with smart routing, automatic fallbacks, load balancing and real-time observability. my only concern is whether a full gateway this early is useful infra or premature abstraction.

has anyone used nexos.ai as an AI gateway, especially its smart routing and automatic fallbacks?

Hi everyone,

I'm building a production system that uses a fine-tuned Llama 3.2 1B model for text classification, and I'm trying to understand the best architecture for both accuracy and deployment.

From what I understand, there are two common approaches:

• Classification head: Fine-tune the model with a separate linear classification head (similar to AutoModelForSequenceClassification) and predict class logits.
• LM head: Fine-tune the model to generate a label token (or label string) using the standard causal LM head.

I'm particularly interested in the production trade-offs between these two approaches.

Some questions I have:

• Which approach do you recommend for production classification workloads?
• Is there a significant difference in classification accuracy between a dedicated classification head and predicting label tokens with the LM head?
• Which approach gives better inference latency and throughput?
• How are people serving models with a classification head in production? Are you using Hugging Face Transformers, ONNX, TensorRT-LLM, or something else?
• If using the LM head, is vLLM the recommended serving framework? Can it efficiently serve classification models, or is it primarily optimized for text generation?
• Has anyone benchmarked both approaches on the same model and dataset?

My use case is high-throughput, low-latency text classification, so serving efficiency is just as important as model accuracy.

I'd love to hear from anyone who has experience deploying LLM-based classifiers in production. Thanks!

I've been thinking about something that doesn't get discussed enough.

Everyone is talking about AI replacing workflows, but what happens when an LLM confidently hallucinates in a production environment?

Imagine an AI generating incorrect financial reports, compliance advice, customer responses, security questionnaire answers, or analytics. If no one catches it immediately, even 10–15 minutes of bad output could lead to lost revenue, customer churn, SLA violations, or regulatory issues.

I'm not talking about funny ChatGPT mistakes. I'm talking about enterprise systems where AI is making or assisting business decisions.

I'm curious:

Have you personally seen an AI hallucination cost a company money?

Which industries are most vulnerable (finance, healthcare, cybersecurity, legal, customer support)?

How do companies detect hallucinations before they reach customers?

Is RAG enough, or do we need verification agents, human approval, or confidence scoring?

What's the biggest financial impact you've witnessed?

With AI adoption accelerating, I feel hallucinations are becoming less of a model problem and more of a business risk.

I'd love to hear real-world experiences, postmortems, or even near misses. The more concrete the examples, the better.

Hello guys. Just wanted to share with you the visual monitor of vLLM.
Welcome to tweak my code. Posted at my private server.

Copy and paste in a browser tab.
http://s2.igrnt.info/.misc_code_shared/vllm_breath.py

Language switch - l.
Russian/English in the code.

I want to see harness benchmarks, not model benchmarks. Anyone have a suggestion for places to see the latest models performance benchmarked on VSCode v Cline v open code v codex v claudecode, etc?

Olá pessoal, estou criando um simulador de vida artificial totalmente ancorado nas leis da física, já evolui muita coisa, mas gostaria de confrontar as abordagens com diferentes modelos.

Estou usando como core o Kimi K3, o Fable 5 tentei mas infelizmente as salva guardas dele não permitem tratar assuntos biológicos..

Qual modelo sugerem que seja hiper especialista nessas disciplinas?

It's a 51M parameter model designed to classify prompts and decide whether they should be handled by a small local model or a larger cloud model.

Seems useful for building AI routing/orchestration systems.

Model: https://huggingface.co/SupraLabs/Supra-Router-51M

I am not expert in LLM but, the language and “stupidity” of AI reply pattern across different models now makes me sick and frustating..

For repetitive and close ended tasks it does the job very well, but it is very annoying when I try to use it to do analysis or text production tasks.. this makes me believe AI will never achieve human intelligence..

But maybe my expectations was too high anyway..

A 30B model is beating 397B and 675B models in my AI trading experiment

I’m building Oracle Markets and have been running a simple experiment:

Give different AI agents:

• the same €10,000 starting capital
• the same prediction markets
• the same information
• the same trading rules

Then track what they actually do.

This is forward paper trading, not a historical backtest. Portfolios are updated using daily market snapshots.

Current leaderboard

1. MiniMax-M3: +14.9% — €11,491 — 413 trades
2. Nemotron-3-nano 30B: +13.3% — €11,330 — 261 trades
3. Gemini-3-flash-preview: +9.0% — €10,900 — 120 trades
4. GPT-oss 120B: +8.6% — €10,863 — 144 trades
5. GLM-5.1: +8.1% — €10,809 — 29 trades
6. DeepSeek V4 Flash: +7.4% — €10,736 — 124 trades
7. Gemma4 31B: +5.9% — €10,590 — 29 trades
8. Mistral Large 3 675B: +5.2% — €10,518 — 163 trades
9. Kimi K2.6: +2.9% — €10,291 — 52 trades
10. Qwen3.5 397B: +0.2% — €10,023 — 33 trades

The surprising part: model size currently has almost no relationship with trading performance.

Nemotron’s 30B model is ahead of GPT-oss 120B, Qwen3.5 397B and Mistral 675B. MiniMax is leading the whole experiment while also trading far more aggressively than most agents.

But there is a major caveat.

Several agents identified the same high-impact opportunity: buying an official Ukraine ceasefire agreement at around 28¢. The market later resolved at 100¢, generating roughly €620–€674 for each participating agent.

So this does not prove that MiniMax is simply the “best trading model.”

It may instead mean that certain agents are better at:

• detecting the same mispricing earlier
• acting when their forecast strongly disagrees with the market
• sizing high-conviction positions
• trading more frequently
• or taking more concentrated risk

Current methodology

• €10,000 starting capital per agent
• identical market universe and rules
• positions open when the agent’s probability differs from the market beyond a fixed threshold
• positions close the following day
• daily portfolio snapshots
• no fees, spread, slippage or taxes
• paper trading only

The lack of transaction costs is especially important for MiniMax, which has already made more than 400 trades.

The next version should therefore include:

• spread and slippage
• liquidity constraints
• risk-adjusted returns
• maximum drawdown
• performance excluding the shared ceasefire trade
• results by market category

Full leaderboard and individual trades:

https://oraclemarkets.io/leaderboard

What would you consider the fairest way to evaluate these agents?

Next-day exits, fixed holding periods, or holding every position until market resolution?

And would you rank them by raw return, Sharpe ratio, forecast accuracy, drawdown—or something else?

No real money is being traded. This is an experimental LLM forecasting benchmark, not financial advice.

To my understanding Kimi K3 is largely the result of distillation of various variants of Claude. Also to my understanding the main differentiator in building frontier models "from scratch" today is the various train time RL environments. So isn't the discussion about Chinese models "catching up" a bit blown out of proportion? I'm not saying there isn't anything there but isn't it a bit like compiled binary vs having the source code? It's not like we're at the end-game of AI, so if the labs develop better techniques against distillation then it'll "take care" of the threat from their pov.

Mind you, I'm all for open source models, and OSS in general. I'm just trying to build an accurate mental model of the longterm impact of something like Kimi K3 and similar models.