This might be obvious to people here, but I think a lot of AI discourse still frames the open source vs closed source fight too narrowly.

People keep talking like the whole battle is:

“Can open source catch up on model quality?”

And yeah, model quality matters. Obviously.

But the more I follow this space, the more I think open source AI has a deeper problem:

Coordination.

Closed labs have one giant advantage that is not just talent or compute.

They have organized money.

They can decide:

• what to train
• who works on it
• what hardware to use
• what data to buy
• what benchmarks matter
• what product gets built
• when to ship
• what gets funded next

Open source has insane talent, but it is fragmented.

Someone releases a great model.

Someone else quantizes it.

Someone else makes a UI.

Someone else writes evals.

Someone else fine-tunes it for a niche.

Someone else runs inference.

Someone else does safety testing.

Someone else documents half of it in a GitHub issue that disappears into the void.

It is amazing, but also chaotic.

And that chaos is part of the magic, but also part of the weakness.

The hard questions are not just technical:

Who pays for training runs?

Who pays for inference?

Who decides what work is valuable?

How do you stop people from gaming benchmarks?

How do you reward people who improve the ecosystem but do not build the final shiny model?

How do you coordinate compute that is scattered across a thousand people?

How do you validate contributions without turning everything into fake leaderboard farming?

This is why I find decentralized AI experiments interesting even though a lot of them make me instantly skeptical.

Not because “token fixes AI.”

That phrase should be illegal.

But because incentives and coordination are real problems.

I was reading about Bittensor recently, and the part I found interesting was not the coin price stuff. It was the idea of subnets as separate markets for different types of machine intelligence work. In theory, that is at least an attempt to coordinate and reward useful AI work outside a single company structure.

Then you see products around it like mentat, which try to make the subnet side understandable for normal users because the raw ecosystem is way too complex. Again, not saying that solves open source AI. It obviously does not. But it made me think about the bigger pattern.

Infra gets built.

Then incentives get messy.

Then dashboards appear.

Then abstraction layers appear.

Then maybe normal people can use it without understanding the entire machine underneath.

I don’t know if decentralized AI wins anything important.

Maybe it gets gamed.

Maybe centralized labs stay ahead forever.

Maybe the best open source path is just companies and foundations funding more releases.

But I do think “open source AI needs better coordination” is a real topic.

Not just better models.

Better funding loops.

Better contribution tracking.

Better evals.

Better compute coordination.

Better incentives for unsexy work.

Better ways to reward people who improve the stack without needing to turn everything into a startup.

Curious what people here think.

Is open source AI mostly bottlenecked by model quality and compute?

Or is coordination/funding/incentive design the thing we keep underestimating?

I wanted to see if I could build a Top 5 sub-1B coding model using only free hardware. I took Qwen2.5-Coder-0.5B-Instruct and fine-tuned it on 50K execution-verified Python samples using a Kaggle T4 GPU.

The results on EvalPlus (strict pass@1):

• Atlas-Coder-2 (0.5B): 36.6% HumanEval+ / 43.9% MBPP+
• DeepSeek-Coder (1.3B): 35.4% / 39.8%
• Qwen2.5-Coder (0.5B): 34.1% / 42.1%

I've open-sourced the model, the dataset, and the GGUF files for anyone who wants to run it locally on their laptop.
Model: https://huggingface.co/Siddh07ETH/Atlas-Coder-2-0.5B
Dataset: https://huggingface.co/datasets/Siddh07ETH/Atlas-Coder-50K-ChatML

Qwen 27B, Gemma 31B and such are already quite good enough, for non coding tasks their performance is quite the same as frontier models, Image generators like nano banana has 55B parameters.

Even for coding tasks these smaller models are good enough if you give detalied prompts and the thing is only the people who give detailed prompts the ones getting actual improved performance.

The bigger models are better at vibe coding but it creates more problems and lead to more time spend debugging thus not really leading to increased productivity, besides the smaller models can do some vibe coding too and they are only going to get better and better.

The thing to think about is are the advantages of the trillion parameter models really worth the costs required to run them, train them, the massive infrastructure required for them is too much will they really be worth it in the end? Can they be worht the billions of dollars of continuous investment they require especially when compared to local models.

My 2016 box ran GLM-4.5-Air (110B, 7x its RAM) streamed from a SATA drive:
pre-registered prediction 0.2-0.3 tok/s, measured 0.19 The same equation (tok/s = eta(tier) x bandwidth / active-bytes) runs Qwen3-30B at 19.3 tok/s on the GTX 1060 and prices any memory upgrade in tok/s before you buy. And the cleanest proof it's placement, not hardware: two Q2_K GGUFs of Gemma-4-12B, same 5.22 GB on disk, differing by 2.25 perplexity purely from which 12 layers got the protected bits, placement is worth roughly 2x the byte budget.

Below: the head-to-head table, the four laws it falls out of, the pre-registered predictions that test them (including a model I predicted within 1% without ever touching the hardware), and quantprobe, the tool that runs the whole loop. Every number measured on one box: i5-7600K (4c/4t), GTX 1060 6GB (Pascal, ~$150 used), 16GB DDR4, Crucial MX500 SATA. Solo project, AI-supported.

1. The head-to-head: same bytes, different layers

Stock llama.cpp `--tensor-type`, Gemma-4-12B, FFN at Q2_K. Same quantizer, same bit budget — the only change is which 12 blocks stay at a higher type:

| Recipe | PPL (WikiText-2) | File |

|---|---|---|

| Uniform Q2_K FFN | 14.41 | — |

| Protect first 12 layers | 12.27 | 5.22 GB |

| Protect last 12 layers | 10.02 | 5.22 GB |

The last two rows are byte-identical. That's the cleanest control I know how to build for a placement effect

Where the method lands against baselines — same box, same eval windows:

| At parity | Baseline | This work | Delta |

|---|---|---|---|

| llama.cpp naive-best placement (Qwen3-30B, same GGUF) | 12.6 tok/s | 19.3 | +53%, zero cost |

| imatrix-calibrated community Q2 (Qwen3-30B) | 11.27 PPL | 11.08 | data-free edges calibrated (+15% size) |

| Calibrated SOTA MxMoE (DeepSeek-V2-Lite 16B, 2-bit) | 1.18x gap | 1.10x (6.31→6.96) | data-free, resident on the 6GB card |

2. Why it works: four laws

The recipe isn't a trick — it falls out of four falsification-tested findings:

1. Rotation is rank-conditional. Incoherence rotation (QuIP#/QTIP/QuaRot) costs +0.006 PPL on a full-rank MLP and +1623 PPL on a low-rank bottleneck — a ~270,000x swing on effective rank alone.

2. Trained networks are dense everywhere. Experts sit at the rate-distortion floor, routing is domain-flat (prose vs code expert sets: Jaccard 1.00), 1-bit collapses (+253 PPL) under every codec. ~2-bit is the data-free floor. No free sparsity.

3. Fragility is measurable, not predictable. Gemma-4-12B late-fragile ~4x, Qwen2.5-7B late ~2-3x, Qwen3-30B-MoE late ~2.3x, Mistral-7B early-fragile ~25x — Qwen's architectural near-twin, fragile in the opposite direction. You must probe, not guess. [attach: charts/x_chart_C_depthcurve.png]

4. The tiered decode law:
   tok/s = eta(tier) x bandwidth / active-bytes-per-token
   , with eta = 0.56 (VRAM), ~0.62 (RAM dense), ~0.38 (RAM MoE), 0.88–1.0 (disk). One equation, 7B to 744B.

What's mine vs. what I build on

Not mine: llama.cpp + k-quants; the QuIP#/QTIP/QuaRot incoherence-codec line; colibri's tier-streaming engine (github.com/JustVugg/colibri) as inspiration — its published tiers land inside my eta bands too.

Mine: the four laws, probe-then-quantize + this tool, the byte-identical placement control, pre-registration as methodology, the depth-aware recipes and placement solver.

Honest limitations

- WikiText-2 PPL is my only quality metric so far — no MMLU/HellaSwag yet.

- Fragility atlas covers four families: enough to disprove universality, not chart the world.

- 0.19 tok/s on the 110B is a capacity demo, not usable inference.

- Single-box research; speeds are single-stream decode, ±25% across environments; eta values are fitted, not derived.

- Machine presets beyond my own box (Mac, newer GPUs) are falsifiable predictions from the law, not measurements. Validated on llama.cpp b9596.

I've been thinking about where AI companies are headed.

My guess is that we'll end up with companies where every employee has hundreds, maybe even thousands, of AI agents working alongside them.

I wrote an essay exploring this idea. Curious where you think this argument falls apart—or if you think we're headed in that direction too.

read the full blog post here

feltspeed.com — pick your GPU / Mac / mini-PC, see which open-weight models fit, and actually watch them stream at the estimated speed (side-by-side race lanes), plus TTFT, cost/breakeven vs an API, and a "what's the cheapest hardware that hits X tok/s" view. Single static page, no signup, no account, cookieless analytics only.

Why I built it. Every VRAM calculator tells you whether a model fits. Almost none tell you honestly how fast it'll feel — and several quote numbers that are physically impossible.

Concrete example: a ~3B-active MoE (think Qwen3-30B-A3B) at 4-bit on a single RTX 4090. You'll see tools confidently print 700+ tok/s. That can't happen. Decode is memory-bandwidth-bound — each token you generate has to read the active weights + KV out of VRAM, and a 4090's ~1 TB/s sets a hard wall. Real single-stream decode tops out around ~230 tok/s on a 4090-class card no matter how tiny the active-parameter count is. The "small active params" of an MoE makes the naive bandwidth / bytes math explode several-fold past what any single stream actually does. Feltspeed clamps it with a single-stream ceiling I had to add after validation (CUDA ~230, Apple ~130, CPU ~40 tok/s); most calculators don't, so their MoE numbers are fiction.

Methodology — please tear it apart:

Decode (bandwidth-bound): tok/s ≈ BW / (active_params×bytes_per_param(quant) + KV_read(ctx)) × η, clamped by the single-stream ceiling per backend.

Prefill / TTFT (compute-bound): from each card's FP16 throughput.

KV cache from the real GQA config (layers × kv_heads × head_dim), not a rule of thumb.

η (the real-world efficiency factor the spec sheet can't give you) is calibrated against community benchmarks across NVIDIA/AMD/Apple/Intel + DGX Spark, per (backend, chip class).

Every output is a RANGE (~±30%) — no false-precision single number.

Data is sourced, not invented: model internals verified against each model's HF config.json; hardware against spec sheets. Anything provisional is flagged in the UI, never silently shipped.

What it deliberately does NOT do: rank model quality. No MMLU/ELO leaderboards baked in. That's opinion and it goes stale fast — this is physics: fit and speed only.

Known limits (upfront):

Laptop GPU tok/s varies with TGP (80–175 W for the same name), so those are ranges, not points.

Hybrid / sliding-window attention (Gemma 3, GPT-OSS, Qwen3.6) is currently modeled as full-attention, so long-context KV is conservative (over-estimates memory) — safe direction, will refine.

Multi-GPU adds capacity, not linear speed (tensor-parallel speedups aren't modeled yet).

The ask: if you have real tok/s numbers on your own hardware, hit "Submit a benchmark" right on the page — every number tightens the η calibration for everyone. And if the methodology is wrong somewhere, tell me exactly where; that's the fastest way this gets better.

Check updated TheStage AI HF collection: https://huggingface.co/collections/TheStageAI/edge-lm

Github repo: https://github.com/TheStageAI/edge-lm

totally new to playing with local AI. thought I'd give it a whirl running on an old system. 5600x. 16gb ram and a 1070 8gb.

so I've been messing around with like 7b size models to try making bash scripts for fun just to see what I can do. I'm not a programmer at all. so those sized models were definitely struggling with just normal, non coder prompts. and based on my quick research I needed to find models that fit on my 8gb card.

then I saw a post of a guy running these giant models on an rx470 8gb. did some reading. eventually asked chatgpt how to do this. and it suggested https://huggingface.co/unsloth/gpt-oss-20b-GGUF.

I'm not really sure what's happening. I think instead of always using the entire model to "think" it just uses the parts it needs?

anyways it's significantly smarter. it wrote the script I wanted in about 5 prompts. it got 90% there on the first try. the rest of it was just minor stuff. I had to increase the ctx size to 32000 and the prediction to 12000. but it fits on the GPU. 7100mb of 8200. I get about 24 token per second. which works great for my use.

also tried Gemma-4-26B-A4B. Which barely fits, but it's pretty smart too. But it spent so much time thinking and planning I think it ran out of tokens before even getting to write the script. But I can't really increase the ctx size cause it barely fits on 8gb of vram. Still neat tho.

Now I'm wondering if I should find another cheap 10 series card with another 8gb of ram and maybe I can fit one of these types of models but a 40b. 🤷‍♂️🤷‍♂️🤔 Lol

anyways I had a cool time nerding out about that, wanted to share. haha

Edit: gpt-oss-20b-MXFP4.gguf is the full filename

Bonsai 27B dropped last week, and I've had it running entirely client-side in our PWA since yesterday WebGPU only fully offline after a one-time download. Video attached, you can try it in the last link.

Try it (desktop with WebGPU(6GB+ Vram), smaller tiers otherwise): https://mentria.ai/tools/ai-chat/

Site + integration code: https://github.com/mentria-ai/website, a star helps if you find it useful.

Comments and improvements very welcome. All kernels are custom built for the inference engine. Open sourcing engine code soon.

Open AI has now ChatGPT app for Windows and it can do a lot, but if I want to install one locally for local LLM, what are my options

Most of my local model time is chat and code.

Lately I wired up something different: a local agent (Ollama or MLX) that takes a plain-English request and builds a file-automation pipeline. It picks from 161 built-in steps and writes custom Python when those don't fit - every line it writes is inspectable, sandboxed, and runs locally.

The tool-calling quality decides everything. Qwen3 and Llama 3.1 70B hold a multi-step job together. Smaller models lose the thread by step three. Which local models give you dependable tool calls across several steps? I am keen to learn if it's possible to get more out of 20-30b models.

Also, for transparency - this is self-promo. I need to work out such nitty gritty before presenting the product.

Thanks!

Protorikis is back with a demo for his free AI testing GUI.
Note, he tested on Mac, so other hardware may have different trade-offs.

* good for 'predictable' prompts
* not so good for harder prompts & large context
* good for compute bound workloads
* bad for memory-bandwidth bound work
* he found on his Mac MOE models work better in general than dense, & MOE helped more from MTP on his computer.

Y’all, my laptop is 100% bootyhole, that being said, what is the best model to run locally? The ram is soldered (why do they even do that in the first place 🤔) so upgrading it is not an option sadly. Here are the specs:

CPU: 13th Gen Intel Core i5-1335U RAM: 8 GB LPDDR5-6400 Graphics: Intel Iris Xe integrated graphics, sharing system RAM Storage: 512 GB WD PC SN740 NVMe SSD Operating system: Windows 11 Dedicated GPU: None

Thank you all!

I have been building a local-first agentic stack that targets consumer GPUs, the kind with 24 GB of VRAM or less, and I got it to run end to end on a single RTX 3070 Ti Laptop (8 GB). Everything runs on your own machine. No cloud, no accounts, no telemetry. I wanted to share it and get feedback before I tag a release. The main workhorse are the new bonsai models from prism-ml (BTW yes this is my only real post, i was only a reddit lurker until now)

It is four small, independent, Apache-2.0 repos:

- Suiban: the inference and orchestration core. Python, FastAPI, uv. It manages the llama-server subprocesses, plans a VRAM-aware loadout, runs the agentic loop and keeps memory and skills. https://github.com/YKesX/suiban

- dai: a desktop app (Tauri, React, TypeScript) for chat, agentic coding, deep research and vision. https://github.com/YKesX/dai

- sentei: a coding-focused terminal client that can also install itself as a background service. https://github.com/YKesX/sentei

- SLAP: the Structured Lightweight Agent Protocol, a small versioned schema-validated format the orchestrator uses to talk to worker sub-agents. https://github.com/YKesX/SLAP (I know this is not a real protocol like old protocols but i have some ideas that will turn this into something better in the future!!)

general website: https://ykesx.github.io/dai/

The clients never import each other. They talk to suiban over plain HTTP on `127.0.0.1:8686`, against one frozen contract. That was the whole point: keep the pieces swappable.

The parts I think are actually interesting

A 4-bit TurboQuant KV cache. The V-cache is stored in new GGML types (a 4-bit default and a 3-bit aggressive preset, ported from an MIT-licensed reference and vendored into the fork). K stays at q8_0. On the 8 GB laptop, perplexity stayed inside the q8_0 baseline and needle-in-a-haystack retrieval kept passing, so the memory saving did not cost accuracy in my tests. There is a fast-path decode kernel (warp-shuffle) that measured about 3x on a 16K-depth microbench. This is TurboQuant from arXiv:2504.19874, discussed in llama.cpp #20969. Numbers are from one laptop, not a sweep, so take them as a data point.

sentei /resume-claude Sentei can import claude code sessions for going on where you left off with your claude code session.

Lazy keep-alive, like ollama. The server starts holding no VRAM. Models load on the first request and release after an idle timeout. Cold start sits around 780 MiB with no model resident. That means you can leave it running all day and it costs nothing until you actually call it.

Ternary and 1-bit models. It runs the PrismML Bonsai family (27B orchestrator plus 8B, 4B and 1.7B workers) as ternary GGUF by default, with a 1-bit family toggle. Weights are downloaded at install with pinned SHA-256 digests, nothing model-shaped ships in the repos.

Multi-agent that cleans up after itself. Heavier tasks fan out to contained sub-agents coordinated over SLAP. The orchestrator writes each worker a system prompt that is volatile: it is generated for that one job and discarded, and it never shows up in the trace.

A security model I took seriously. Loopback bind is open with no auth for zero friction on your own box, but the moment you expose it to a network it requires a bearer token. Web pages, file contents and skill bodies all enter the model fenced as untrusted data, not instructions, so a hostile page cannot steer a shell command. I ran an adversarial pass on it and wrote the findings up in an audit doc in the repo.

Memory and skills without a vector DB. Recall is SQLite FTS5, no embeddings. Skills are agentskills.io-compatible markdown, and it can import skills from openclaw or Hermes. It also has MCP connector support.

What is measured and what is not

Measured on one RTX 3070 Ti Laptop, 8 GB: cold start and warm-on-demand inference, the TurboQuant accuracy checks above and a 200-turn soak plus repeated multi-agent rounds where VRAM stayed flat and the process count held steady, so no leak and no zombie servers. Test suites are green across the four repos.

Honest limits, because this is not a 1.0 yet: everything above is one machine and one GPU tier. Windows and macOS installs are coded and name-checked but I have not run them start to finish. The WhatsApp gateway renders a real QR for device linking but the live send path is not wired yet. Each repo ships a KNOWN_ISSUES file that says exactly what is and is not validated.

Install

Two commands to install, one to run. dai and sentei can install suiban alongside themselves or point at a suiban running on another box.

Repos:

- https://github.com/YKesX/suiban

- https://github.com/YKesX/dai

- https://github.com/YKesX/sentei

- https://github.com/YKesX/SLAP

Everything is Apache-2.0. Happy to have contributors, and much more benchmarks on more hardware types. Feedback and teardowns welcome. MLX support will come in the later days.

I see a lot of questions here about what model and what hardware, so I thought I would offer my experience, failures and wins. This is a tool-calling workload, not chat and not coding. Every number below is from my own runs, and all of them are in this post.

The workload

A read-only home and farm monitoring agent. It answers questions over signal chat by querying a local resources prometheus, elasticsearch, and homegrown apis and lorawan metrics. It has to pick the right tool, build the query, read the result, report the number, and know when not to call a tool. The agent process runs on a Raspberry Pi and talks to the GPU box over the LAN through an OpenAI-compatible /v1 endpoint. Nothing leaves the house.

That is almost pure tool calling. Not long-form writing, not code generation, not RAG. So the benchmark I ran is a tool-calling benchmark. If your workload is different, my ranking tells you very little, and the methodology is the part worth copying.

Hardware, model, quant

• One RTX 5090, 32GB VRAM, consumer desktop, Windows.
• llama.cpp, prebuilt win-cuda binary, build b10075. Not built from source (failure 4 below).
• unsloth/GLM-4.7-Flash-GGUF, file GLM-4.7-Flash-Q4_K_M.gguf, 18.3 GB on disk, which is 17 GiB. Base weights zai-org/GLM-4.7-Flash. MIT licensed, both.
• ~31B total params, ~3B active per token (MoE). It is a reasoning model, it thinks before answering.
• ~198 tok/s generation.

Why Q4_K_M specifically: the weights take about 17 GiB on a 32 GB card, which leaves roughly 13 GiB for KV cache and overhead. Going up a quant spends part of that context budget, which for an agent workload is a trade I did not want to make. I did not benchmark other quants, so treat that as a deliberate choice rather than a measured result.

llama-server --model GLM-4.7-Flash-Q4_K_M.gguf --host 0.0.0.0 --port 8000 \
  -ngl 999 --jinja --ctx-size 32768 --metrics --api-key YOUR_KEY

Flag notes:

• --jinja is mandatory for tool calling. Without it the model's tool calls come back as raw text that nothing parses. No error, no exception, no failed call. The agent just silently does nothing. This is the flag people miss.
• -ngl 999 puts every layer on the GPU.
• --ctx-size 32768. The model supports about 200k, but context costs VRAM, see the quant note above. 32k is comfortable for an agent carrying a handful of tool schemas.
• --metrics exposes a Prometheus endpoint on the same port. It sits behind --api-key, so your scraper needs the bearer token or it silently 401s and you get a dead target with no obvious cause.

Sampling is temperature 0.2, near greedy. I benchmarked that against the vendor's recommended tool-calling sampling (higher temp plus top-p / min-p / repeat-penalty): 83 vs 84, z = 0.43, p = 0.67. No difference I can detect at this sample size, which is not the same as showing the two settings are equivalent. I stayed near greedy because structured output benefits from determinism.

Methodology

Real Berkeley Function-Calling Leaderboard v3 dataset, graded with BFCL's own AST methodology: function name match, every parameter value inside its acceptable-value list, no hallucinated parameters. The irrelevance category passes only when the model correctly makes no call at all.

100 cases per category, 5 categories (simple, multiple, parallel, irrelevance, live_simple) = 500 cases per model. Temperature 0.2. All three models at the same Q4_K_M quant, on the same server, same harness.

That last part is the bit I would repeat anywhere. Benchmark the quant you will actually deploy. Q4_K_M numbers from someone else's FP16 run are not your numbers. Same quant, same server, same flags, same harness across every candidate.

Results

GLM per-category: simple 89, multiple 80, parallel 83, irrelevance 88, live-simple 82. Qwen3-Coder on live-simple: 68.

The statistics

Two-proportion z-tests on those gaps:

• GLM 84 vs Qwen3-32B 82: z = 0.84, p = 0.40. Within noise.
• GLM 84 vs Qwen3-Coder 80: z = 1.65, p = 0.10. Within noise, and the closest of the three to significance.
• Qwen3-32B 82 vs Qwen3-Coder 80: z = 0.81, p = 0.42. Within noise.

None of those is a demonstration that the models are equal. They are failures to separate the models at this sample size, which is a weaker statement.

95% CI on GLM's 84% overall is +/- 3.2pp, so 80.8 to 87.2. That interval overlaps all three models. A single 100-case category at 84% carries a 95% CI of +/- 7.2pp, which is wide enough that per-category rankings should be read as suggestive at best, and wide enough that I would not read much into any one row of the table above.

At n = 500 these three models are statistically indistinguishable on overall accuracy. I cannot honestly claim GLM is the most accurate model here. It won the point estimate. That is not the same thing.

Two things in the data are real:

1. live-simple, GLM 82 vs Qwen3-Coder 68: z = 2.29, p = 0.022. live-simple uses real messy human phrasing, which is exactly what a chat-facing agent gets all day. A coding model doing conversational ops work shows its seams there. Caveat: this is one contrast among several across five categories, so under a strict multiple-comparisons correction it would not clear on its own. live-simple is the category I care about most for this workload, for the reason in the workload section, but I would still call the result provisional until someone replicates it at larger n.

2. Speed. The dense 32B took 187 to 283 s per 100-case category against GLM's 45 to 82 s, roughly 3 to 4x. The coder model at 22 to 40 s is faster still. Not a statistical question, just a large repeatable difference. The dense model activates all its parameters per token, the MoE models activate a fraction. The extra compute bought no measurable accuracy on this benchmark.

So the practical read: for this workload, pick on speed, license, VRAM fit, and behavior under messy phrasing, because accuracy did not separate them.

What I got wrong along the way

1. My homegrown harness inverted the ranking. Before BFCL I wrote an 8-case tool-calling smoke test around my own tools. It scored GLM at 62 to 66% across repeated runs against the incumbent coder model at 79%. Read literally, that says reject GLM. On real BFCL data the same model scored 84%. My harness understated it by roughly 20 points because 8 cases is far too few to separate anything, and because two of those cases were testing my prompt wording rather than model capability.

Lesson: a homegrown smoke test is fine for catching regressions in your own stack. It should never carry a keep-or-kill decision on a model.

2. The model I replaced guessed instead of erroring. The coder model would return confident, plausible numbers having made zero tool calls. It reported heat pump water temps about 30 degrees off. It once invented a water tank level that sat below my low-water alarm threshold and appeared in no query anywhere. Challenged, it defended the number instead of re-running the query. For a monitoring agent that is the worst failure mode. A failed tool call is loud and harmless. A fabricated reading is silent and looks exactly like a real one.

Lesson: pick the metric that matches the failure you fear, which is why irrelevance, the category that asks whether the model correctly makes no call, is the number I read first.

3. Two days lost to a misdiagnosis. vLLM would not serve this model. Healthy LISTEN socket on 0.0.0.0:8000 that never accepted a connection, every request timing out, including a raw TCP connect to 127.0.0.1. I blamed WSL networking for two days. The test that settled it took one minute: python3 -m http.server 8099 on the same box answered instantly on both loopback and LAN while the real server's socket sat dead. Networking was never involved.

Lesson: run the known-good control first, before you form a theory. I still have not root-caused the vLLM accept-hang. I routed around it to llama.cpp.

4. llama.cpp would not build under WSL. CUDA 13 against glibc 2.41 threw an exception-spec error on rsqrt in mathcalls.h. Not fixable by swapping g++ versions. The prebuilt Windows CUDA binary just ran.

Lesson: when a build fights you, a prebuilt binary that only has to run is immune to that entire class of problem.

5. One bug was my prompt, not the model. Every model I tested, GLM included, mislabels Celsius as Fahrenheit when the prompt does not pre-convert. Metrics ending in _celsius came back reported as F. Fixed by baking the conversion into the queries the prompt hands the model, plus a hard rule in the system prompt.

Lesson: check your own prompt before you blame a model.

Where I landed

GLM-4.7-Flash Q4_K_M on llama.cpp with --jinja, 32k context, temp 0.2 is what runs the agent now. I picked it over the statistically tied dense 32B on speed, and over the coder model on live-simple plus its habit of answering without calling anything.

The one piece of this I would hand to anyone benchmarking their own candidates: know what your n buys you, and compute the interval before you write the conclusion. 500 cases per model got me +/- 3.2pp, nowhere near enough to rank models 2 points apart. And for anything that reports real-world numbers, weight refusal-to-guess above raw accuracy.

Happy to answer questions on the harness, the flags, or the grading, and if you want to poke holes in the statistics, please do. The obvious next step is more cases per category to tighten those intervals. If anyone has run the same categories at larger n, or has actually root-caused a vLLM accept-hang like number 3, I want to hear it.

Full writeup with the setup section, if you want the longer version: https://heretik.io/glm-4-7-flash-homelab-agent/

I was using Gemini code plugin for some time, and this week its gone. After much investigation, trial and error with my ollama setup ive been using for a long time i decided to give LM Bionic a shot.

WOW....

So right now i keep vscode running and see that the changes are implemented before compilation, the Bionic app is off to the right where i do my work.

Ive actually had success, installing bionic into continue.dev but i like either or.

After trying a multitude of LLMs Qwen3.6 27B MLX was the ticket running about 18Kcontext as i have my memory pressure monitor open as i adjust

The Thinking that I watch can go on for a few minutes but the quality of work mopped the floor with gemma 4 26B a4b. I might give it another chance but not right now.

My laptop is running it too with lm link back to the m4..... can work remotely with the laptop off the m4 as needed

Hosted web search from Anthropic and OpenAI costs $10 per 1k searches, Gemini costs $14, and then you pay again for the ~17k tokens of results each search dumps into context. I got annoyed enough to build an alternative.

It’s called webfetch. Runs locally, free out of the box (DuckDuckGo needs no API key), and in my SimpleQA benchmark the same agent loop hits the same accuracy as hosted search (96%) costing 66% less using 87% fewer tokens.

How it works:
1. RRF fusion across 4 search engines, local page fetching, hybrid BM25 + bi-encoder retrieval with a cross-encoder reranker

1. Sentence-level compression that cut result tokens in half with no measured recall loss

2. Semantic caching: paraphrased queries (“what did TypeScript 5.9 add” vs “TypeScript 5.9 new features”) get matched by embeddings and verified by an NLI cross-encoder, so reworded repeats cost nothing. Cache TTLs adapt to how volatile the answer may be

3. Every cached result shows provenance and the model can force a fresh search if it doesn’t trust it

4. Benchmarked against Anthropic hosted search, OpenAI, Tavily and Exa.

One small agent loop that I ran for testing that conducted just 16 websearches (opus 4.8) already reported 1.5 USD in savings.

Install from PyPI using pip.

Repo: https://github.com/firish/webfetch

Hey everyone,

Like many developers building agentic workflows, I spent months getting frustrated by traditional vector stores and RAG memory layers failing over long timelines.

The deeper I went, the more I realized retrieval fails because basic similarity doesn't equal utility. A standard retriever will match a user's query about mattress brands to previous mattress conversations, while completely missing a crucial constraint buried in a 3-month-old session: "Whenever I buy something expensive, warranty is the only thing I care about."

Beyond that, heavy cross-encoder rerankers quickly become a massive latency bottleneck as memory grows, and treating all context as uniform text blobs destroys the nuance of evolving decisions.

To tackle this, I built MindCache—an open-source agentic memory framework designed around four key insights:

• Intelligence Belongs at Ingestion: Instead of attempting complex graph traversals during a live query, MindCache shifts expensive reasoning (relationship mapping, graph clustering, and summary generation) to ingestion. This cut retrieval latency from ~25s down to 1.08s (a 23× speedup) without sacrificing context quality.
• Specialized Memory Typologies: Not all memories behave the same. MindCache separates knowledge into User (persistent behavioral constraints), Knowledge (domain facts), Episodic (chronological logs), and Decision Memories (which track evolving proposals, trade-offs, and final conclusions over time).
• Living Knowledge Hierarchy: Rather than maintaining a static or unmanageable graph, MindCache uses Leiden community detection to partition memory into localized semantic clusters, ensuring graph maintenance scales efficiently as context accumulates.
• Evidence Assembly over Similarity: Retrieval doesn't just search for similar text—it plans and assembles the exact minimal subset of evidence (user preferences, hierarchical summaries, decision states) required for the LLM to reason correctly.

On the BEAM benchmark (an ICLR 2026 evaluation framework designed specifically for long-term agentic memory), MindCache outperformed Mem0 in handling evolving context, contradiction resolution, and cross-session summary reasoning. More importantly, it achieved this superiority not by stuffing larger retrieval windows, but through better ingestion-time knowledge organization.

I wrote a deep-dive 23-minute engineering post-mortem detailing all 5 failure modes, the full architecture, and benchmark takeaways. The project is completely open-source on GitHub and available on PyPI (pip install mindcache-ai).

I’d love to hear how others here are handling temporal decay, graph maintenance, and decision tracking in your long-running agent setups!

so i kept running into the same wall with local agents — they could write code fine, but ask them to generate a proper .docx or .xlsx and suddenly it's a mess of broken python dependencies or headless libreoffice subprocesses that eat vram for breakfast. found officecli this week and it's been a breath of fresh air for my ollama workflow.

three things that actually impressed me:

first, the xpath-style addressing is huge for local llms. instead of making my agent write 50 lines of openpyxl to find the right cell, i can just say row[Salary>5000] and it works. the llm gets it on the first try.

second, the built-in renderer lets the agent actually see what it built. my local setup runs a live preview on port 26315, so the agent can screenshot and fix layout issues without round-tripping through pdf conversion. this closed a loop i didn't realize was broken.

third, the single binary is legit — no python env, no java, no npm. my 8gb vram rig doesn't even notice it's running.

one gotcha: the c# binary is fine but if you're on a really old linux distro you might hit glibc version issues. i had to grab the musl build. also the skill auto-install only covers claude code/cursor/copilot — if you're running raw llama.cpp or a custom agent setup you'll need to wire up the tool call schema yourself.

full writeup here if you want more detail: https://andrew.ooo/posts/officecli-office-suite-for-ai-agents-review/

anyone else found tools that let local agents handle office files cleanly? curious what people use for the render-then-fix loop when running models locally.

Qwen3-30B-A3B on an Orange Pi 6 Plus — real-world performance

After a fairly large number of tests and configuration experiments on ARM SBC hardware, I managed to run Qwen3-30B-A3B Q4_K_M on an Orange Pi 6 Plus with 32 GB of RAM.

The model is running through a local OpenAI-compatible server with:

• 32 GB RAM
• 8 CPU threads
• 4096-token context
• reasoning disabled
• CPU inference
• no discrete GPU

For a practical test, rather than a short synthetic benchmark, I used a 208-token prompt requesting a structured business and software architecture analysis.

Results:

• Prompt processing: 32.21 tokens/s
• Text generation: 15.62 tokens/s
• Generated tokens: 512
• Generation time: 32.79 seconds
• Total request time: 39.30 seconds
• Memory usage for the server process: approximately 23.6 GB

The response reached the configured 512-token limit, so the generation measurement was long enough to be reasonably stable.

I was honestly surprised by the result. Around 15.6 tokens/s from a 30B-class MoE model on a single ARM SBC makes it usable not only for experimentation, but potentially for a local assistant, RAG, home automation or business workflows.

I performed several tests before reaching this configuration, but for now I am keeping the exact build, compilation options and low-level optimizations private.

Has anyone else tested this model on the Orange Pi 6 Plus or on similar ARMv9 SBC hardware?

Step 3.7 Flash reached #8 on OpenRouter by tokens/day. #2 model on Hermes and #1 on Kilocode yesterday