Paying $20 a month for Claude Pro is objectively better for text and coding because local LLMs are nowhere near as capable. The only scenario where running models locally actually makes sense is for creative media generation. Is there any model you would recommend running locally?

Qwen3.6 really is nice, I am currently running the qwen3.6:35b version on my local GPU. I have now tried to jailbreak a bit, such that I can use it more freely and make it sound more honest and direct, but none of my attempts have worked so far. Anybody know, how to disable the anti-jailbreaking measures?

I am trying to solve a pretty (in my head) simple use case. Intake a list of companies, proceed to make search queries about these companies (news, announcements, results) for articles posted within the past 7 days and dump title, snippet, url, etc into a file for later processing. Silly me, apparently search is really really hard even in 2026.

So far I've tried: Exa, Tavily, Serper, Serpbase, Firecrawl, SearXNG and some others and none seem to produce anything even remotely acceptable.

1. This is a big one, vast majority of search backends either outright do not support "freshness" or produce bad to non-existent results when you try to employ it. Meanwhile I can go to Google, make the same exact query and get the desired results.
2. With Google I can enter "COMPANYNAME news announcements results" as a single query and get decent results. With various search backends, I seem pigeonholed into making 3 separate queries to get anything even remotely reasonable. Is this a deliberate tactic to get people to burn through their API credits?
3. Results are often cached?

With self-hosted models, I feel like I went 2 years back in time and this is acceptable to me. With search, however, I feel as if the jump is 30 years back, something of the Altavista age. How is any of this acceptable? How are people PAYING MONEY for this quality? What are the big boys using for their searches, Google deals behind closed doors (Google no longer offers search API directly)?

What are you using and how did you have to wrangle with it to get acceptable behavior of it?

My specs:
Windows 11 laptop with
rtx 4050 6gb vram. 64 gb ram. 4tb ssd with dram. Ryzen 5 CPU(6 cores)

I want to be able to use hermes with it or do coding for light scripts.

Hello, guys.

I want to upgrade my PC mainly for local agentic coding, not gaming.

Current setup:

• Ryzen 7 5700X
• Gigabyte B450 AORUS Pro
• 32GB DDR4
• GTX 1660 Super 6GB
• Corsair CV650 650W Bronze
• Windows 11 LTSC + CachyOS
• llama.cpp / ik_llama.cpp with Pi Agent

I currently pay around $100/month for Claude and still hit limits.
I do not expect local models to replace frontier models, but I would like to move repetitive, private and token-heavy coding tasks locally.

OPTIONS

RTX 3060 12GB used: 180€-230€

Cheap, CUDA-compatible and works with my PSU. But is 12GB enough for decent models, or only small models with very limited context?

RX 9060 XT 16GB new: 430€-500€

Good VRAM per euro and efficient, but I am concerned about ROCm, ik_llama.cpp compatibility and mixing AMD with my current Nvidia card. Still not sure about the LLMs that could fit here.

RTX 5060 Ti 16GB new: 465€-600€

Probably the easiest and most efficient option, but it feels expensive for only 16GB.

RTX 3090 24GB used: 750€-1000€

Much better for 27B-35B models, but I would also need a new 750-850W PSU. Total cost would be around €700-850, and good used units are difficult to find.

However, I would also need a better 750-850W PSU, making the realistic total cost approximately €850-1,150.

Good units are difficult to find, and I am also concerned about power consumption, heat, card condition and whether it fits my case.

MODELS

Models I am considering include Qwen 3.6, Gemma 4, Bonsai and other coding-focused GGUF models.

My motherboard has a second PCIe 2.0 x4 slot, but I do not think it is a sensible base for dual GPUs.

Questions:

• Is an RTX 3060 12GB actually useful for coding agents?
• Is 16GB a meaningful upgrade or still too limiting?
• At what total price does a used RTX 3090 stop being good value?
• Has local inference actually reduced your Claude or API spending?
• Which option would you choose at these prices?

Thank you so much for your help.

This morning I posted a local tool-calling benchmark and someone pointed out I'd skipped Qwen3.6-27B. Fair hit, lets see how it performs and why its important to look at the bigger picture and actual use cases.

Full writeup: https://heretik.io/qwen36-vs-glm-flash-rerun/

Qwen3-30B-A3B on an Orange Pi 6 Plus — real-world performance

After a fairly large number of tests and configuration experiments on ARM SBC hardware, I managed to run Qwen3-30B-A3B Q4_K_M on an Orange Pi 6 Plus with 32 GB of RAM.

The model is running through a local OpenAI-compatible server with:

• 32 GB RAM
• 8 CPU threads
• 4096-token context
• reasoning disabled
• CPU inference
• no discrete GPU

For a practical test, rather than a short synthetic benchmark, I used a 208-token prompt requesting a structured business and software architecture analysis.

Results:

• Prompt processing: 32.21 tokens/s
• Text generation: 15.62 tokens/s
• Generated tokens: 512
• Generation time: 32.79 seconds
• Total request time: 39.30 seconds
• Memory usage for the server process: approximately 23.6 GB

The response reached the configured 512-token limit, so the generation measurement was long enough to be reasonably stable.

I was honestly surprised by the result. Around 15.6 tokens/s from a 30B-class MoE model on a single ARM SBC makes it usable not only for experimentation, but potentially for a local assistant, RAG, home automation or business workflows.

I performed several tests before reaching this configuration, but for now I am keeping the exact build, compilation options and low-level optimizations private.

Has anyone else tested this model on the Orange Pi 6 Plus or on similar ARMv9 SBC hardware?

Protorikis is back with a demo for his free AI testing GUI.
Note, he tested on Mac, so other hardware may have different trade-offs.

* good for 'predictable' prompts
* not so good for harder prompts & large context
* good for compute bound workloads
* bad for memory-bandwidth bound work
* he found on his Mac MOE models work better in general than dense, & MOE helped more from MTP on his computer.

What hardware would you use for running local LLM?

I have been using Claude Max 5 to build a personal project. As it’s growing, i may require a machine to run the system multiple times in the background.

I was wondering to use desktop/Mac mini. Also, thought is Claude code subscription price may increase and if I am buying hardware might as well see if i can do a local LLM setup.

Hardware would be used by sister for content creation so anyway plan was to get a graphic card. But then costs have been soaring and we normally don’t buy these things every year.

What would you suggest? What would hardware look like? Which model can work on it?

My assumption is custom desktop would be more powerful than Mac mini at same price.

I'm looking to get into local LLMs and start running my business using agents. I have the opportunity to buy a lightly used Macbook Pro 16" with m4 max and 128gb of RAM, 8TB hard drive for about $5K locally. Would this be a solid set up and a good deal? Looks like the equivalent M5 set up is selling for about $10k

Hi everyone!

Just wanted to know if anyone here is from NYC - because I'm hosting an event next week for Local AI. We will be talking about training / inference / models / and leaders in the space.

A lot of CEO/CTO's in the space are signing up for this in Luma already. Would love to have anyone interested in local to come to this event. If it goes well, I'm going to try my best to make this happen bi-weekly.

If you have any questions please let me know!

I recently asked Claude Sonnet and ChatGPT if I could run Unsloth Gemma 4 12B (QAT Q4) on the MacBook Air M5 (16GB Unified Memory) I'm getting my laptop delivered to me this week.

I'm not expecting this machine to run local models perfectly at all. Having a chat with a local LLM and some roleplay through a terminal would be great.

When I had asked both models if I could that model onto that device, they gave me different answers about the weights, runtime footprint, and KV Cache. Both of them kept giving out different answers about the KV Cache footprint upon every generation and I wasn't sure to trust their word on that.

Can anyone tell me how much would 32K tokens would be in Q8 KV Cache for the Gemma 4 12B QAT Q4 model? Or a rough estimate? I'm just curious.

(By the way, if anyone wants to recommend me try to any different models for this laptop, feel free to recommend if any.)

My 2016 box ran GLM-4.5-Air (110B, 7x its RAM) streamed from a SATA drive:
pre-registered prediction 0.2-0.3 tok/s, measured 0.19 The same equation (tok/s = eta(tier) x bandwidth / active-bytes) runs Qwen3-30B at 19.3 tok/s on the GTX 1060 and prices any memory upgrade in tok/s before you buy. And the cleanest proof it's placement, not hardware: two Q2_K GGUFs of Gemma-4-12B, same 5.22 GB on disk, differing by 2.25 perplexity purely from which 12 layers got the protected bits, placement is worth roughly 2x the byte budget.

Below: the head-to-head table, the four laws it falls out of, the pre-registered predictions that test them (including a model I predicted within 1% without ever touching the hardware), and quantprobe, the tool that runs the whole loop. Every number measured on one box: i5-7600K (4c/4t), GTX 1060 6GB (Pascal, ~$150 used), 16GB DDR4, Crucial MX500 SATA. Solo project, AI-supported.

1. The head-to-head: same bytes, different layers

Stock llama.cpp `--tensor-type`, Gemma-4-12B, FFN at Q2_K. Same quantizer, same bit budget — the only change is which 12 blocks stay at a higher type:

| Recipe | PPL (WikiText-2) | File |

|---|---|---|

| Uniform Q2_K FFN | 14.41 | — |

| Protect first 12 layers | 12.27 | 5.22 GB |

| Protect last 12 layers | 10.02 | 5.22 GB |

The last two rows are byte-identical. That's the cleanest control I know how to build for a placement effect

Where the method lands against baselines — same box, same eval windows:

| At parity | Baseline | This work | Delta |

|---|---|---|---|

| llama.cpp naive-best placement (Qwen3-30B, same GGUF) | 12.6 tok/s | 19.3 | +53%, zero cost |

| imatrix-calibrated community Q2 (Qwen3-30B) | 11.27 PPL | 11.08 | data-free edges calibrated (+15% size) |

| Calibrated SOTA MxMoE (DeepSeek-V2-Lite 16B, 2-bit) | 1.18x gap | 1.10x (6.31→6.96) | data-free, resident on the 6GB card |

2. Why it works: four laws

The recipe isn't a trick — it falls out of four falsification-tested findings:

1. Rotation is rank-conditional. Incoherence rotation (QuIP#/QTIP/QuaRot) costs +0.006 PPL on a full-rank MLP and +1623 PPL on a low-rank bottleneck — a ~270,000x swing on effective rank alone.

2. Trained networks are dense everywhere. Experts sit at the rate-distortion floor, routing is domain-flat (prose vs code expert sets: Jaccard 1.00), 1-bit collapses (+253 PPL) under every codec. ~2-bit is the data-free floor. No free sparsity.

3. Fragility is measurable, not predictable. Gemma-4-12B late-fragile ~4x, Qwen2.5-7B late ~2-3x, Qwen3-30B-MoE late ~2.3x, Mistral-7B early-fragile ~25x — Qwen's architectural near-twin, fragile in the opposite direction. You must probe, not guess. [attach: charts/x_chart_C_depthcurve.png]

4. The tiered decode law:
   tok/s = eta(tier) x bandwidth / active-bytes-per-token
   , with eta = 0.56 (VRAM), ~0.62 (RAM dense), ~0.38 (RAM MoE), 0.88–1.0 (disk). One equation, 7B to 744B.

What's mine vs. what I build on

Not mine: llama.cpp + k-quants; the QuIP#/QTIP/QuaRot incoherence-codec line; colibri's tier-streaming engine (github.com/JustVugg/colibri) as inspiration — its published tiers land inside my eta bands too.

Mine: the four laws, probe-then-quantize + this tool, the byte-identical placement control, pre-registration as methodology, the depth-aware recipes and placement solver.

Honest limitations

- WikiText-2 PPL is my only quality metric so far — no MMLU/HellaSwag yet.

- Fragility atlas covers four families: enough to disprove universality, not chart the world.

- 0.19 tok/s on the 110B is a capacity demo, not usable inference.

- Single-box research; speeds are single-stream decode, ±25% across environments; eta values are fitted, not derived.

- Machine presets beyond my own box (Mac, newer GPUs) are falsifiable predictions from the law, not measurements. Validated on llama.cpp b9596.

My Mac mini is M4 pro 24GB Ram 512 ssd

I am thinking to run Claude code in remote control mode just for the sake of editing or querying my Obsidian notes through my phone when I’m out of home.

Since I assume such tasks are relatively simple, I thought that I can save quota of my cc subscription I use for coding and leverage local model solely for this purpose. Which model would fit my Mac mini and provide good experience and quality for this task?

100b parameter local run

I want buy

Laptop VS pc build VS nvidia dgx spark (mac mini m4 other)

which is best option for me

Suggest me

Hosted web search from Anthropic and OpenAI costs $10 per 1k searches, Gemini costs $14, and then you pay again for the ~17k tokens of results each search dumps into context. I got annoyed enough to build an alternative.

It’s called webfetch. Runs locally, free out of the box (DuckDuckGo needs no API key), and in my SimpleQA benchmark the same agent loop hits the same accuracy as hosted search (96%) costing 66% less using 87% fewer tokens.

How it works:
1. RRF fusion across 4 search engines, local page fetching, hybrid BM25 + bi-encoder retrieval with a cross-encoder reranker

1. Sentence-level compression that cut result tokens in half with no measured recall loss

2. Semantic caching: paraphrased queries (“what did TypeScript 5.9 add” vs “TypeScript 5.9 new features”) get matched by embeddings and verified by an NLI cross-encoder, so reworded repeats cost nothing. Cache TTLs adapt to how volatile the answer may be

3. Every cached result shows provenance and the model can force a fresh search if it doesn’t trust it

4. Benchmarked against Anthropic hosted search, OpenAI, Tavily and Exa.

One small agent loop that I ran for testing that conducted just 16 websearches (opus 4.8) already reported 1.5 USD in savings.

Install from PyPI using pip.

Repo: https://github.com/firish/webfetch

Hey everyone,

I’ve been testing out the new DSpark (DFlash) speculative decoding integration in llama.cpp using an Intel Arc Pro GPU running on the SYCL backend.

Comparing it against MTP (Multi-Token Prediction), I’m not seeing much of a speedup overall. Additionally, I ran into an issue where spec-draft-n-max cannot be set higher than 7 without crashing/failing.

Here are my configs and benchmark results for comparison.

Shared Global Settings

Ini, TOML

jinja = true
flash-attn = on
no-mmap = true
mlock = true
b = 2048
ub = 4096
cache-reuse = 256
cache-type-k = q4_0
cache-type-v = q4_0

1. Multi-Token Prediction (MTP)

Config:

Ini, TOML

model = /home/james/models/Qwen3.6-27B-MTP-Q4_K_M-New.gguf
ctx-size = 256000
reasoning = on
spec-ngram-simple-size-n = 4
spec-ngram-simple-size-m = 4
spec-type = draft-mtp
spec-draft-n-max = 3

Results:

2. DSpark (DFlash Speculative Decoding)

Config:

Ini, TOML

[qwen3.6-27b-coding-Dspark]
model = /home/james/models/Qwen3.6-27B-MTP-Q4_K_M-New.gguf
mmproj = /home/james/models/Qwen3.6-27B-MTP-Q4_K_M-New-mmproj-F16.gguf
ctx-size = 256000

# --- DFlash Speculative Decoding ---
np = 1
spec-type = draft-dflash
spec-draft-model = /home/james/models/Qwen3.6-27B-Dflash/Qwen3.6-27B-DFlash-Q8_0.gguf
spec-draft-n-max = 7
spec-draft-p-min = 0.75

Results:

Key Takeaways & Questions

• Text Generation (tg32): DSpark saw a small bump (~42.8 t/s vs 38.5 t/s on MTP).
• Prompt Processing (pp2048): MTP is visibly faster (258.5 t/s vs 214.8 t/s) and has noticeably lower time-to-first-token.
• Draft Limit Issue: Any value for spec-draft-n-max greater than 7 breaks/fails under DSpark on this build.

Has anyone else testing DSpark on Intel/SYCL backends seen similar scaling caps, or tuned spec-draft-p-min / batch sizes to get a bigger generation lift?

If anyone knows where to get help to get that extra boost let me know, also if anyone knows how to increase my prompt processing in llama.cpp let me know too!

I got tired of "AI in the browser" projects that are just thin wrappers around remote APIs, so I built Brimkern, a WebGPU inference engine that runs real models entirely on your machine. No server, no API, zero data leaving your browser.

What’s under the hood:

• Hand-written WGSL shaders: The forward pass uses custom compute shaders (fused quantized matmuls int8/int4/int3, GPU-resident KV cache, single-submit decode). No onnxruntime-web or transformers.js for inference (transformers.js is only used for tokenization).
• Custom .brik format: Weights are pre-quantized in the exact layout kernels read (no dequant-on-load), self-describing (architecture + tokenizer inside), and Range-streamed so the UI opens in seconds while resuming partial downloads.
• Per-architecture kernels: RoPE variants, QK-norm, SwiGLU/GEGLU, GroupNorm, causal + temporal attention, short-conv for hybrid models, and direct conv2d for diffusion. Every kernel self-validates against a CPU reference on load to prevent silent failures.

What it runs today:

• LLMs / SLMs: Qwen 3 (0.6B / 4B), Qwen 2.5, Llama 3.2, Gemma 2, Ministral 3, DeepSeek-R1 distill, LFM2 (hybrid conv+attention), RWKV-7.
• Vision & Image/Video: Qwen2-VL, SD-Turbo / SDXS, AnimateDiff-Lightning. You can also drag-and-drop your own GGUF files (converted to .brik in-browser).

Honest limitations: WebGPU only (Chrome/Edge, recent Safari & Firefox). Large models need a discrete GPU, though lighter ones (like a 149 MB hybrid model) run on integrated GPUs and phones. Video generation is compute-bound and slow. It's a solo project, so expect some rough edges.

Bonus: There’s also an early-access embeddable SDK. A single <script> tag drops an on-device assistant onto any site, compute runs on the visitor’s GPU, making it private by design and free to scale.

• Repo (MIT):https://github.com/RomainKH/Brimkern
• Live Demo: https://brimkern.romainkhanoyan.fr/

Would love any feedback, especially regarding kernel performance and the .brik format design!

https://www.reddit.com/r/pdf/s/EOipfBlRdq

Hi everyone! 👋

​I'd like to share an open-source side project I've been developing called AnvilAI — a native Android client designed to run Large Language Models (LLMs) completely on-device without relying on cloud APIs or external servers.

​Most mobile AI wrappers require cloud subscriptions or send user data to remote servers. I wanted to build something native, fast, private, and secure for Android devices.

​Key Features:

​Vulkan GPU Acceleration: Built with a C++ NDK engine layer to leverage mobile GPUs for real-time token generation.

​100% Offline & Private: Zero cloud dependency and zero telemetry. Your prompts and outputs never leave your device.

​Encrypted Storage: All local chat history and settings are encrypted at rest using SQLCipher.

​Modern UI: Built 100% in Jetpack Compose (Material 3) with clean architecture (Hilt, Coroutines, Flow).

​Source Code & Download:

The project is 100% open-source! You can check out the source code, inspect the architecture, or download the pre-built APK from the GitHub Releases tab here:

​https://github.com/denizaydogan1902/AnvilAI

​I would love to get your thoughts, UI/UX feedback, or ideas for future updates. Feel free to leave a star ⭐️ on GitHub if you find it useful!