Laguna seems very promising model for my setup(RTX 3090, RTX 3080 TI, DDR 48GB), especially as the alternatives from Qwen 3.6 27B, so I tried Unsloth's IQ4 ver. I haven't tried coding yet, but when I firstly say hello to him, somehow He mentioned COVID-19. Is that because Laguna is the US's model? I don't know, but feel funny.

Qwen3-30B-A3B on an Orange Pi 6 Plus — real-world performance

After a fairly large number of tests and configuration experiments on ARM SBC hardware, I managed to run Qwen3-30B-A3B Q4_K_M on an Orange Pi 6 Plus with 32 GB of RAM.

The model is running through a local OpenAI-compatible server with:

• 32 GB RAM
• 8 CPU threads
• 4096-token context
• reasoning disabled
• CPU inference
• no discrete GPU

For a practical test, rather than a short synthetic benchmark, I used a 208-token prompt requesting a structured business and software architecture analysis.

Results:

• Prompt processing: 32.21 tokens/s
• Text generation: 15.62 tokens/s
• Generated tokens: 512
• Generation time: 32.79 seconds
• Total request time: 39.30 seconds
• Memory usage for the server process: approximately 23.6 GB

The response reached the configured 512-token limit, so the generation measurement was long enough to be reasonably stable.

I was honestly surprised by the result. Around 15.6 tokens/s from a 30B-class MoE model on a single ARM SBC makes it usable not only for experimentation, but potentially for a local assistant, RAG, home automation or business workflows.

I performed several tests before reaching this configuration, but for now I am keeping the exact build, compilation options and low-level optimizations private.

Has anyone else tested this model on the Orange Pi 6 Plus or on similar ARMv9 SBC hardware?

Hello, guys.

I want to upgrade my PC mainly for local agentic coding, not gaming.

Current setup:

• Ryzen 7 5700X
• Gigabyte B450 AORUS Pro
• 32GB DDR4
• GTX 1660 Super 6GB
• Corsair CV650 650W Bronze
• Windows 11 LTSC + CachyOS
• llama.cpp / ik_llama.cpp with Pi Agent

I currently pay around $100/month for Claude and still hit limits.
I do not expect local models to replace frontier models, but I would like to move repetitive, private and token-heavy coding tasks locally.

OPTIONS

RTX 3060 12GB used: 180€-230€

Cheap, CUDA-compatible and works with my PSU. But is 12GB enough for decent models, or only small models with very limited context?

RX 9060 XT 16GB new: 430€-500€

Good VRAM per euro and efficient, but I am concerned about ROCm, ik_llama.cpp compatibility and mixing AMD with my current Nvidia card. Still not sure about the LLMs that could fit here.

RTX 5060 Ti 16GB new: 465€-600€

Probably the easiest and most efficient option, but it feels expensive for only 16GB.

RTX 3090 24GB used: 750€-1000€

Much better for 27B-35B models, but I would also need a new 750-850W PSU. Total cost would be around €700-850, and good used units are difficult to find.

However, I would also need a better 750-850W PSU, making the realistic total cost approximately €850-1,150.

Good units are difficult to find, and I am also concerned about power consumption, heat, card condition and whether it fits my case.

MODELS

Models I am considering include Qwen 3.6, Gemma 4, Bonsai and other coding-focused GGUF models.

My motherboard has a second PCIe 2.0 x4 slot, but I do not think it is a sensible base for dual GPUs.

Questions:

• Is an RTX 3060 12GB actually useful for coding agents?
• Is 16GB a meaningful upgrade or still too limiting?
• At what total price does a used RTX 3090 stop being good value?
• Has local inference actually reduced your Claude or API spending?
• Which option would you choose at these prices?

Thank you so much for your help.

My Mac mini is M4 pro 24GB Ram 512 ssd

I am thinking to run Claude code in remote control mode just for the sake of editing or querying my Obsidian notes through my phone when I’m out of home.

Since I assume such tasks are relatively simple, I thought that I can save quota of my cc subscription I use for coding and leverage local model solely for this purpose. Which model would fit my Mac mini and provide good experience and quality for this task?

Hosted web search from Anthropic and OpenAI costs $10 per 1k searches, Gemini costs $14, and then you pay again for the ~17k tokens of results each search dumps into context. I got annoyed enough to build an alternative.

It’s called webfetch. Runs locally, free out of the box (DuckDuckGo needs no API key), and in my SimpleQA benchmark the same agent loop hits the same accuracy as hosted search (96%) costing 66% less using 87% fewer tokens.

How it works:
1. RRF fusion across 4 search engines, local page fetching, hybrid BM25 + bi-encoder retrieval with a cross-encoder reranker

1. Sentence-level compression that cut result tokens in half with no measured recall loss

2. Semantic caching: paraphrased queries (“what did TypeScript 5.9 add” vs “TypeScript 5.9 new features”) get matched by embeddings and verified by an NLI cross-encoder, so reworded repeats cost nothing. Cache TTLs adapt to how volatile the answer may be

3. Every cached result shows provenance and the model can force a fresh search if it doesn’t trust it

4. Benchmarked against Anthropic hosted search, OpenAI, Tavily and Exa.

One small agent loop that I ran for testing that conducted just 16 websearches (opus 4.8) already reported 1.5 USD in savings.

Install from PyPI using pip.

Repo: https://github.com/firish/webfetch

TL;DR: I challenged myself to ship a real, working product using only a local model — no cloud, no shortcuts. Result: a Chrome extension — select text in any editable field, a floating action bar appears, pick an action (Fix / Improve / Translate), and the text is replaced in place by a local Ollama model. Each action has its own prompt, model and icon, and you can add your own presets. Private, fast, offline. 10-sec demo below.

Why I actually wanted this: I personally write most of my emails in a rush — typos, half-finished sentences, whatever comes out first. If you care at all about how the email you send looks, this genuinely helps: select, click Fix, done, still in your own words. Same for a Reddit post before you hit submit. It's not a novel idea — I already have an enhancer feature built into CO_DE (my main project). But I wanted to rebuild it from zero as a standalone Chrome extension that works everywhere in the browser. I knew exactly what I was aiming for going in.

Setup: unsloth/Qwen3.6-35B-A3B-MTP (Q4) on llama-server, Apple M5 Max 128 GB, MTP speculative decoding. Same server, same model for the whole build. ~5 hours across two sessions — struggled one evening, closed it the next.

Getting a local model to an actually-working, actually-useful result end-to-end is NOT trivial — it took real iteration to get here, and I'm not going to pretend it was a one-shot. If you try it yourself, expect to fight the model a bit. That's kind of the point of the challenge.

Asked the model itself what got fixed today — this is its own list, not mine:

Bugs fixed today:

Popup JS ID mismatch — HTML used lte-prefixed IDs, JS used non-prefixed ones. Every control returned null.

Global model selector — JS treated it as <select>, HTML had a <button>. Rewrote to use dropdown model selector.

Master toggle — JS used .checked (checkbox), HTML used custom button with on attribute. Fixed to toggle on attribute.

normalizeEndpoint was async without await — URL became "[object Promise]/api/chat". Made it synchronous.

window.__lteLoaded threw in cross-origin iframes — wrapped in IIFE + try-catch.

Content bar never appeared — bar was created only after chrome.storage.local.get() resolved (async). Fixed: bar created synchronously immediately on script load.

Focusin listener missed events — was using non-capturing mode. Changed to capturing.

Collapse/expand broken — toggleCollapse checked the wrong element and returned early. Rewrote.

Action cards were hardcoded open then broken — fixed default collapse state.

Sublabels overlapping content — <label> elements colliding with inputs/textarea. Changed to <div>.

Padding too tight — increased from 10px 12px to 14px 16px.

Still some polish left to do — but polishing with a local model isn't the best time/value spend past a certain point, so I called it here.

The extension: MV3, plain JavaScript, no build step, no external libraries, everything local (Ollama at localhost:11434). Works in Gmail, Reddit, anywhere text is editable. Popup config: endpoint, global model, per-preset prompt / color / model.

happy building local!

attached and the fixes, work from today

Hey folks — I’ve been tinkering with MLX and Apple Silicon for a while, and I finally wrapped up a project I’m pretty proud of.

I built a fully native Swift + MLX implementation of FLUX.2 [klein].
No Python. No diffusers. No external dependencies. Just Swift, MLX, and Metal doing their thing.

What it can do:

• Text‑to‑image
• Image‑to‑image
• Mask‑guided editing (this part was surprisingly fun to build)
  • remove stuff from an image
  • replace backgrounds
  • add objects
  • recolor regions
  • semantic edits
• Pixel‑space color grading (exposure, contrast, hue, saturation)
• Experimental latent‑space transforms
• Memory system with:
  • bf16 / fp16 / int8 / int4 quantization
  • staged model residency
  • VAE tiling for big resolutions
  • memory reporting + low‑memory mode

Everything runs entirely on Apple Silicon.
It hits seed‑42 parity with the MLX Python reference, so the outputs match exactly.

I also added:

• a dependency‑free CLI
• a SwiftPM library
• a tiny SwiftUI demo template
• docs + tests
• a contributor guide
• a roadmap if people want to help build more editing tools

Why I built it:

MLX is honestly a joy to work with, but most diffusion pipelines are still glued to Python.
I wanted something that felt native — something you could drop straight into a macOS or iOS app without dragging half the Python ecosystem along with it.

Repo:

https://github.com/icakinser/mlx-flux2-swift

If you’re into MLX, Apple Silicon, or just like messing with local image generation/editing, give it a look.
Happy to answer questions or help anyone get it running.

I am trying to solve a pretty (in my head) simple use case. Intake a list of companies, proceed to make search queries about these companies (news, announcements, results) for articles posted within the past 7 days and dump title, snippet, url, etc into a file for later processing. Silly me, apparently search is really really hard even in 2026.

So far I've tried: Exa, Tavily, Serper, Serpbase, Firecrawl, SearXNG and some others and none seem to produce anything even remotely acceptable.

1. This is a big one, vast majority of search backends either outright do not support "freshness" or produce bad to non-existent results when you try to employ it. Meanwhile I can go to Google, make the same exact query and get the desired results.
2. With Google I can enter "COMPANYNAME news announcements results" as a single query and get decent results. With various search backends, I seem pigeonholed into making 3 separate queries to get anything even remotely reasonable. Is this a deliberate tactic to get people to burn through their API credits?
3. Results are often cached?

With self-hosted models, I feel like I went 2 years back in time and this is acceptable to me. With search, however, I feel as if the jump is 30 years back, something of the Altavista age. How is any of this acceptable? How are people PAYING MONEY for this quality? What are the big boys using for their searches, Google deals behind closed doors (Google no longer offers search API directly)?

What are you using and how did you have to wrangle with it to get acceptable behavior of it?

I'm looking to get into local LLMs and start running my business using agents. I have the opportunity to buy a lightly used Macbook Pro 16" with m4 max and 128gb of RAM, 8TB hard drive for about $5K locally. Would this be a solid set up and a good deal? Looks like the equivalent M5 set up is selling for about $10k

Most voice-agent demos are too polite.

User speaks clearly. Agent waits. User gives one intent. No one interrupts. No one changes their mind. No background noise. No bad mic. No weird names.

Real users are not like that.

My eval set now is basically “people being annoying on purpose.”

Test calls:

1. user gives phone number, then corrects it

2. user says “don’t cancel”

3. user talks while agent is speaking

4. user asks two things at once

5. user changes date mid-call

6. user has bad mic

7. user pauses too long

8. user is angry

9. user gives address with landmark

10. user spells email

11. user says “actually never mind”

12. user asks for human

13. user uses slang

14. background noise

15. call reconnects

For each test, score separately:

• transcript accuracy

• entity accuracy

• correction capture

• barge-in

• latency

• task success

• handoff quality

• summary accuracy

When testing STT, I’d do one thing very strictly:

Keep everything else fixed.

Same prompt. Same voice. Same workflow. Same call audio. Swap only STT.

That’s where Smallest AI Pulse can be evaluated fairly: not as a landing-page claim, but as the real-time transcription variable inside chaotic voice-agent evals.

Happy-path demos prove almost nothing.

What ugly test case would you add?

Hi everyone!

Just wanted to know if anyone here is from NYC - because I'm hosting an event next week for Local AI. We will be talking about training / inference / models / and leaders in the space.

A lot of CEO/CTO's in the space are signing up for this in Luma already. Would love to have anyone interested in local to come to this event. If it goes well, I'm going to try my best to make this happen bi-weekly.

If you have any questions please let me know!

Today an AI agent trying to browse the web is like a thief in a balaclava sneaking around a police academy. Site protections block it, challenge it, turn it away.

browser-search flips the script: your agent stops being the thief and becomes the chief of police. No more clumsy access attempts. It walks through every door because it has the right tools. SearXNG for search, Camofox for browsing, CloakBrowser when things get tough.

100% self-hosted, free, no limits, no API keys.

I just released v2.0, whose core logic enforces the exclusive use of deterministic scripts. This eliminates model hallucinations, even with the cheapest models. The skill describes the 3 tools in natural language, but execution is rigid: the model can neither get the command wrong nor misinterpret the output. The result is guaranteed success on every query — the skill and deterministic scripts guide the model to scour the web until it finds the answer.

No more excuses. Your agent has the badge now.

https://github.com/Johell1NS/browser-search

I recently asked Claude Sonnet and ChatGPT if I could run Unsloth Gemma 4 12B (QAT Q4) on the MacBook Air M5 (16GB Unified Memory) I'm getting my laptop delivered to me this week.

I'm not expecting this machine to run local models perfectly at all. Having a chat with a local LLM and some roleplay through a terminal would be great.

When I had asked both models if I could that model onto that device, they gave me different answers about the weights, runtime footprint, and KV Cache. Both of them kept giving out different answers about the KV Cache footprint upon every generation and I wasn't sure to trust their word on that.

Can anyone tell me how much would 32K tokens would be in Q8 KV Cache for the Gemma 4 12B QAT Q4 model? Or a rough estimate? I'm just curious.

(By the way, if anyone wants to recommend me try to any different models for this laptop, feel free to recommend if any.)

Qwen3.6 really is nice, I am currently running the qwen3.6:35b version on my local GPU. I have now tried to jailbreak a bit, such that I can use it more freely and make it sound more honest and direct, but none of my attempts have worked so far. Anybody know, how to disable the anti-jailbreaking measures?

Assume cost is even. Leaning 7900XTX because VRAM but I've read about improved AI performance with RDNA4.

Thoughts? Worth it or no? 32GB DDR4 and I'm not fussed with speed so I'm fine waiting if output from higher quant/parameters necessitate it.

Use case: coding, image gen.

My specs:
Windows 11 laptop with
rtx 4050 6gb vram. 64 gb ram. 4tb ssd with dram. Ryzen 5 CPU(6 cores)

I want to be able to use hermes with it or do coding for light scripts.

Hi!

I'm part of a university research and development team, and we're planning to build our own local AI infrastructure.

Our goal is to keep as much as possible on-premise because we often work with confidential data, student projects, research datasets, and industry collaborations where sending data to external APIs isn't ideal. We'd like researchers and students to be able to access AI services over our local network while keeping all data inside our infrastructure.

As a first step, we'll probably purchase a smaller workstation with an RTX 5090 so we can start experimenting while our main budget is being approved.

The bigger purchase is where we'd really appreciate some advice.

The budget isn't fixed yet, it will depend on what we can justify to management. We believe we could potentially secure up to around $30,000, but we need to demonstrate that the hardware is the right fit for our use cases. If our needs can be met with a significantly smaller investment, we'd rather know that than simply spend the maximum budget.

Our expected workloads include:

• Coding assistants (Qwen3-Coder, DeepSeek, etc.)
• General-purpose LLMs and RAG
• Image generation (FLUX, Stable Diffusion)
• Computer vision research
• Speech-to-text / text-to-speech
• Multimodal models
• AI agents and automation
• Fine-tuning and experimentation where practical
• Running local inference for multiple researchers and students simultaneously

This machine would support a variety of university projects, from software engineering and AI research to computer vision, robotics, NLP, and other student research projects.

Some questions:

• What would you consider the sweet spot in terms of price/performance?
• If you had a maximum budget of ~$30k, would you actually spend it all?
• One large multi-GPU server or multiple smaller machines?
• Which GPUs would you choose today?
• Is it worth waiting for upcoming hardware?
• Any recommendations regarding CPU, RAM, storage, networking, cooling, or chassis?
• Any lessons learned from your own builds?

We're much more interested in the optimal solution than the most expensive one.

Thanks! I'm looking forward to hearing your recommendations.

Hey everyone,

I’ve been testing out the new DSpark (DFlash) speculative decoding integration in llama.cpp using an Intel Arc Pro GPU running on the SYCL backend.

Comparing it against MTP (Multi-Token Prediction), I’m not seeing much of a speedup overall. Additionally, I ran into an issue where spec-draft-n-max cannot be set higher than 7 without crashing/failing.

Here are my configs and benchmark results for comparison.

Shared Global Settings

Ini, TOML

jinja = true
flash-attn = on
no-mmap = true
mlock = true
b = 2048
ub = 4096
cache-reuse = 256
cache-type-k = q4_0
cache-type-v = q4_0

1. Multi-Token Prediction (MTP)

Config:

Ini, TOML

model = /home/james/models/Qwen3.6-27B-MTP-Q4_K_M-New.gguf
ctx-size = 256000
reasoning = on
spec-ngram-simple-size-n = 4
spec-ngram-simple-size-m = 4
spec-type = draft-mtp
spec-draft-n-max = 3

Results:

2. DSpark (DFlash Speculative Decoding)

Config:

Ini, TOML

[qwen3.6-27b-coding-Dspark]
model = /home/james/models/Qwen3.6-27B-MTP-Q4_K_M-New.gguf
mmproj = /home/james/models/Qwen3.6-27B-MTP-Q4_K_M-New-mmproj-F16.gguf
ctx-size = 256000

# --- DFlash Speculative Decoding ---
np = 1
spec-type = draft-dflash
spec-draft-model = /home/james/models/Qwen3.6-27B-Dflash/Qwen3.6-27B-DFlash-Q8_0.gguf
spec-draft-n-max = 7
spec-draft-p-min = 0.75

Results:

Key Takeaways & Questions

• Text Generation (tg32): DSpark saw a small bump (~42.8 t/s vs 38.5 t/s on MTP).
• Prompt Processing (pp2048): MTP is visibly faster (258.5 t/s vs 214.8 t/s) and has noticeably lower time-to-first-token.
• Draft Limit Issue: Any value for spec-draft-n-max greater than 7 breaks/fails under DSpark on this build.

Has anyone else testing DSpark on Intel/SYCL backends seen similar scaling caps, or tuned spec-draft-p-min / batch sizes to get a bigger generation lift?

If anyone knows where to get help to get that extra boost let me know, also if anyone knows how to increase my prompt processing in llama.cpp let me know too!

Open AI has now ChatGPT app for Windows and it can do a lot, but if I want to install one locally for local LLM, what are my options

totally new to playing with local AI. thought I'd give it a whirl running on an old system. 5600x. 16gb ram and a 1070 8gb.

so I've been messing around with like 7b size models to try making bash scripts for fun just to see what I can do. I'm not a programmer at all. so those sized models were definitely struggling with just normal, non coder prompts. and based on my quick research I needed to find models that fit on my 8gb card.

then I saw a post of a guy running these giant models on an rx470 8gb. did some reading. eventually asked chatgpt how to do this. and it suggested https://huggingface.co/unsloth/gpt-oss-20b-GGUF.

I'm not really sure what's happening. I think instead of always using the entire model to "think" it just uses the parts it needs?

anyways it's significantly smarter. it wrote the script I wanted in about 5 prompts. it got 90% there on the first try. the rest of it was just minor stuff. I had to increase the ctx size to 32000 and the prediction to 12000. but it fits on the GPU. 7100mb of 8200. I get about 24 token per second. which works great for my use.

also tried Gemma-4-26B-A4B. Which barely fits, but it's pretty smart too. But it spent so much time thinking and planning I think it ran out of tokens before even getting to write the script. But I can't really increase the ctx size cause it barely fits on 8gb of vram. Still neat tho.

Now I'm wondering if I should find another cheap 10 series card with another 8gb of ram and maybe I can fit one of these types of models but a 40b. 🤷‍♂️🤷‍♂️🤔 Lol

anyways I had a cool time nerding out about that, wanted to share. haha

Edit: gpt-oss-20b-MXFP4.gguf is the full filename

Laguna S 2.1 is really good at coding for being a 118b model.

It solidly beats Nvidia's Nemotron 3 Ultra (550b).

It's a huge contribution to the open weight community to have such a specialized coding model.

Protorikis is back with a demo for his free AI testing GUI.
Note, he tested on Mac, so other hardware may have different trade-offs.

* good for 'predictable' prompts
* not so good for harder prompts & large context
* good for compute bound workloads
* bad for memory-bandwidth bound work
* he found on his Mac MOE models work better in general than dense, & MOE helped more from MTP on his computer.