Is there anything that can get either Qwen3, Omnivoice or Chatterbox to run on Comfy nodes running ROCm? (like not via Docker or some external solution)

For example, I don't see a way to get this to work through ROCm instead.

Edit:

I'm on a 9060XT

https://github.com/woct0rdho/transformers5-qwen3.5-recipe

It's time for GGUF to replace bitsandbytes as the base model format for low-VRAM LoRA training. It's actively supporting new model types such as MoE, linear attentions, and DeepSeek WTF attentions, and new quant types such as 1-bit quants.

Thanks to APEX quant which makes Qwen3.6-35B-A3B as small as 13.3 GiB, and with fused dequant-matmul/MoE kernels, it's possible to train Qwen3.6-35B-A3B with batch size 1, context chunk length 2048, LoRA rank 4, in 16 GiB VRAM without CPU offloading.

I've tested it on Strix Halo and it runs at 6.5 s/it. Arguably VRAM size is not the biggest problem on Strix Halo, but it should just work on RDNA3 GPUs, and not too hard to port to other GPUs.

A byproduct is https://github.com/woct0rdho/torch-ggml-ops , which provides PyTorch bindings of the GGUF fused dequant-matmul/MoE kernels.

I completed a full-weight, non-quantized Qwen-Image-2512 render on a single 16GB AMD Radeon RX 7800 XT using my custom native AMD inference engine.

Configuration

• Official Qwen-Image-2512 image transformer
• 20.425B BF16 parameters plus 5.58M FP32 parameters
• Complete 7.071B-parameter BF16 text-conditioning tower
• Official BF16 VAE
• Native 1664×928 Qwen resolution bucket
• Official 50-step sampling
• True CFG 4.0 with separate positive and negative conditioning
• No Lightning LoRA
• No quantization in any learned component used for text-to-image generation

The unused Qwen vision tower and language-model output head are not loaded because the text-to-image path does not execute them. Every learned component actually used to produce the image remains full-weight and non-quantized.

Result

• Total render time: 39m23.6s
• First attempt
• Zero retries
• Zero errors
• No out-of-memory failure

The full model does not reside inside the GPU’s 16GB VRAM. Gridline uses a custom streamed inference architecture that manages the workload across the available storage and memory tiers while keeping GPU residency bounded.

This is not a claim that the model fits conventionally inside 16GB, nor that it matches the speed of a fully resident datacenter accelerator. The result is that the complete full-quality Qwen-Image-2512 text-to-image path can execute successfully on an ordinary consumer Radeon without quantizing the model.

The machine also remained fully usable during the render: two monitors active, more than 20 browser tabs open, and video streaming without instability or obvious memory thrashing.

I maintain faster production lanes for routine generation. This full-BF16 50-step route is intended for final-quality images, complex scenes and highly constrained prompts where the additional render time is justified.

The broader takeaway is that VRAM capacity alone does not define what consumer AMD hardware can execute. Runtime architecture and memory management can substantially change the practical limits.

I would be interested to hear from others working on ROCm/HIP inference, low-residency execution or large-model deployment on consumer Radeon hardware.

I built flash attention 2 from source for using with ComfyUI. Results were not as good as I had hoped.

I have RX 9070 XT, paired with Ubuntu 26.04 and ROCm 7.14.0.

It took 3 tries, some debugging with Claude, and over 6 hours in total.

Finally, the build succeeded and Flash Attention was installed.

I run ComfyUI with —use-flash-attention and queue a few image generations with Krea 2.

There’s no visible generation speed up.

I switch to and back —use-pytorch-cross-attention.

It’s the same. There is no difference.

Did I waste 6+ hours? Or am I doing something wrong?

Been running quantized diffusion models (Krea2, Flux) on a 7800 XT through [ComfyUI-INT8-Fast-ROCM] https://github.com/aksugat/ComfyUI-INT8-Fast-ROCM — a fork of patientx's fork of BobJohnson24's original int8 node. The underlying Triton kernels were tuned for NVIDIA/CDNA hardware (large tiles, num_warps=8, deep pipelining) — a poor fit for RDNA3's 60 CUs and 32-wide wavefronts, so I put together an RDNA3-specific autotune config set instead.

What changed:

• waves_per_eu as an autotune key (the ROCm/Triton occupancy knob for this backend, instead of CDNA's MFMA-specific matrix_instr_nonkdim)
• Smaller tiles (≤128) across more configs, num_stages=1-2
• Additional BLOCK_K=128 configs for K-heavy layers (large reduction dim, small output — e.g. MLP down-projections)
• Fixed a correctness issue (other=0.0 float literal on an int8-typed masked load)
• Same tuning applied to the int4/ConvRot kernel, which had zero prior tuning (single fixed config) — bigger gains there since there was more room

Benchmarks (RX 7800 XT, gfx1101, real layer shapes pulled from an actual quantized Krea2 checkpoint, weighted by how often each shape occurs in a full forward pass, multiple runs averaged):

Honest caveats:

• Small layers (≤1536-dim, small batch) — plain fp16 can beat int8 outright, quantize/dequant overhead doesn't pay for itself at that size. Worth excluding tiny layers from quantization rather than assuming int8 always wins.
• The int4 kernel doesn't have a fused dequant epilogue yet (separate rescale step) — the int8 kernel does. Open item.
• RDNA1/RDNA2 will hang the GPU — no WMMA/MFMA on that hardware, there's no instruction path for tl.dot on int8 to lower onto. Not a tuning limitation, just doesn't work there.
• Only validated on RDNA3 (gfx1101) so far. RDNA4 shares the same matrix-core support and 64KB/CU LDS budget, so it should work, but I haven't benchmarked it — CU count differs enough (64 on a 9070 XT vs 60 here) that I wouldn't assume these exact configs are optimal there without a fresh autotune pass.
• Repo: https://github.com/aksugat/ComfyUI-INT8-Fast-ROCM
• DONT FORGET FOR INT4 there is separate node inside my package so you wont try load int4 files into int8 node.

I was using ROCm 7.2, and in comfyui it used to take 90+ seconds to generate 1 image with Krea 2 turbo.

I just installed ROCm 7.14.0, now it takes 30-50 seconds.

I'm trying to run Klein 9b in ComfyUI.

When I click run, the processes 'requested to load Flux2TEmodel' and 'requested to load Flux2' take 3 times as long as they do on the portable version in Windows. (The sample image gets generated in 4.5 minutes on Windows, and 15 minutes on Ubuntu)

I'm running 0.2.8 with ROCm 7.2, python 3.14 and pytorch 2.13. I'm using a 9060XT with 64GB of system ram.

The drives ComfyUI is installed on are formatted in ext4 and NTFS respectively.

Does anyone know what's making it take so long?

If your gfx906 card runs plain HIP kernels fine but every rocBLAS/rocSPARSE/rocALUTION call segfaults the moment a kernel launches, check this:

rocminfo | grep -o 'amdgcn-amd-amdhsa--gfx906[^ ]*' | head -1

If it prints gfx906:sramecc-:xnack-, that's the problem. The prebuilt ROCm math libraries only ship code objects for the sramecc+ variant (the usual datacenter MI50/MI60 config), and the HIP runtime segfaults instead of returning an error when it can't find a matching code object. With AMD_LOG_LEVEL=3 you'll see:

No compatible code objects found for: gfx906:sramecc-:xnack-

HSA_OVERRIDE_GFX_VERSION does not fix it — the mismatch is the sramecc feature flag, not the gfx version.

The fix is rebuilding rocBLAS, rocSPARSE, and rocALUTION from AMD's sources targeted at the exact ISA string, into a user-local prefix (no root, system ROCm untouched). I hit this on 2x Radeon Pro Vega II (Mac Pro 2019 running Linux, ROCm 7.2.3) and put everything up here:

https://github.com/Intermountainh8ter/rocm-gfx906-sramecc-fix

• Prebuilt release tarball (ROCm 7.2.3 basis) if you just want it working — extract, set LD_LIBRARY_PATH, done
• Parametrized build scripts if you're on a different ROCm version (~2-4h, rocBLAS/Tensile is the slow part)
• A validation suite ending in a real rocALUTION GPU CG solve so you can prove it works on your card

Results on my hardware: rocALUTION GPU CG converges bit-identical to a scipy reference, ~4.4x faster than a tuned 28-thread CPU CG at 4M DOF. (Below ~1M DOF the CPU still wins — bandwidth-bound problem.)

Everything's MIT (it's AMD's own code, just rebuilt), licenses included. Not affiliated with AMD. Hope it saves someone the debugging time it cost me.

I installed 7.14 in my ComfyUI venv to try it out, and immediately uninstalled it and reverted to my working version after around 15 min, lol.

I'm using an R9700.

First issue - I was getting OOM errors for resolutions that I had no problems with before. I fixed that by adding 2GB VRAM from my 7900 GRE as donor VRAM (via the MultiGPU library). Once it started sampling, I ran into this:
[ERROR] !!! Exception during processing !!! CUDA error: invalid argument

I wasn't in the mood to go down a rabbit hole troubleshooting this. I'm using JAMM's SageAttention fork (https://github.com/thu-ml/SageAttention/pull/368), so I'm wondering if there's a conflict with it and 7.14. I tried uninstalling it and recompiling with 7.14; same error.

Edit: Wait, now I'm wondering if the reason is because I installed the gfx1201 libaries as per AMD docs. I don't think this should matter since the GRE shouldn't be doing any work (it's just a memory donor; I'm not running anything with tensor parallelism), though.

I know there are a lot of different ways to use your gpu to run an llm or multiple model routing, but this is my version. Figured I’d share it here. It was purpose built because I had a good graphics card and wanted to use it as a reviewer inside codex. Open source. Hope it helps someone! You can get it here:

https://thalenai.com/

Is there a equivalent for AMD or Intel devices and how hard is to build one?

TL;DR: Raising the MI50’s power cap from 100W to 150W bought a 19% decode speedup for very little extra heat. Going all the way to 250W uncapped only added another ~12% on top of that, for a lot more power draw. Sweet spot was 150W.

Setup

• GPU: AMD Instinct MI50 (16GB VRAM), fan: a Delta AUB0912VH (DC 12V, 0.60A), speed-controlled by an RP2040-Zero (a Raspberry Pi Pico-compatible board) driving PWM based on live GPU temp readings.
• Host: HP Z440 workstation, Fedora 44, x86_64.
• Inference stack: llama.cpp compiled for gfx906, ROCm 7.2.4 backend, running in a rootful podman container.
• Model: Qwen3.5-9B, Q4_K_M quantization (~5.4GB), 32K context.
• Baseline config going in (“energy saving first”): power cap locked to 100W (down from the card’s 250W default), SCLK capped to ≤1372MHz, MCLK left at full range. This was for "energy saving first" and "silent fan".
• Ambient room temperature: 24°C during testing.

What I did

Ran the exact same inference workload (a mid-length reasoning prompt, 256 generated tokens, 3 iterations per config) across four different power/clock profiles, controlled directly via amdgpu sysfs knobs (power1_cap, pp_dpm_sclk, power_dpm_force_performance_level=manual). Between each profile we let the card cool back down below 45°C junction before starting the next one, so results aren’t contaminated by thermal carry-over from the previous run.

What I learned

• Decode throughput scales with power cap, but with steep diminishing returns. Going 100W→150W (+50W) bought +19% decode speed — by far the biggest jump. 150W→200W (+50W) only bought another +9%. 200W→250W (+50W) bought a measly +2.5%. Each additional 50W buys roughly half the gain of the previous 50W.
• Prefill/prompt-processing speed barely moved at all (66.7 → 72.6 tok/s, +9% across the entire 100W→250W range). This tells us the workload is decode-bound, not compute-bound — consistent with something we’d already found in earlier ROCm-vs-Vulkan benchmarking: single-user autoregressive decode on this card is HBM2 memory-bandwidth-limited, not SCLK-limited. Since we held MCLK constant across every profile (full range, same ceiling), raising the power cap mostly just lets the GPU sustain higher boost clocks during generation — it doesn’t touch the actual bandwidth bottleneck. That’s why prefill (which is much more compute-parallel and less bandwidth-starved) stayed almost flat while decode moved.
• Thermals stayed very tame the whole way, topping out at 55°C junction even fully uncapped at 250W — nowhere near the passive card’s 90°C+ throttle territory we’ve seen under long sustained sessions. Caveat: this was a short 256-token burst test, not a sustained multi-minute load, so longer real-world generations would likely run hotter than what’s shown here.
• Fan response was noisy/non-monotonic across profiles in the raw log — that’s just the independent temperature-tracking fan-curve daemon reacting live to instantaneous temp, not a property of the power profile itself. Not a useful signal on its own here.

My Recommendation

150W (T1) is the sweet spot for a “balanced performance + energy saving” profile. It captures the single biggest performance jump (+19% decode) for the smallest thermal cost (44°C→50°C, still nowhere near throttle), while T2 and T3 spend an extra 50-100W of continuous draw for comparatively little in return. Adopting 150W as the new default power cap (up from the original ultra-conservative 100W energy-saving baseline).

Card and host are otherwise idle/dedicated hardware — no other GPU workloads running concurrently during the test. Numbers are from a single benchmark pass, not statistically rigorous across many runs, but consistent enough between iterations within each profile to trust the trend.

This is running in Ubuntu. I'm on Pytorch 2.13.0 and ROCm 7.2 running on a 9060XT

I'm trying to generate an image in Flux Klein9b, it errors on the text encoder stage with the message:

## Error Details
- **Node ID:** 75:67
- **Node Type:** CLIPTextEncode
- **Exception Type:** torch.AcceleratorError
- **Exception Message:** torch.AcceleratorError: CUDA error: device kernel image is invalid
Search for `hipErrorInvalidImage' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Device-side assertion tracking was not enabled by user.

It looks like it's looking for CUDA still.

Anyone have a solution?

Initially tested ROCm 7.13 on Ubuntu 24.4 LTS and found that the CPU remained at 100% after ComfyUI image generation completed. It was noticeable because the fan on the Strix Halo was on full power when the machine should be mostly idle. Reverted to 7.2.4 and the issue went away. Now with 7.14, the cpu/fan issue has returned. I have to kill ComfyUI server whenever it is not in use.

I have been looking at nvtop to measure. During image generation, the cpu shows > 3000% then drops to 100% when it should be idle.

Hi guys,

I have a problem with my PC. The entire computer suddenly shuts down (complete power off, no BSOD) while or right after generating images in ComfyUI.

The shutdown happens randomly — it can occur on the first generation, the second, or after 4-5 generations. It only happens during AI image generation.

What I’ve already tried:

• Replaced the power supply (didn’t solve it)
• Checked temperatures (CPU & GPU are normal)
• Tested different ROCm + PyTorch versions:
  • rocm-7.12.0 + torch-2.10.0+rocm7.12
  • rocm-7.14.0 + torch-2.13.0+rocm7.14
• Used flags: --disable-smart-memory --disable-pinned-memory

System Specs:

• OS: Windows 11
• GPU: AMD Radeon RX 6600 XT 8GB
• CPU: Intel Core i5-10400F
• RAM: 32GB DDR4 3200MHz
• Driver: AMD Adrenalin 26.6.2 (WHQL)

Any ideas what could be causing this?

There are tutorials for installing on Nvidia GPU's, WSL, RyzenAI, Archlinux and every other combination under the sun.

Can anyone point me a guide for installing ComfyUI on Ubuntu with Radeon 9000's?

How can i get Comfy-UI in Unraid with ROCm?

"As a follow-up to the article over ROCm 7.14 being tagged, AMD has formally announced the availability of ROCm 7.14 and it's their new production release rather than being a tech preview.

Since ROCm 7.9, the 7.9+ releases have served as "tech preview" releases while ROCm 7.0/7.1/7.2 were stable series. Up to now my assumption was that the ROCm 7.9+ tech preview releases would culminate with ROCm 8.0 being stable. Now as quite a surprise, ROCm 7.14 was announced today -- ahead of next week's AMD Advancing AI event -- as the new production release. Quite the version numbering mess.

ROCm 7.14 "marks the start of our future production releases. users on 7.2 and older are encouraged to migrate and follow the transition guide on rocm docs" per the AMD documentation. Beyond the build system overhaul with TheRock, new to ROCm 7.14 is officially adding support for the Ryzen AI 400 series, including the new Ryzen AI Max+ PRO 495 / PRO 490 / PRO 485 products and lower-end Ryzen AI 430/440 series offerings.

ROCm 7.14 also adds official support for RHEL 9.8 and RHEL 10.2, SUSE Linux Enterprise 15 SP7 / SLES 16 / Debian 13 for the Instinct MI350P hardware, and more."

Spent some time today to make this work because I wanted to use my 6700XT for LLM inference under WSL. Works well for now; rocminfo detects my GPU.

https://github.com/the-zucc/librocdxg-rdna2

Which distro is recommended for compatibility and performance (using less Vram).

Are lighter versions of Ubuntu like Lubuntu just as good for running ROCm LLM's and ComfyUI?

Previous post x1.7 promt eval on dual GPU for Qwen 3.6
I added support as well for Teranry-Bonsai PQ2, and below you can find performance results for a single RX9070 XT and for dual mode (for Qwen Q3_K_S , in the table results are in non-MTP)

In additional did perfomance tests for the Q4_K_M model, and as I retested, I noticed that actually I can as well run this model in agent mode with long prompts, before I said that they spill to RAM. They ran with q8 KV. So, for someone, if q4 KV is okay, you will have enough VRAM still. But an important note, Vulkan takes much less VRAM, will work on it, as it seems I will fix it, ROCm will outperform Vulkan a lot in long promts

In conclusion, I didn't expect that two AMD GPUs would work very well together, and even though the second one gains a lot in prompt evaluation speed, if AMD adds peer-copy as well in the future ROCm HIP updates, it can also give additional speed for dual setups or who knows more GPUs in parallel, as rx9070xt are still cheap for their performance and memory.
Repo: MrLordCat/llama.cpp-with-GUI: LLM inference in C/C++