Low-bit LLMs for AMD GPUs

Frontier-size models on one 128 GB Strix Halo machine

ROCmFPX is a llama.cpp fork tuned for AMD GPUs, especially the Ryzen AI Max+ 395 (Strix Halo). The goal: run the big open models at home, at a quality you can actually use.

This is a first iteration. The current releases fit big models into 128 GB by removing the least-used experts and using existing 1 to 2 bit quantizations. That works, but it costs quality.

We are actively working on new ROCmFPX formats at around 2.1 bits per weight and below 2 bits that keep much more of the original quality and run fast on AMD GPUs. This will take time, and we need the community's help: testing on real hardware, reporting results, and telling us which models matter to you.

Releases

available

DeepSeek V4.1 Flash for Strix Halo

About 97 GiB on the GPU, Engram table read from the SSD. Community test release.

Hugging Face

in progress

MiniMax M3.1 Flash for Strix Halo

428B MoE, cut to fit 128 GB with room for a 64K context. Building and checking now.

Follow on Hugging Face

research

New ROCmFPX low-bit formats

About 2.1 bits per weight and below, built for AMD kernels. Early work.

Watch the repository

What should we quantize next?

Pick the one you care about more. Voting is a 👍 on GitHub, so you need a GitHub account, and each account counts once per option.

Big models, squeezed to fit

Frontier-size MoE models (DeepSeek, MiniMax, GLM, big Qwen) cut and quantized to 1 to 2 bits so they run in 128 GB. More knowledge, some quality loss, lower speed.

–

I'm interested: vote on GitHub

Smaller models, done well

Models that already fit (up to about 120B) at 4 to 8 bits, with the best quality and speed on AMD.

–