Thx in advice.
Buy the cheapest graphics card with 16 or 24GB of VRAM. In the past people bought used NVidia 3090 cards. You can also buy a GPU from AMD, they’re cheaper but ROCm is a bit more difficult to work with. Or if you own a MacBook or any Apple device with a M2 or M3, use that. And hopefully you paid for enough RAM in it.
I actually use an AMD card for running image generation and LLMs on my PC on Linux. It’s actually not hard to set up.
I have Fedora installed on my system (don’t know how the situation is on other distros regarding rocm) and my GPU is an RX 6700 XT. For image generation I use stable duffusion webui and for LLMs I use text generation webui. Both installed everything they needed by themselves and work perfectly fine on my AMD GPU. I can also give you more info if there’s anything else you wanna know.
I’m not the original person you replied to, but I also have a similar setup. I’m using a 6700XT, with both InvokeAI and stable-diffusion-webui-forge setup to run without any issues. While I’m running Arch Linux, I have it setup in Distrobox so its agnostic to the distro I’m running (since I’ve hopped between quite a few distros) - the container is actually an Ubuntu based container.
The only hiccup I ran into is that while ROCm does support this card, you need to set an environmental variable for it to be picked up correctly. At the start of both sd-webui and invokeai’s launch scripts, I just use:
export HSA_OVERRIDE_GFX_VERSION=10.3.0
In order to set that up, and it works perfectly. This is the link to the distrobox container file I use to get that up and running.
An M1 MacBook with 16GB cheerfully runs llama3:8b outputting about 5 words a second. A second hand MacBook like that probably costs half to a third of a secondhand RTX3090.
It must suck to be a bargain hunting gamer. First bitcoin, and now AI.
edit: a letter
Ok. I get it now. I’ve been trying to build something cheap as a Linux gaming setup and I’ve come to the conclusion that I’m better off buying the steam deck.
