Portable LLMs with llamafile [LWN.net]

Portable LLMs with llamafile

Posted May 15, 2024 8:13 UTC (Wed) by taladar (subscriber, #68407)
Parent article: Portable LLMs with llamafile

GPU support does not seem to work for me on AMD. It seems to produce an assertion failure in hip on Gentoo

llava-v1.5-7b-q4.llamafile: /var/tmp/portage/dev-util/hip-5.7.1-r2/work/clr-rocm-5.7.1/rocclr/os/os_posix.cpp:310: static void amd::Os::currentStackInfo(unsigned char**, size_t*): Assertion `Os::currentStackPtr() >= *base - *size && Os::currentStackPtr() < *base && "just checking"' failed.

error: Uncaught SIGABRT (SI_TKILL) at 0x3e8001df83d on <hostname removed>
./llava-v1.5-7b-q4.llamafile
File exists

to post comments

Portable LLMs with llamafile

Posted May 15, 2024 14:26 UTC (Wed) by flussence (guest, #85566) [Link] (1 responses)

Does *anything* work in ROCm? My impression of it over the past 5-10 years, from the volume of complaints on the internet about it, has been that it's the best advertising campaign Nvidia could've wished for.

Portable LLMs with llamafile

Posted May 18, 2024 9:42 UTC (Sat) by Felix (subscriber, #36445) [Link]

At least on my system (Fedora 40), I can run simple models like the llama3-8b using llamafile+rocm and I see a pretty decent speedup when using the GPU. I'm using the rocm packages provided by Fedora so I think the situation is not that bad even though there is a lot of things which are not great (e.g. support for more GPUs, more AMD work regarding distro integration, ...).