Portable LLMs with llamafile
Portable LLMs with llamafile
Posted May 18, 2024 9:42 UTC (Sat) by Felix (subscriber, #36445)In reply to: Portable LLMs with llamafile by flussence
Parent article: Portable LLMs with llamafile
At least on my system (Fedora 40), I can run simple models like the llama3-8b using llamafile+rocm and I see a pretty decent speedup when using the GPU. I'm using the rocm packages provided by Fedora so I think the situation is not that bad even though there is a lot of things which are not great (e.g. support for more GPUs, more AMD work regarding distro integration, ...).