Portable LLMs with llamafile [LWN.net]

Portable LLMs with llamafile

Posted May 14, 2024 17:40 UTC (Tue) by snajpa (subscriber, #73467)
Parent article: Portable LLMs with llamafile

I don't get this need to fork llama.cpp just so that people can claim they have something of their own. This should have been upstreamed, just as pretty much every other llama.cpp fork. But most never try and fork right away. I fail to see how is that approach useful to anyone.

to post comments

Portable LLMs with llamafile

Posted May 14, 2024 17:46 UTC (Tue) by snajpa (subscriber, #73467) [Link] (5 responses)

I mean, llama.cpp is still a pretty young project where the codebase changes rapidly and these kinds of changes are what defines how the codebase is going to evolve - trying to spill that evolution out of the central pot just makes way more mess than necessary. If one takes a look at how the forks are doing - they all eventually fall way behind. It's a shame.

Portable LLMs with llamafile

Posted May 15, 2024 7:07 UTC (Wed) by rsidd (subscriber, #2582) [Link] (4 responses)

She is sending patches upstream. The whole point of open source is you can fork it for your own interests, which may not match upstream. But if upstream is interested, the fork can re-merge.

This project, as I understand it, is basically about (a) building llama.cpp with cosmocc (for improved portability: you can literally run the same output file on linux, windows, macos); (b) speeding up linear algebra (which she is feeding upstream where relevant) with the goal of (c) bundling LLM weights and llama.cpp into single, portable, fast executables that can use GPU when possible, or run with adequate speed on CPU, without user intervention or configuration.

Portable LLMs with llamafile

Posted May 15, 2024 9:55 UTC (Wed) by snajpa (subscriber, #73467) [Link] (3 responses)

I don't find it a normal practice at all. I have never seen people fork a project, spend an effort to come up with a unique name, make the effort to claim it's their project, only to be able to contribute to the original project. People just open a PR for that. But no, llama.cpp world seems special, everyone needs to claim _they_ invented a piece of this AI thing, like it's the last thing anyone will ever invent. IMHO an explanation like yours doesn't work (here).

Portable LLMs with llamafile

Posted May 15, 2024 9:59 UTC (Wed) by snajpa (subscriber, #73467) [Link] (1 responses)

Btw @ single exec file, this has been a done thing now for a while also. There's ollama, which also packs it all (and if it doesn't support all three platforms it's certainly their goal), but unlike _this_ llama.cpp fork, that project has a real added value. It really makes running LLMs easy for people, it abstracts the llama.cpp's rough edges to present a smooth workflows for people who never touched any of this stuff. It's also original code, which *includes* llama.cpp, rather than just "rebrands" it.

Portable LLMs with llamafile

Posted May 15, 2024 14:47 UTC (Wed) by daroc (editor, #160859) [Link]

I apologize if I gave the impression in my article that llamafile is only a rebranded llama.cpp — llamafile has a bunch of additional code (under a different license, even) that wraps llama.cpp. See this part of the source. The project has both a copy of llama.cpp which sends patches upstream, and support code which is used to produce the final binaries.

Portable LLMs with llamafile

Posted May 15, 2024 17:03 UTC (Wed) by niner (guest, #26151) [Link]

What about OpenWRT? LEDE was forked off OpenWRT in 2016 and in 2018 the two projects merged again.

Or earlier, GCC was forked into EGCS. Later that fork became so successful, that the remaining GCC developers decided to join the effort and EGCS was officially adopted as the new GCC.

Such things _do_ happen.

Portable LLMs with llamafile

Posted May 14, 2024 18:36 UTC (Tue) by ballombe (subscriber, #9523) [Link]

It seems to me this project is mostly about providing users binaries for llama.cpp.
Providing users binaries can seriously distract a project from its goal, so it can be a net benefit if they are provided by someone else.