Questions about machine-learning models for Fedora [LWN.net]

By Joe Brockmeier
March 13, 2024

Kaitlyn Abdo of Fedora's AI/ML SIG opened an issue with the Fedora Engineering Steering Committee (FESCo) recently that carried a few tricky questions about packaging machine-learning (ML) models for Fedora. Specifically, the SIG is looking for guidance on whether pre-trained weights for PyTorch constitute code or content. And, if the models are released under a license approved by the Open Source Initiative (OSI), does it matter what data the models were trained on? The issue was quickly tossed over to Fedora's legal mailing list and sparked an interesting discussion about how to handle these items, and a temporary path forward.

Defining terms

AI/ML is becoming, or at least feels, nearly omnipresent these days. However, the terminology may be confusing to those who are outside of that bubble, so Tim Flink helpfully supplied some definitions of terms when he carried the discussion from the FESCo ticket to Fedora legal.

To sum up Flink's definitions, a model describes an artificial neural network (ANN), which is structured data "consisting of neurons (nodes containing some value) organized into layers with various connections between the neurons". A model describes a specific ANN, how its layers are configured, how the data is structured, and the learning algorithms that will be used when the model is trained on data to assign values (weights) to its nodes.

A model by itself is not of much use until it has been trained on data and has had weights assigned to it to provide the "exact value of how the connections affect flow through the network". As an example, one might want to use PyTorch's torchvision library, which offers models for tasks like image classification and object detection, to determine if there's a cat in a picture. One of the torchvision models might be suitable for this, but it would need to be trained on data (pictures with and without cats) before it would be able to do its job.

Users could do the training themselves but, as Flink noted, doing the training to create the weights "is a very expensive and time consuming process". So projects like PyTorch provide pre-trained weights—that is, models that are populated with training data—so that users can make use of the models without having to do the training, or even having access to the data used to train the model. For example, the PyTorch project offers torchvision models with and without weights.

Herein lies the perceived problem: while the weights and models may be offered under a license that is approved for Fedora, there's uncertainty about the data used to do the training. Copyright holders have been suing over the use of their data in training models, and the AI/ML SIG has hesitated to package weights due to those lawsuits and the questions about the input data used—even if the model is under a license accepted by Fedora.

In his first message in the thread, Flink asked whether weights are normal non-code content, or if they "require special handling" even if the upstream offers the models under a license "acceptable for non-code content in Fedora". Do packagers have any responsibility for reviewing the training data, in other words, or is the upstream's choice of license sufficient to make that call? So far, the SIG has erred on the side of caution and refrained from packaging weights, but that has its own drawbacks.

Supplying software like PyTorch without weights is a lot like supplying a spellchecker without the dictionary it needs as a word list. It would be easier for users if they didn't have to seek out the weights separately from installing the software—but the SIG wanted public guidance from FESCo (and then Fedora legal) before proceeding.

Are AI/ML weights special?

Richard Fontana, a member of Red Hat's legal team who advises Fedora on legal issues, wrote that he had recommended bumping the questions to FESCo because he thought FESCo might decide weights were "analogous to object code" that users must be able to modify. Fontana alluded to discussions being led by the OSI over the definition of "open-source AI" and said there is "definitely some sentiment among participants in that effort" that training data must be open because "this is necessary for users to exercise rights of modification." It would seem, however, that FESCo is satisfied with the idea that weights are content unless Fedora legal decides otherwise.

Fontana said that he was "struggling to see a justification" not to consider weights as content. However, he indicated that Fedora should be cautious, at least initially, and take weights "on a case-by-case basis" until Fedora has more experience with this type of content. Fontana said there may be circumstances when Fedora might not want to package the weights "given what is disclosed, or not disclosed, about how a model was trained". That was, he said, "unlikely, but not impossible".

In a followup message, Fontana elaborated on this and said that he wants to do further review "for any specific pre-trained weights that will actually be included in Fedora packages, for some initial period", since it would be "highly impractical" to expect packagers or package reviewers to do this type of review. Fontana said that if there are technical issues, such as "if there ought to be some standards around packaging of upstream pre-trained weights" he would not be able to give guidance "beyond my initial suggestion to raise this topic with FESCo which seems to have been unsuccessful".

FESCo member Neal Gompa replied:

With my FESCo hat on, the main question to answer is how we classify and identify them for package reviews, which is largely a Fedora Legal question. Personally, it's basically content to me, we do probably need some explicit documentation of this for the guidance that the AI/ML SIG can use to write packaging guidelines for FPC [Fedora Packaging Committee] to review.

What about build-time downloads?

Flink had also asked about scenarios where software packaged for Fedora might download weights upon first use. He used torchvision as the example of a library that is already packaged for Fedora as python-torchvision with the weights removed. If a user calls one of the models when using the torchvision library, such as the vit_b_16 model then the weights are downloaded from a third-party site on first use. Flink noted that some of the weights are under licenses acceptable as content for Fedora, some are under Creative Commons licenses that are not acceptable for content packaged for Fedora, and others have no explicit license at all.

Those scenarios, Gompa suggested, were "in the same bucket" as Python's pip, Ruby's gem, and other software with package-manager functionality that downloads software from sources outside of Fedora's repositories. Fontana agreed, but Flink wrote that "the capabilities do overlap but in my opinion, the intended uses are different and that may be worth noting". He also explained that downloads happen "transparently to the user with no warning outside of a log message when the weights are first downloaded". This is a bit different than, say, installing Python software with pip because users have to explicitly run a pip install command to download software. Flink said he was not arguing against having this functionality, but wanted to be sure it was explained correctly.

Fontana replied that he felt Flink was raising a more general issue that was not specific to pre-trained models. He noted that any Fedora package could download things with no warning to the user, and that he was unaware of any Fedora technical or packaging guidelines that address that issue. At this point, Gompa seemed impatient with the discussion, noting that lots of Fedora packages have similar functionality, and suggested that packagers "tweak pytorch to require configuration or make a prompt when it triggers the first time or something" if the packagers were concerned about that functionality.

Next steps

To date, no packages including pre-trained weights have been submitted for review. I followed up with Flink by email, and he confirmed plans to package weights for torchvision so that Fedora packages could be used alongside upstream PyTorch vision-related tutorials. It will be interesting to see what the case-by-case review looks like and what packages, if any, may be held back from Fedora due to concerns over the training data used.

to post comments

Questions about machine-learning models for Fedora

Posted Mar 13, 2024 21:58 UTC (Wed) by mattdm (subscriber, #18) [Link] (3 responses)

Note that we _do_ allow CC0 for "content" -- just not for "code".

Questions about machine-learning models for Fedora

Posted Mar 13, 2024 22:00 UTC (Wed) by mattdm (subscriber, #18) [Link] (1 responses)

CC-NC is not allowed in general, though.

Questions about machine-learning models for Fedora

Posted Mar 13, 2024 22:57 UTC (Wed) by gutschke (subscriber, #27910) [Link]

Non-commercial licenses such as CC-NC are a legal nightmare anyway. They were popular in the 1990s, when people experimented with a wild assortment of variations on the general shareware theme. Most of these licenses were written without input from any lawyers and also never saw their day in court. Fortunately, we collectively moved away from them for the most part.

They still exist in some niches though, and nobody has authoritatively ruled on what exactly constitutes commercial use. It's such a poorly defined term that you could easily run afoul of accidentally. I agree with a policy to simply avoid any such products. It's not worth the headache.

I sometimes see CC-NC in the context of 3D models, and I always feel it's completely pointless. It won't stop sufficiently determined overseas resellers. If those shady outfits aren't scared making cheap copies of famous brands, they won't hesitate to rip off a small hobbyist who posted their own creation either. But by marking a model as CC-NC, it potentially scares off lots of legitimate users who are less morally lacking. And that's a pity. Why even release the model in the first place unless you want it to be enjoyed by others.

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 12:39 UTC (Thu) by jzb (editor, #7867) [Link]

Thanks for the correction. Sorry about that - I've removed the link. I haven't found a better one, though, that only spells out the unacceptable content licenses.

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 0:58 UTC (Thu) by rfontana (subscriber, #52677) [Link] (1 responses)

> And, if the models are released under a license approved by the Open Source Initiative (OSI), does it matter what data the models were trained on?

Note that Fedora allows licenses for code, and "content", that are not limited to the relatively small set of OSI-approved licenses, and there are also a few OSI-approved licenses that are not allowed in Fedora.

Questions about machine-learning models for Fedora

Posted Mar 27, 2024 18:55 UTC (Wed) by sammythesnake (guest, #17693) [Link]

There's the question of what constitutes the "source code" for a model with trained weights (if it's to be treated like compiled code, which is a reasonable position to take)

The only logical answer I see is that the full training set is the source, while the definition of the layers and the training procedure would be the "build scripts" and the engine would be the "compiler". Trained models rarely come bundled with the whole (potentially massive) training data. I'm sure quite a few publicly available models are trained on open training sets (like those at https://www.openml.org/search?type=data for example) though.

To distribute a model as software under a must-provide-source license would require clearing up questions like these. For the GPL, for example, the availability of the training data must be guaranteed for some period of years, which might be difficult if it's hosted by some third party - especially if it's a dataset subject to change over time...

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 3:24 UTC (Thu) by PengZheng (subscriber, #108006) [Link] (1 responses)

Weights are content. But the real issue is that content needs review, to ban child pornography etc.

Right now how to review a pile of numbers without access to the training process?

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 18:10 UTC (Thu) by Heretic_Blacksheep (guest, #169992) [Link]

Arguably that's the crux of the problem. Models should be required to have an audit trail of the data they were trained on so it can be reviewed for both integrity and legal landmines. Neither of these cases are sufficiently addressed in LLM public models (hence all the lawsuits). It's also why SLMs strictly trained on curated private internal data is likely the way forward. The legal landmine for wild publicly accessible data is definitely an open question right now. It's a bad idea to ignore the legal side and potential copyright entanglements just because you don't understand the legal principles involved. When in doubt, don't! I cringe every time I see someone expounding on how great Copilot is, as I have some sense of the legal can of worms being opened there should the AI companies spectacularly lose those IP lawsuits (especially with some of the cringe-worthy defenses I've seen filed). It's not clear AI model training is legally fair use or where license exhaustion occurs if at all.

In Fedora's specific case, the question about downloading the pre-trained model content can only be legitimately decided by the end users. Ultimately its their legal risk with potentially co-mingling licensed output in their projects. Therefore, a suitable non-judgemental (non-lawyers can't render legal opinions in the US at least) statement about the potentially legal gray area, for a definitive answer consult a qualified attorney in your legal jurisdiction yada yada... for models along with a link to the audit trail (no audit trail, don't include it at all regardless of license), then a simple Yes/No response is appropriate.

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 6:32 UTC (Thu) by IanKelling (subscriber, #89418) [Link]

Another potential problem: The licenses of the training data may significantly apply to the model. OpenAI & Microsoft are paying a lot of money based on that premise:

https://www.wnycstudios.org/podcasts/otm/segments/news-ou...: "The multi-year agreement compensating Axel Springer for the content OpenAI will use to generate answers on ChatGPT and train its models." "The Associated Press signed a similar deal with OpenAI earlier last year." "OpenAI is reportedly in talks with CNN, the Fox News Corporation, and Time to license their work"
The New York Times lawsuit against OpenAI for copyright infringement happened after NYT wanted more money than OpenAI wanted to pay for a licensing deal.

We know they often output copyrightable chunks of their training data and a judge has to decide if that is ever fair use. So, many models which are claiming to be under a single license could actually be under the many licenses of their training data.

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 14:12 UTC (Thu) by rrolls (subscriber, #151126) [Link] (10 responses)

I do rather like the comparison between AI weights and object code.

In theory, the problem is quickly resolved if one mandates that weights can only be distributed alongside the training data (or a binding offer to receive it). Reviewing weights is practically impossible as far as we know, but reviewing training data is straightforward (even if time-consuming for large data sets). If, somewhat similar to the GPL for code*, one were to stipulate that weights may only be distributed alongside the training data and the tools and instructions to reproduce the output weights bit-for-bit to prove that that's where they came from, anyone reviewing a package of weights would merely have to a) review the training data and b) rebuild the output for verification. The weights can then be distributed under the same license that any conventional combination of all the inputs would be.

(*Before anyone feels the need to point this out: no, the GPL does not require bit-for-bit reproducibility for code; that's besides the point. Hence my saying "somewhat similar".)

In practice, there are currently three problems with this approach:

1. rebuilding weights might require a lot of computational power, especially for large data sets
2. obtaining verified "clean" data sets, where all inputs have been acceptably licensed, might require a lot of work, either to obtain acceptable licenses for existing inputs, or to collect new inputs
3. it isn't status quo: status quo is that, in general, people don't bother with this and don't care about the legality (or morality) of their outputs

I think the theoretical ideal is worth chasing, and I know that humanity has the ability to achieve it. Until then, anyone trying to solve the legal and moral issues with AI will carry on going round in circles, because there isn't really any other proper answer.

I'm certainly not holding my breath for anyone to decide this is hill to die on, though.

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 15:05 UTC (Thu) by Paf (subscriber, #91811) [Link] (6 responses)

“ 1. rebuilding weights might require a lot of computational power, especially for large data sets”

I think this may misstate the severity of this problem.

The most powerful models have compute costs to train in the hundreds of millions. That makes reproduction simply impractical, I think. That matters…

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 16:10 UTC (Thu) by epa (subscriber, #39769) [Link] (5 responses)

Sometimes finding the answer is hard but checking the answer is easy. (Factoring a large number into its two prime factors is the classic example.) That's probably not the case for training current models, but it might be possible if you allow a probabilistic response. Given a training corpus and a set of weights, can you state with 99% probability that the weights were derived from that corpus? And do so much more quickly than retraining the model from scratch?

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 18:32 UTC (Thu) by Paf (subscriber, #91811) [Link]

That’s a very very interesting question! I have no idea of the answer…

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 18:45 UTC (Thu) by kleptog (subscriber, #1183) [Link] (3 responses)

I'd say that's kind of questionable. If you use 800GB of training data to get a 1GB model, in what sense is any part of the training data actually contained in the model? Compression is not lossless and while there's undoubtably a lot of redundancy in (say) Wikipedia, it's not clear if there's enough redundancy to reproduce any significant part of it from 1/800th of the size.

You can sort of make a guess at how likely it is to output any particular sequence of words but unless it's really unique it doesn't prove anything. The statement "the answer is 42" is now so well known you can't use it to say Douglas Adams was in the training set. If you took 500 books reviews from 500 different people, how much of the book could you reproduce? Is this a problem?

FWIW, I think in the future we'll evolve from the current "the model has the knowledge of the language and knows about stuff" to a common model that understands your questions, and the things it talks about is a separate model which is interchangeable. We just haven't worked out how yet.

Questions about machine-learning models for Fedora

Posted Mar 14, 2024 19:14 UTC (Thu) by Heretic_Blacksheep (guest, #169992) [Link] (1 responses)

In a determinate system the same input will always result in the same output from the same program. It's also the case the reverse is true, because there *are* lossless compression algorithms. Any good archival compression algorithm can go backwards as well as forward in a deterministic fashion. While training the model, given the same program code, hardware (hardware glitches you know), exact same data, you should always get the same statistical model. If you don't, the system is no longer deterministic and accurate auditing becomes improbable. However, the lack of proof is not proof of a negative, so there's no way to know for sure till someone manages a formal proof that you can't go back from the model to the original data. It may just be impractical rather than impossible.

Besides, you're neglecting that it's possible to expose the original training data in most, if not all, models by crafting the correct prompts. This is what prompted the current lawsuits in the US against OpenAI, etc. In those cases, it's not necessary to go backwards from the resultant model as the model itself contains that information already. You don't always need the whole original picture, when you can piece together enough of it to be convincing to your audience. Or, in some cases where the shape of the data, and its metadata, can create a convincing portrait of the original even if some of the details are wrong or missing. One of the problems technologists have is tunnel vision. It's easy to forget that just because digital data is missing, doesn't mean other information can't point to the correct answer. (Such as deleting access logs while forgetting five people can place them in the room at the time of the deed by passing the person in the hall.) Ultimately, you can trace someone answering "42" back to Douglas Adams regardless of where the individual picked up the response no matter whether they read _Hitchhiker's_.

Questions about machine-learning models for Fedora

Posted Mar 15, 2024 22:38 UTC (Fri) by kleptog (subscriber, #1183) [Link]

> While training the model, given the same program code, hardware (hardware glitches you know), exact same data, you should always get the same statistical model.

Many training methods are non-deterministic for various reasons, but one common one is that for any complex deterministic system there exists an input that causes it fail in some catastrophic way. By adding randomness you can ensure it never gets stuck for any execution within this lifetime of the universe. Even quicksort uses randomness for this purpose, even though the output is 100% deterministic.

> In those cases, it's not necessary to go backwards from the resultant model as the model itself contains that information already. You don't always need the whole original picture, when you can piece together enough of it to be convincing to your audience.

But then you need to think about which this is a copyright problem or not. The point of copyright is to protect *copies*, in particular those that affect the market of the original work. So if you can engineer a prompt that produces a specific output, something that is similar to an existing work is not necessarily a copy. The expression is copyrighted, not the ideas. The same ideas in different text is not copyright infringement. And the prompt itself has to not contain information based on the input you are trying to prove the existance of (no "leading the witness" so to speak).

From a pure practical view: if a model can reproduce articles that means there is room to make the model smaller while maintaining fidelity. It's not something model makers actually want so this is just a temporary problem. Hence I think this is a pointless battle whose only effect will be to make lawyers rich. There's already academic research indicating that a large part of the problem is lots of duplicates in the training data, which is simple to remedy. And saves money to boot.

Questions about machine-learning models for Fedora

Posted Mar 15, 2024 8:20 UTC (Fri) by ballombe (subscriber, #9523) [Link]

What about that:
An hypothetical LLM finds the 0.1% of the pages that are statistically more likely to be relevant to query and keep that exactly. The rest is compressed using that as a reference corpus.
That would mean it could only reproduce perfectly 0.1% of the pages, but it will be the 0.1% that 99% of queries ask for.

This is not how LLM work, but it is closer to the truth than 'they keep 0.1% of each pages'.

Questions about machine-learning models for Fedora

Posted Mar 17, 2024 3:03 UTC (Sun) by jmspeex (guest, #51639) [Link] (2 responses)

The part about reproducing the model is way trickier than just the compute power (though that is also an issue). Training DNN models involves a large amount of randomness. The model starts with a random initialization. Unless you freeze the seed, you'll have a completely different models every time. Even once the initialization is done, the batching is done randomly, there's often random noise being injected, and more. In theory, you could still try and reproduce the seed, but there's no guarantee that RNGs from different versions of the toolkit are going to produce the same thing. Nor are they going to do the arithmetic in the same way, which can get things to diverge later on. Reproducible models is at the very least a huge undertaking. Now, there may be ways to check that the data is "compatible" with the model by verifying that the gradient for the final model using the data is small enough for the data to have been used for training. But still it depends on your threat model. It won't catch anyone trying to insert something evil in the model.

Regarding the datasets, I think the rules should depend on whether the model is considered a derivative work of the data. And in turn it would at least make sense to me that the answer would depend on the kind of model. In many cases, huge image generative models have shown they can actually reproduce the input, and that's a problem. OTOH, a small image classification model (this is a cat, this is a dog, ...) can't produce an image and is probably way too small to contain its training data in any usable form. My *opinion* is that shouldn't be considered a derivative of the data.

Questions about machine-learning models for Fedora

Posted Mar 17, 2024 4:04 UTC (Sun) by somlo (subscriber, #92421) [Link] (1 responses)

> OTOH, a small image classification model (this is a cat, this is a dog, ...) can't produce an image and is probably way too small to contain its training data in any usable form. My *opinion* is that shouldn't be considered a derivative of the data.

I'd rephrase that by asking "How lossy would a compression algorithm have to be before the result could no longer be considered a derivative of the original data being lossily compressed?"

Because, when it's all said and done, the weights of a machine learning model are, ultimately, a (lossily compressed) representation of the training data used as input source...

Questions about machine-learning models for Fedora

Posted Mar 18, 2024 7:23 UTC (Mon) by jmspeex (guest, #51639) [Link]

>Because, when it's all said and done, the weights of a machine learning model are, ultimately, a (lossily compressed) representation of the training data used as input source

That assumption is incorrect in general. While there may be some truth to that (but even then it's more complicated) for *some* generative models, it is just not the case for many other classes of ML models. For examples, classification algorithms just learn the fundamental characteristics of patterns they attempt to recognize. In fact, they will try discarding any irrelevant information.

There's also the question how how the models are used in the first place. While an LLM could theoretically output some copyrighted work, the output of an image classifier might be the word "cat" or "dog", not anything actually copyrightable. Similarly, an ASR will just output what was said, which again has no copyright issue. Anything stored in the model would require very clever analysis to extract (assuming there's something to begin with).

Personally, I like to just compare the copyright issue with ML models to what happens with human brains. When you read a book, your brain "absorbs" copyrighted material, but you are not violating copyright just by remembering that book. OTOH, if you start writing a book that reads like that book you read, then there's a problem. Of course, when you start distributing models -- which fortunately we cannot do with brains -- then things get a bit more complicated...