LWN: Comments on "Two visions for the future of sourceware.org"

Two visions for the future of sourceware.org

fuhchee — Mon, 02 Jan 2023 20:10:31 +0000

https://gcc.gnu.org/pipermail/gcc/2022-December/240436.html

Two visions for the future of sourceware.org

ceplm — Sat, 19 Nov 2022 19:07:10 +0000

Bradley Kuhn on http://faif.us/cast/ seems to have quite bitter opinions about 501(c)(6) organizations, it’s probably best to ask him. E.g., http://faif.us/cast/2019/apr/02/0x65/ and others.

Two visions for the future of sourceware.org

paulj — Mon, 03 Oct 2022 09:13:35 +0000

"was this the end goal of some of the participants all along? Transitioning from GNU-managed infrastructure run by people we know and trust to infrastructure run by minions of faceless corporations because we pay them money, with those corporations given explicit governing power over the project?"

Yes.

Not in any organised conspiracy way, but simply as an emergent property of the fact that corporations (i.e., the managers who allocate resources) want some level of control in return for the resources they allocate. And they can be _very_ ruthless about how they go about it. They will hire programmers to work on an open-source project, then work to get control - in back-room dealing kinds of ways (see my other comment on how corporate politics can work).

Just speaking as someone who was at the sharp end of this on another project. And - I'm sorry Jon - but LWN completely uncritically ran the LF press release on that one, without reference to the other side.

Two visions for the future of sourceware.org

Wol — Fri, 30 Sep 2022 21:17:07 +0000

Speaking as a volunteer in the charity sector, my experience is that most people are decent.

The big problem I hit is that the professionals just don't "get" that many of us are elderly and disabled. Okay, that's not true of me but it is true of most of my fellow volunteers. And every now and then they have to be forcibly reminded that we just can't do what they want.

(Then throw into the mix what I see far too often on LWN and other views of America - the *assumption* of bad faith and conspiracy - and you have a toxic mix that sadly makes stories like this commonplace.)

What's that I heard (that I've mentioned here before)? If you assume other people are typically decent, well meaning, and rational - you know, just like you - then when you have disagreements the obvious conclusion is that one of you is mis-informed. And what's the betting it's you!

Cheers,
Wol

Two visions for the future of sourceware.org

paulj — Fri, 30 Sep 2022 19:59:13 +0000

Agreed, that's what would happen.

Scenario I was envisaging is the subversive does /not/ initiate the "break glass" procedures, but just waits patiently for the big "break the glass, all hands on deck to debug this massive outage!" event to happen (which is inevitable, with current culture in many big tech places) - and then capitalise on that.

Will it be picked up? Who knows.

But I know for a fact that in some of these big outages, ad-hoc binaries get built on devs machines and pushed out with dirty scripts, outwith all the CI and stuff, with quick-fixes. (The CI and other normal deployment tools may be broken as part of the outage anyway). And those binaries may end up running for a week or two.

Two visions for the future of sourceware.org

mjg59 — Fri, 30 Sep 2022 19:21:43 +0000

Everywhere I've worked, use of breakglass triggers immediate notifications and pages whoever's on-call to review what's happening and verify that it's associated with a legitimate situation. I think you're right that it's a situation where an attacker is more likely to be able to achieve something without being noticed immediately, but I'd hope that it also increases the probability that it'll be picked up afterwards.

Two visions for the future of sourceware.org

Cyberax — Fri, 30 Sep 2022 19:03:47 +0000

> As these emergencies happen fairly regularly (consequence of the "err on the side of pushing, can always fix later" culture), a good strategy for an attacker would be to prepare whatever internal exploit/subversion, and then just wait for the next emergency and push stuff out unnoticed amid the chaos.

In environments where I worked, emergency access still logs everything. And the logging systems are really separate from everything else (and rarely need maintenance anyway).

Two visions for the future of sourceware.org

pebolle — Fri, 30 Sep 2022 19:01:01 +0000

> The one thing I have seen is that /some/ in the corporate set can be /very/ ruthless about getting their way.

My gut feeling is that some in the non-corporate set can be just at bad. Not restricted by corporate metrics - market share, profits, etc. - some people in non-profit roles, volunteers, etc. show very limited restraint. Many such cases documented on lwn.net.

Two visions for the future of sourceware.org

donbarry — Fri, 30 Sep 2022 17:13:00 +0000

"was this the end goal of some of the participants all along?"

Yes. I think this will wake up many people whose salaries are not tied to the silent actors behind this.

Two visions for the future of sourceware.org

law@redhat.com — Fri, 30 Sep 2022 13:45:17 +0000

Paolo,

You're a respected voice in the broader free software community and many of us in the GCC community absolutely remember your numerous contributions through the years. Definitely chime in to the discussion.

Jeff

Two visions for the future of sourceware.org

paulj — Fri, 30 Sep 2022 13:23:35 +0000

Yes, sure, there'll be a incident review a month later. And all the diffs and commands that people involved thought were relevant to the cause, the diagnosis and the fix will be linked to from the review page in the incident review tool.

Does that mean some subversion hidden somewhere in there will be caught? I don't know.

Does it mean that a subversion that was pushed /outwith the control or visibility of the normal CI tools/ (cause, shit was down! fix fast! maybe even the CI infra was down, cause it depends on the site being up!) will be caught? That's a lot less likely.

A manually pushed out subversion could easily run for weeks, until the next update of whatever component it is in.

Two visions for the future of sourceware.org

zdzichu — Fri, 30 Sep 2022 12:18:39 +0000

It's not really like that. After the incident one is supposed to create a normal report and get it approved. That's the time when you through shell histories, all commits made through emergency, diff the config files etc. Suspicious stuff should be caught at this step.

Two visions for the future of sourceware.org

paulj — Fri, 30 Sep 2022 11:50:49 +0000

The "break glass in case of emergency" thing is interesting.

When this happens, you've likely got engineers running commands all over the place trying to debug a complex, multi-faceted problem. There's no review! "Get the {major system} back!" is the only concern.

As these emergencies happen fairly regularly (consequence of the "err on the side of pushing, can always fix later" culture), a good strategy for an attacker would be to prepare whatever internal exploit/subversion, and then just wait for the next emergency and push stuff out unnoticed amid the chaos.

Two visions for the future of sourceware.org

paulj — Fri, 30 Sep 2022 11:46:02 +0000

I don't think it's even bringing someone into your conspiracy. The review culture is generally perfunctory - cause that follows from the "push first, fix later" mentality - so you can generally always find someone willing to hit the "approve" button.

"generates an audit trail, so if an incident is noticed after the fact, there is a better ability to figure out what happened."

Right, this is the main thing. You can figure out /who/ did something, hopefully. And fire them / hand them to the police / sue them.

Two visions for the future of sourceware.org

foom — Thu, 29 Sep 2022 14:28:14 +0000

Of course it is not a 100% solution. Nothing involving humans can be.

But any sort of intentional wrongdoing is significantly more difficult/risky -- and would also likely take longer -- when you either need to involve multiple appropriately-privileged people in the conspiracy, or else hope your non-conspiring counterpart doesn't notice what you're doing.

And of course, all of this also generates an audit trail, so if an incident is noticed after the fact, there is a better ability to figure out what happened.

There is plenty of history in organizations large and small -- going back even to very early system administrators -- where an individual who has the ability, will abuse their privilege to, e.g., spy on someone they're romantically interested in. (Not to mention nation state hacking operations...)

These sorts of policies can make both of those attacks much trickier to pull off -- and possibly even prevent them.

bbb vs galene

jch — Thu, 29 Sep 2022 12:14:56 +0000

> Folks invest hundreds into their phone and accessories to customize their listening/speaking tool to fit them exactly, don't turn that away. Even (or especially) if they use a browser to view the screenshare.

I would be interested in understanding this comment, but I don't wish to hijack this discussion. Perhaps you could describe the UI you envision by mail, either privately (jch at irif.fr) or, preferably, on the mailing list (galene at lists.galene.org)? (No need to subscribe, I'll manually whitelist you.)

Two visions for the future of sourceware.org

paulj — Thu, 29 Sep 2022 11:58:45 +0000

If you've worked in a sufficiently big organisation, particularly any of the modern big-tech ones with "push first, fix later" (aka "Move fast") cultures, there's generally always some one you can find who will approve changes with cursory or no review.

That culture + heavy reliance on automation there leads to default attitudes of "If the CI passes, push it" .

Two visions for the future of sourceware.org

foom — Wed, 28 Sep 2022 13:09:23 +0000

Non-unilateral access doesn't mean "need to ask for permission from two people before you receive unfettered access to modify and/or extract data from the system". It means that every single individual action you wish to take must be reviewed and approved by someone else before it can be done. Even when you have the highest level of access given to anyone.

Of course, regular deployments are done with automation, which auto-deploys securely and reproducibly built binaries, built from code which has had mandatory enforced code reviews for all changes. So in the regular course of business, you never need to touch the production system.

But stuff happens.

So, want to tweak the configuration of your running service? You need to get a second person to approve the particular config change -- that's enforced.

Want to run an arbitrary shell command on the production system, to fix an emergency problem? Get the actual command-line you're going to execute reviewed by a second person, before it can be executed -- again, enforced.

There will always be a "break glass in case of emergency" access available to a small number of people which bypasses the complex enforcement systems and gives you direct control -- but that should be used so rarely, so as to be actually notable if the glass is indeed broken. And invoking such extraordinary access should still require multi-party concurrence of some sort -- and be logged and audited after the fact.

Two visions for the future of sourceware.org

paulj — Wed, 28 Sep 2022 12:11:21 +0000

Of course, the problem is that in very large organisations, step 5 may be very weak.

E.g., you might have to create a request for a permission in a system, get it approved by a line manager, and then that request goes to another security RBAC-overview team to sign-off. The line manager probably doesn't have great insight into exactly why you might need a permission - you just invent some reason why it's needed for your job. The RBAC security sign-off team also has no great insight - they're dealing with these requests from across the many systems and people in this large organisation, and unless there are clear red-flags they're going to approve a request that has whatever "I need X to do Y for my job" rationale that is signed off by line management.

In some systems in large orgs, some security roles in RBAC systems may be team specific and there may be a requirement that another IC on that team already in that security role has to sign-off (rather than line management; or perhaps in addition, depends). But this is obviously subvertable too.

Some roles have extra-ordinary powers too. E.g. "network root". Even if you could break that role up and make it more fine-grained, end of the day you will still end up with some super-power roles. There's just no avoiding that some kinds of tech management work require super-admin powers, and there'll be no avoiding that in very large tech organisations, the number of people engaged in such work will not be trivial (even if that role scales well and those with super-roles comprise a relatively small ratio of the overall tech staff).

Two visions for the future of sourceware.org

NYKevin — Tue, 27 Sep 2022 17:45:49 +0000

Yes, I agree, which is why I never claimed this was intractable. I claimed it was a process. That process looks like this:

0. Provide a precise operational definition for root-equivalent, as well as "close enough to root-equivalent that we have to lock it down."
1. Find all the root-equivalent (and "close enough") services.
2. Figure out who has unilateral access to them.
3. Figure out who actually needs access (unilateral or otherwise).
4. Set up a system whereby the people who need access can get it when they need it.
5. Make that system non-unilateral (i.e. require a second person to approve and/or supervise the access).
6. Make sure that a second person is actually available when required.
7. Have some sort of contingency plan for when the system breaks.

Every single one of those steps has complicated technical and administrative implications. Realistically, this could easily take a large organization years to churn through, even assuming it's a high priority (many organizations take a checklist/audit approach to security, and will only do this sort of thing in narrow domains where someone is specifically checking their work).

Two visions for the future of sourceware.org

paulj — Tue, 27 Sep 2022 09:47:08 +0000

Thanks for explaining why.

I don't think it was proper to be repeatedly interrupted, even if the reaction was too strong. I assume everyone has learned from this.

Two visions for the future of sourceware.org

mjg59 — Mon, 26 Sep 2022 18:14:16 +0000

Most of these are mitigated by blocking unilateral access - nobody has access to a piece of infrastructure that could be subverted to disable access control without someone else signing off on their need to do it first. Systems may be root-equivalent, but individual users shouldn't be.

Two visions for the future of sourceware.org

NYKevin — Mon, 26 Sep 2022 08:13:18 +0000

Reducing unilateral access in large corporate environments is not an event. It is a process. You cannot simply snap your fingers and say "now nobody has root anymore!" Because of course somebody still has root:

* If you have a computer, and someone has physical access to it, then that person has root on that system. If nobody has physical access to it, then when it breaks, you can't fix it.
* If you have more than one computer, and those computers talk to each other automatically (for any reason), then they need to authenticate each other, which means they need certificates, which means you need a key that signs those certificates. That key is root-equivalent, or very nearly so.
* You can break it up into multiple keys, but now you have to solve for key management, so your key management system is probably root-equivalent.
* If you have any sort of access control (authorization) system, then anyone who can administer that system probably has root (within the context for which that system is authoritative).
* Does your access control system have a storage backend? Who administers that?
* What about the network? Do you trust certain network segments more than others? Could that be exploited to become root?
* And on, and on, and on...

Two visions for the future of sourceware.org

pbonzini — Sat, 24 Sep 2022 12:59:29 +0000

Yeah I noticed that while reviewing the post on overseers, which is a bit more polished. :)

bbb vs galene

WolfWings — Sat, 24 Sep 2022 08:31:53 +0000

The other comment lists the same three I would have, but ever since Asterisk basically became readily installable on most distro's there's been solid audio conferencing options IMHO. It was just a matter of appropriate interface hardware in some cases, but with modern VoIP phone number availability it's far easier now.

And that's one oversight a lot of 'pure web based' attempts has made over the years IMHO is they forget to support POTS dial-in at times. Folks invest hundreds into their phone and accessories to customize their listening/speaking tool to fit them exactly, don't turn that away. Even (or especially) if they use a browser to view the screenshare.

And regarding 'real time subtitling' yeah, TTS, most folks I encounter use the former not the latter term but I recognize the latter is the more accurate technical term to most implementations. But from a UI/UX perspective... it's being able to turn on subtitling. Doesn't matter if it's using TTS on the back-end or not.

bbb vs galene

jch — Sat, 24 Sep 2022 02:43:12 +0000

Galene's JavaScript client library is described here <https://galene.org/README.FRONTEND.html> (full JSDoc here <https://galene.org/galene-api/>). Janus provides something similar or better; I'm not sure about Jitsi Meet.

The more interesting thing, in my opinion, would be for multiple server implementations to agree on a common well-documented protocol, so that a single client can work with multiple servers. There's been some effort in that direction with WHIP <https://www.ietf.org/archive/id/draft-ietf-wish-whip-04.html>, but that's limited to ingress (traffic in the client to server direction only, which is useful for IP cameras and video broadcasting software), and it's too early to say whether it will gain any traction.

bbb vs galene

pabs — Fri, 23 Sep 2022 23:55:50 +0000

It might be worth writing a single protocol library that can then be used by all of the clients, including the web one by compiling to emscripten and or WASM. That would make it even easier to get more clients and a variety of different clients, all with a better level of support for the protocol.

Two visions for the future of sourceware.org

mjw — Fri, 23 Sep 2022 23:42:02 +0000

Thanks for also posting to overseers@ I think this discussion is very relevant and will reply there.

Just wanted to reply to this part here:

> (e.g., adding a public-inbox instance does seem like a good idea)

Done :) https://inbox.sourceware.org/

This was actually part of the original BoF topics, including discussion on how to use b4 for patch attestation and how that might help with the secure supply chain discussions: https://gnu.wildebeest.org/~mark/sourceware/presentation....

Two visions for the future of sourceware.org

mjw — Fri, 23 Sep 2022 23:27:04 +0000

I am not sure whether I was on the "stakeholder map", but somewhere at the start of the year there was some rumors of a GNU Toolchain Infrastructure initiative which was conducted on google meet, with documentation on google docs and to participate in the discussion you had to join a groups.io "list". I obviously objected to all of that. Especially after being asked not to talk about it.

I was told I could talk about the technical aspects of it publicly though. Which is when we started the Sourceware GNU Toolchain Infrastructure roadmaps on overseers (one of which even made it to lwn). We recently dropped the GNU Toolchain part though, since that seemed slightly confusing.

Some of the controversy now seems to come from that, because as far as I understand all of the infrastructure to discuss/participate is still proprietary. And none of our technical roadmaps seem to be part of GTI.

Two visions for the future of sourceware.org

mjw — Fri, 23 Sep 2022 22:57:04 +0000

Sorry about that, clearly I was also confused about the intent of the second spillover BoF, I had assumed it was for spillover if we ran out of time on the GTI topic as the schedule said. We had tried to coordinate, but failed to have any public discussion before the talk.

But Carlos and I do actually agree on most things, we just differ in style and sometimes the path to take to a common goal. So when he proposed to join the BoF I had assumed the other prepared discussion topics that included how to use the new service to do pre-commit ci, testcase analysis, patch attestation for a secure supply chain, contribution tracking, etc. were all of common interest to both him and the whole community attending, so I really wanted to do that discussion first.

I had added the new part about the future of Sourceware and becoming a Conservancy member because I thought it was a nice bridge to Carlos topic. I hadn't intended it to be something controversial or a debate on different visions. Just that having a nonprofit fiscal sponsor for the community without funds seemed like it would be a good bridge to a topic about finding sponsors with funds to spend.

Two visions for the future of sourceware.org

mjw — Fri, 23 Sep 2022 21:25:17 +0000

> "bordered on physical violence at one point." <raises eyebrows>

Yeah, it isn't really that clear in the recording, but I really was at a
loss when someone started shouting and even pushing other participants
to shut up. That had never happened to me before and I didn't really
know how to handle it. So I lost control. What I of course should have done was simply tell that person to calm down and state that there would be plenty of time later for other discussion topics they wanted to bring up. Now it looked like I didn't want them to speak which made the optics even worse.

Two visions for the future of sourceware.org

carlos.odonell — Fri, 23 Sep 2022 20:40:11 +0000

Sorry to disappoint you nix. Please keep following the public proposals and commenting, I appreciate the candid feedback from the outside.

Two visions for the future of sourceware.org

mjw — Fri, 23 Sep 2022 20:38:27 +0000

> Part of me wants to write a long discussion of (c)3/(c)6 distinctions, funding nuance, etc.

If you could that would be really appreciated. Specificaly if you could explain whether it has to be either/or? Since one solution here seems to mix-and-match the different approaches. For example that LF/GTI or OpenSSF as (c)6 funds Sourceware through the Conservancy as (c)3. At least it seems that OpenSSF does something like that already with PyPi. Or maybe the other way around as jsm28 seems to suggest below. That funds for Sourceware are raised through Conservancy which then hires the LF/IT team to provide certain services?

> But most of me is just sad at what sounds like a fairly unhealthy community situation, piled onto an already unhealthy GNU/FSF situation.

I am not denying that it sounds fairly unhealthy right now, but I actually think things are already (very slowly) healing. And I am really happy to see the BBB discussion result in various of these organizations are now exchanging their setups on the sourceware overseers list. If more projects start to offer BBB (or similar) services that would be a great outcome.

Two visions for the future of sourceware.org

jhhaller — Fri, 23 Sep 2022 20:19:50 +0000

There is an interesting dichotomy of how software is acquired from FOSS and commercial sources. Commercial vendors have to respond to a cybersecurity assessment. This includes things like company training for phishing awareness, use of 2 factor authentication, and use of validated toolchains. This starts a chain of wanting to have that provenance go back to FOSS tools used in building that commercial software. Much of this is an attempt to prevent typo-squatting, or attempts to insert malicious code into project sources. While 2FA may not be required for drive-by FOSS contributions, I would not be surprised if there was a desire to require 2FA on code committers. Similar control over FOSS project resources may be desired, and easier to implement in a public cloud.

I don't know what the actual cybersecurity concerns are, but this could be part of the concerns.

bbb vs galene

ballombe — Fri, 23 Sep 2022 19:40:10 +0000

> People like to see faces and hand gestures. The latter are often used by speakers to make a point.

and yet... I have been (virtually) to a large number of video conference where one almost never see the speaker, only their shared computer screen for all the talk.

How is sourceware.org run

smoogen — Fri, 23 Sep 2022 14:31:37 +0000

My previous comments painted things very poorly and were made when I was already grumpy. I should have sat on it for a day and posted it later if I felt them to be useful.

When I originally read the article, I felt that I could map various people's comments to previous discussions. Some things seemed to be like ones on #sourceware, others various discussions on various lists after the Cygnus/Red Hat merger, and even before that in the early 1990's to gnu.misc.discuss threads about when Cygnus was seen as the potential enemy of Free Software and could go evil some day. Trying to do that was a mistake on my part as things are much more complicated, and I did people a disservice.

That said, I still agree with my ending comment. I feel like people were doing these things each from a place of what they thought was best, and trying to avoid various problems they had run into a long lifetime of discussions. It just rarely works out when trying to do that.

bbb vs galene

jch — Fri, 23 Sep 2022 14:25:58 +0000

> Are there any non-web-based clients for these sorts of systems?

In the specific case of Galene, the protocol was designed to be easy to implement in native clients. As a proof of concept, I've written a lightweight Android client for Galene, which you can download at <https://galene.org/galene.apk> (the source code is not available yet, I'm only just learning Android programming and don't feel comfortable developing in the open yet).

As to the other clients, I am aware of a desktop client for Jitsi Meet, but it uses Electron, and I suspect it's essentially a wrapper around the web client. I am not aware of any native clients for Janus.

Two visions for the future of sourceware.org

pbonzini — Fri, 23 Sep 2022 13:32:05 +0000

I wasn't sure because I have not been a contributor to sourceware projects for several years, but I'll accept the offer. :)

Funding from the LF

corbet — Fri, 23 Sep 2022 13:24:15 +0000

Through the end of 2019, LWN received some travel sponsorship from the LF that enabled us to get to events and was much appreciated. For some strange reason we stopped travelling in 2020 and that sponsorship ended; we have received no funds from the LF since that time. So the claim in the above comment is not really true.

We might see if they are willing to renew the travel money at some point, but we have not even asked that question. The steady stream of "I got COVID" reports coming in from the events of the last two weeks has not increased our urgency on that point.

Two visions for the future of sourceware.org

fuhchee — Fri, 23 Sep 2022 10:22:24 +0000

Please bring this stuff up on overseers@.