The Linux Foundation picks up FRRouting [LWN.net]

The Linux Foundation picks up FRRouting

Posted Apr 4, 2017 17:50 UTC (Tue) by jhoblitt (subscriber, #77733) [Link] (30 responses)

Out of curiosity, what were the main reason(s) for forking?

The Linux Foundation picks up FRRouting

Posted Apr 4, 2017 19:06 UTC (Tue) by jejb (subscriber, #6654) [Link] (18 responses)

They published a slide set justifying it

https://www.slideshare.net/apnic/freerangerouting-a-new-q...

The implied reason is that Quagga development isn't open enough and doesn't respond to community pressure (meaning the people who forked). I don't know how legitimate these reasons are.

The Linux Foundation picks up FRRouting

Posted Apr 4, 2017 19:27 UTC (Tue) by flussence (guest, #85566) [Link] (7 responses)

My own contribution to Quagga (a 1-character documentation fix) took 8 months to get a response and another 4 to get into a release. I'm not sure if that's representative of how they treat everyone.

I don't buy that excuse for forking though. If they value openness and community involvement, why the hell would they go to the Linux Foundation?

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 1:44 UTC (Wed) by mdolan (subscriber, #104340) [Link] (3 responses)

I'm not sure what your concern is with the LF, but unfortunately your experience in contributing to quagga was very similar to others' or even better than others. What they really needed was an open contribution model that allowed scaling the maintainers and the LF was able to assist with setting up a governance model to help. The community setup the project in less than a fourth of the time it took to get your 1-character documentation patch into a release. At a certain point, forking is what makes FOSS work and there are many options for sharing back up into upstream quagga too.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 7:11 UTC (Wed) by paulj (subscriber, #341) [Link] (2 responses)

What rubbish.

It's taken ye since last July to do this fork. It's been 9_ months. Ye wheedled the contact details for my manager, from a colleague of mine who happened to go a Quagga meeting at IETF last July. Then ye _PHONED MY DAMN MANAGER_. Just out of the goodness of your hearts, of course, so HPE could do their "planning".

Under whom did Quagga patch integration shrivel up? It was under _NetDEF_ - or whatever company it is that that is a front for.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 7:18 UTC (Wed) by paulj (subscriber, #341) [Link]

(deliberate use of the plural "you" there; plus, it's still regularly used and perfectly idiomatic in the dialect of English that I speak).

The Linux Foundation picks up FRRouting

Posted Apr 7, 2017 16:02 UTC (Fri) by paulj (subscriber, #341) [Link]

To avoid any misunderstanding, my employer has no interest in any of this.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 6:10 UTC (Wed) by paulj (subscriber, #341) [Link] (2 responses)

When was that?

A huge backlog of stuff built while NetDEF effectively had the reins of Quagga. Which I believe was not entirely by accident.

There has been a high degree of politics at play here.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 21:56 UTC (Wed) by flussence (guest, #85566) [Link] (1 responses)

If my mail client's dates are accurate, this was 6-7 years ago now. I was pretty surprised when the response came back.

The Linux Foundation picks up FRRouting

Posted Apr 6, 2017 4:03 UTC (Thu) by paulj (subscriber, #341) [Link]

Well, you either have an ru., ch., or org. email address then I guess.

I was mostly away then. I was busy with academic stuff from about '09 to late '14. During that period some others did most of the maintenance. The period you mention I think the primary maintainer wasn't being paid, he was over-worked, and he unfortunately didn't try to involve others (which may have been down to there not really being others to help with maintenance).

From about '12 on to late '14 someone else was primary maintainer. They were paid, through an ISC project initially, called "OpenSourceRouting". ISC had some internal issues, which led to them refocusing on core stuff. That maintainer and another ISCer, with some others, setup a non-profit tax-exempt corporate - "NetDEF" - to take over that ISC routing project in late '13.

Unfortunately, they chose to take a strategy of using control over commit access to 'encourage' corporate sponsors to sponsor them. They were also less than open to others about the fact they had some kind of commercial contracting business on the side - and the public record (the email list) is clear that they would try push such work into Quagga, even while there was a backlog of regular contributions.

I was finishing up my academic stuff in '14-'15, and contracted with them from late Sept. '14 to begin May '15. I left them because of differences on the above. From about Sept. '15 onward patch integration greatly increased. The backlog of patches was dealt with by about summer '16. Patches, whether submitted by email or bugzilla, are dealt with promptly - either integrated or returned with queries or comments.

When NetDEF lost their maintainers, they - as per other long comment - agitated for a fork behind the scenes, since about May '15. Assisted by Cumulus, who have their own agenda. They've got their fork. Now they're spreading somewhat alternative facts about what happened.

The facts are there though, in the commit history and the email lists.

The Linux Foundation picks up FRRouting

Posted Apr 4, 2017 21:03 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

Quagga is... strange. Its tools are designed to be Cisco lookalikes and are extremely clumsy. Its source code is not exactly a mess, but definitely not a nice read.

Personally, I'd switched the systems I maintained to Bird ( http://bird.network.cz/ ).

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 8:40 UTC (Wed) by NAR (subscriber, #1313) [Link]

A long time ago, when all of that was called zebra, I used it. To be Cisco-lookalike helped the learning curve. Also there was a perl module which was designed to configure Cisco routers remotely - that worked with zebra too.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 8:54 UTC (Wed) by excors (subscriber, #95769) [Link] (3 responses)

I made some small but fairly intrusive changes to Quagga many years ago for a research project and, as far as I remember, the code seemed basically alright to me. Not excellent but not bad either. It was fairly easy to read and to change, though with a lot of annoyingly repetitive string-parsing code for the Cisco-like configuration syntax, and quite a lot of code duplication between protocols. It looked not particularly buggy, but probably buggier than you'd like for a vital infrastructure service that talks to strangers over the internet. (E.g. there was the time when an attempt in Egypt to demonstrate that 32-bit AS numbers wouldn't break the internet broke (some of) the internet by crashing Quagga routers around the world with a buffer overflow bug, where it was trying to do clever dynamic buffer allocation and got it wrong.)

(I never looked at BIRD much, but it did seem to have better abstractions that let it share a lot more code between protocols, and a less ad-hoc configuration syntax, though I don't have any practical experience to know whether its vaguely C-like filter language is nice and powerful or just becomes a horrid mess when used by network operators.)

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 13:03 UTC (Wed) by paulj (subscriber, #341) [Link] (1 responses)

My bug, probably, sorry. It wasn't a buffer-overflow though.

Also, we try insulate all network parsing code from direct memory access, by making the parsing code interact with network supplied buffers only through a bounded buffer abstraction, which checks all access. Any unbounded access through that bounded buffer results in an emergency stop / assert. Still a security issue, but a DoS rather than arbitrary memory and remote code.

We do have some legacy bits of code doing direct pointer twiddling parsing on untrusted input, also sometimes we get contributors - who perhaps have never had the pleasure of having to scramble at unsociable hours to deal with security issues - add code with direct-pointer-twiddling parsers.

Generally though, Quagga tries to make sure all network parsing code goes through a safe, checked, bounded buffer. (Something inherited from GNU Zebra - Kunihiro Ishiguro is wise).

The Linux Foundation picks up FRRouting

Posted Apr 6, 2017 4:40 UTC (Thu) by paulj (subscriber, #341) [Link]

Oh, much of the hard work of AS4 support was by Juergen Kammer. Most of the bugs would have been due my changes.

The Linux Foundation picks up FRRouting

Posted Apr 6, 2017 19:14 UTC (Thu) by Lennie (subscriber, #49641) [Link]

Many years ago when tasked with choosing a server based BGP-daemon I went for OpenBGPd on OpenBSD. It has many good qualities for that and if I remember correctly, was much more memory efficient than Zebra was at the time. The configuration file format is easy to learn too.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 15:12 UTC (Wed) by paulj (subscriber, #341) [Link]

BIRD is neat. It's very fast on lower numbers of routes. Though, I think Quagga scales better with increasing routes. It has some interesting optimisations it can pull with multi-RIB mode. If I had more time I'd try make BIRD more inter-operable with Quagga.

Kudos to the BIRD people.

The Linux Foundation picks up FRRouting

Posted Apr 4, 2017 21:55 UTC (Tue) by jhoblitt (subscriber, #77733) [Link] (2 responses)

It looks like Quagga is using the patches via smtp model, which, IMHO, is pretty painful vs. a pull-request model with threaded review comments.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 6:12 UTC (Wed) by paulj (subscriber, #341) [Link]

Yes, that used to be the case. A previous maintainer preferred that.

You may also post a patch to Bugzilla (better). If you submit patches regularly, you will be given commit access and asked to push your changes to a review branch.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 15:56 UTC (Wed) by lsl (subscriber, #86508) [Link]

Threaded review comments? Isn't that exactly what you get with email patches?

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 1:38 UTC (Wed) by mdolan (subscriber, #104340) [Link] (5 responses)

There are a few thousand patches, including high severity security patches, that were not being accepted or even looked at by the sole maintainer of quagga. Companies were maintaining all the patches on their own, each duplicating effort. This is where GPLv2+ licensing just isn't enough - everyone had everyone else's code but the governance of the project needed to open up to scale with additional maintainers.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 5:37 UTC (Wed) by pabs (subscriber, #43278) [Link]

Are there CVEs for those security issues?

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 6:17 UTC (Wed) by paulj (subscriber, #341) [Link] (1 responses)

Which high severity security patches? And under which maintainer?

It's worth noting that patch integration shrivelled up, and a huge backlog built up, under NetDEFs' stewardship[1].

There has been a huge amount of corporate gamesmanship and politicking in this fork, do note.

1. Or whatever company it is the people associated with it work for. They are extremely opaque about their business, and the person they claimed in early '14 they had working on Quagga never worked for NetDEF it seems. I'm the only person who ever worked on Quagga for NetDEF, for about 6 months on a part-time basis, going by their public Form 990s.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 6:55 UTC (Wed) by paulj (subscriber, #341) [Link]

Worked on maintaining Quagga, that is.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 7:03 UTC (Wed) by paulj (subscriber, #341) [Link]

Oh, and the existence of a few thousand patches is meaningless. Half of the patches could be obviously good. Of the other half, half again might be going in the right direction, but doing things in the wrong way, fixing things in the wrong place (band-aiding over problems, etc.). The remained may have serious architectural issues, or just be plain inappropriate or even wrong.

To figure that stuff out requires actually engaging in review. Which is something, e.g., Cumulus _refused_, repeatedly, to do.

So yeah, few thousand patches. Means nothing. Till you sort the good stuff from the hundreds-of-monkeys-with-typewriters stuff.

And I'm not the only Quagga hacker who thinks Cumulus' high degree of self-belief isn't always justified.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 17:21 UTC (Wed) by paulj (subscriber, #341) [Link]

Oh, by "the sole maintainer", you must mean the period where NetDEF claimed to be funding the sole active maintainer (except, go look at NetDEFs' Form 990s - they were not; the maintainer they claimed they were funding was working on other, commercial contracting work).

In that period, NetDEF allowed a huge backlog of patches to build up. They later tried to get patches for experimental rev-0 draft stuff into Quagga, insufficiently modularised that would have caused a huge amount of rejiggling of other people's patches - which they had allowed to pile up.

Quagga will recover from what these people have done.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 6:50 UTC (Wed) by paulj (subscriber, #341) [Link] (1 responses)

The reasons come down:

1. The people who present themselves as "NetDEF"[1] having tried to "monetise" Quagga, when - partly by accident (I was away for 4 or 5 years doing academic stuff), partly by design - they controlled the one active maintainer, by seeking "sponsorship" to help with maintenance. Of course, no sponsorship, and your patches might just - completely accidentally - just not get looked at.

We can see this pattern just from the commit statistics.

When NetDEF lost their maintainers, because of these games (one was suspended for unilaterally removing another non-NetDEF maintainer, and for other unacceptable behaviour, asked to explain and just left; the other - myself - left soon after due to these things), NetDEF suddenly started agitating behind the scenes for a fork. The continuing maintainers tried to reconcile with them and their suspended/left maintainer (who never answered the suspension charges).

However, they continued to agitate - they had lost their business model. When this FRR fork was floated they wanted some kind of bronze/silver/gold sponsorship model. I guess this is how they get it.

2. Along came Cumulus. Cumulus are a startup. They don't really have time for others to question their patches. Plus, they know everything - who _are_ others to question their patches, really? Also, they came along right when NetDEF had control. At some stage I guess Cumulus did not sponsor NetDEF and, completely co-incidentally, Cumulus' patches started to pile-up (just look at the commit stats).

Now, at some point - I don't know exactly when - Cumulus did start sponsoring NetDEF. Unfortunately, this was possibly just after NetDEF lost their maintainers. Which I guess was a bit annoying to Cumulus. By this stage, Cumulus patches were going in at a rate like never before, and the ones not going in were being reviewed and comments sent to them.

However, see above - Cumulus don't like dealing with patch reviews. They would just resubmit a patch train, with the same patches with queries outstanding, included - effectively just ignored review - over and over again. Rather than have the developers of the changes manage the upstreaming of their own patches, they gated that work through one person. They seem to have been using some SCM internally with a major impedance mismatch to git (our upstream git anyway). The "Author" field on resubmitted patches would change from patch train to patch train. They submitted 3rd parties patches with the author field set to themselves (we've had immense grief from 3rd parties over Author field in the past, so I'm sensitive to that now).

Cumulus and NetDEF then decided to rewrite the constitutional document of Quagga. Mostly it was was high-level, unobjectionable stuff, except all decisions to be made by Google Hangouts meetings, by majority vote, and to be final.

They refused to discuss or negotiate on any of this. Votes were by company effectively (shows the thinking) and companies that had _never_ contributed to company just _happened_ to pop-up on the conference call where Cumulus asked people to vote on this. A board member of NetDEF works for the company that just magically appeared on that call.

---

Essentially, the cause of the split is that corporates wanted control of Quagga.

Corporates who I don't believe have much feel for free software communities.

Good luck to them.

1. Or whatever entities they actually work for. They are extremely opaque about things. Only 1 of their people seems to work for NetDEF. I am fairly sure there is another corporate involved.

The Linux Foundation picks up FRRouting

Posted Apr 5, 2017 12:54 UTC (Wed) by paulj (subscriber, #341) [Link]

Oh, and to be clear, there is absolutely nothing wrong with corporate involvement in Quagga. Positively encouraged even (we once maintained a "commercial" page on the website, until another maintainer unilaterally removed it). There is nothing wrong with contributors advancing a corporate interest either, e.g. their employer's, indeed that's expected.

As long it is done a reasonably open, mutually beneficial, and "win win" manner, where people show respect for each other and work in a pro-social way.

The Linux Foundation picks up FRRouting

Posted Apr 16, 2017 5:26 UTC (Sun) by jch (guest, #51929) [Link] (2 responses)

> Out of curiosity, what were the main reason(s) for forking?

It is my understanding that there were a number of long-term conflicts within the Quagga project. At some point, the situation became poisonous, and everyone I know joined FRR, with just the lone maintainer remaining in the original Quagga project.

I wish FRR all the best, but a number of people who had attempted to work with Quagga have moved on to other things, and since the open source routing community is not that large to start with, it might not be easy to rebuild a friendly crowd around FRR. I think it's a terrible waste.

The Linux Foundation picks up FRRouting

Posted Apr 21, 2017 20:51 UTC (Fri) by paulj (subscriber, #341) [Link]

Going by the name, you are one of the people responsible for the poison.

The Linux Foundation picks up FRRouting

Posted Apr 22, 2017 5:13 UTC (Sat) by paulj (subscriber, #341) [Link]

BTW, tell me this: Which of the people who've joined FRR are not sponsored by, contracting with, or customers of Cumulus?

The Linux Foundation picks up FRRouting

Posted Apr 26, 2017 17:18 UTC (Wed) by raven667 (subscriber, #5198) [Link]

I hope this is a good thing, I'm not an expert on the Quagga community but I am tangentially involved because of work in network applications and it seems that there are a log of vendors quietly building products on top of Quagga, including Cumulus, some doing bug fixes, maintenance, feature enhancements, etc. that are kept in private forks, so that the "base" project is always behind the competing forks. Getting this into the Linux Foundation vendor consortium might provide some incentive for these proprietary competitors to cooperate instead on a central project and cut down on some of the forks, making mainline Quagga better in the process.

The Linux Foundation picks up FRRouting

Posted Oct 28, 2017 15:44 UTC (Sat) by paulj (subscriber, #341) [Link]

Addendum, referring to not long after the above thread.