LWN: Comments on "Development quotes of the week"

Re: Joey Hess

jhhaller — Fri, 08 Jun 2018 21:35:13 +0000

See Note-db, which is how Gerrit stores almost all data in git since release 2.15. Only the reviewed flag is stored in a database, for performance reasons. While Gerrit doesn't have an issue system, it does have user accounts and code review records, all of which are stored in Git. While some information has external indices for performance, the primary data is still in git. Gerrit is used in a number of high-profile projects, including OpenStack, Eclipse, and Android. Not all Gerrit instances have completely migrated from the older database, but support for the external database will be removed in the 3.0 release.

rate limited API

jwilk — Fri, 08 Jun 2018 09:31:22 +0000

You shouldn't back up forks. It wasn't submitted upstream == it doesn't exist.

Also, anybody can make a fork and fill it with garbage. If you're backing up all forks blindly (like github-backup does by default), you're susceptible to DoS.

Re: Joey Hess

rahulsundaram — Thu, 07 Jun 2018 23:54:24 +0000

>To my knowledge aside from FOSSIL, none actually exist that receive any kind of real-world usage (pet projects aside, one of which was mentioned here).

Pagure is sponsored by Red Hat, used extensively by Fedora and was strongly considered by Debian and it works this way.

Re: Joey Hess

excors — Thu, 07 Jun 2018 19:55:04 +0000

Judging by discussions around https://lwn.net/Articles/754779/ on GitHub vs GitLab, projects often prefer GitHub primarily because it's more popular, meaning more potential contributors are familiar with it and already have accounts there, lowering the barriers to them becoming actual contributors. That's what makes GitHub's millions of users so valuable to Microsoft - they will continue to use GitHub, even if it were technically trivial to migrate to another service, and will attract more users, simply because of the network effects that apply to any online service with interaction between users.

https://docs.gitlab.com/ee/user/project/import/github.html makes it look straightforward to copy data from GitHub anyway, so their choice to not implement issue tracking with an awkward and inefficient form of database (i.e. something layered on Git) doesn't seem to be a significant obstacle to getting your data out.

rate limited API

dw — Thu, 07 Jun 2018 19:24:18 +0000

> Finding comments attached to commits is additional API calls (and there was not a way to discover them other than trying every commit in turn last I checked

Seems that wasn't any time recently:

- https://developer.github.com/v3/issues/comments/#list-com...
- https://developer.github.com/v3/issues/events/#list-event...

> 12 hours to check something out is not fertile ground for a distributed ecosystem to develop

To be clear it was 12 hours to sync 40k tickets and then per the APIs above, O(changes) thereafter.

rate limited API

joey — Thu, 07 Jun 2018 18:48:32 +0000

Checking each issue for new comments takes an API call. The API is paginated == more API calls the more issues and comments there are. Finding comments attached to commits is additional API calls (and there was not a way to discover them other than trying every commit in turn last I checked). Repositories can have forks, which can have issues, which can have comments, for piles more API calls. (I know about this in some detail because I've written software to deal with all of it. It's constantly rate limited.)

And then, most developers are not involved with a single software project. It's not uncommon to have dozens of dependencies you want to keep an eye on or are tangentially involved with in your work on a single project. Out of all your other projects.

12 hours to check something out is not fertile ground for a distributed ecosystem to develop: As evidence see the lack of such an ecosystem.

Re: Joey Hess

dw — Thu, 07 Jun 2018 17:53:17 +0000

Finally a cursory search reveals the "rate limited API" permits 5,000 authenticated requests per hour. The largest repo I could find (Ansible) has only 42k tickets all time, or to put it another way, less than 12 hours for an export cron job to complete. This doesn't even slightly count as lock-in.

Development quotes of the week

Cyberax — Thu, 07 Jun 2018 17:47:32 +0000

Why should the data like issues and PR metadata be stored in git? Flat files are awful for these sorts of things.

This data just needs to be accessible for export.

Re: Joey Hess

dw — Thu, 07 Jun 2018 17:42:48 +0000

I find the dig at Microsoft deeply distasteful, they are hardly pioneers of "Git with bag on the side" style project management tools. To my knowledge aside from FOSSIL, none actually exist that receive any kind of real-world usage (pet projects aside, one of which was mentioned here).

The idea of stashing everything in Git is a great one, and of course it occurred to more than just the comment author, however if you explore this space even slightly, it's clear that stuffing everything in a DB *is much simpler* than giving control of a set of complex interlinked structures to end-users who can barely work a Git desktop interface at the best of times, all while somehow keeping a best-in-class project management UI in sync with whatever atrocities the users unleash behind the scenes.

If I use 'git rebase' to modify a far right-leaning issue comment's author to Linus Torvalds, what should show up in the UI? Should I have the right to purge comments from the UI at leisure? Should the purges be tracked historically somewhere? (In a DB? In Git.. again?) Can I do a pull request for a change to a project's tickets? But a pull request is a type of ticket!

Of course the simplest explanation is also the cheapest, most crowdpleasing and most paranoid one, and I'm very sad to see it repeated in LWN.

"Is that data stored in a git repository?"

mattdm — Thu, 07 Jun 2018 14:50:51 +0000

> Is that data stored in a git repository?

If you like this, please consider helping build Pagure, git forge software built around this idea. "Easy fix" tickets here: https://pagure.io/pagure/issues?status=Open&tags=easyfix