Ingebrigtsen: The End of Gmane?

[Posted July 28, 2016 by jake]

On his blog, Gmane creator and maintainer Lars Magne Ingebrigtsen warns that the email-to-news (and web) gateway may be disappearing soon. The site, which is hosted by his employer, has been under a distributed denial of service (DDoS) attack for the last few weeks, but there are other problems as well. "And now the DDoS stuff, which I have no idea why is happening, but I can only assume that somebody is angry about something. Probably me being a wise ass. So… it’s been 14 years… I’m old now. I almost threw up earlier tonight because I’m so stressed about the situation. I should retire and read comic books and watch films. Oh, and the day job. Work, work, work. Oh, and Gnus. I’m thinking about ending Gmane, at least as a web site. Perhaps continue running the SMTP-to-NNTP bridge? Perhaps not? I don’t want to make 20-30K mailing lists start having bouncing addresses, but I could just funnel all incoming mail to /dev/null, I guess…" The site, which has been relied on by many (including LWN) since it started in 2002, is down now and it appears to be unclear when (or if) it will be back.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 16:11 UTC (Thu) by rsidd (subscriber, #2582) [Link] (6 responses)

I get that this was a very useful service for many lists and for many people. But I always found it annoying when LWN linked to it when the primary archive used some more friendly web interface, such as mailman. If the target audience was nntp users, well, how many of those are around anyway?

The DDOS is another matter -- could be something serious, could be something personal (and if so scary), could be some random script kiddie.

Hope gmane has backups in some form that can be mirrored elsewhere...

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 16:20 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

I use NNTP because I don't want a firehose of email directly into my inbox. Also, the Gmane interface is *way* better than Mailman if you want to go through a thread. Mailman only indents so many times and (I believe) will not link across months, so you need to keep figuring out what is going on. Plus, it show what messages you've already read.

DDoS against FOSS infrastructure has been happening recently (spam accounts/edits on Fedora infrastructure, our company Gitlab instance got hit by Quickbooks spam, and I heard of other stories), so I wouldn't be surprised if this is just another step in that event.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 17:57 UTC (Thu) by broonie (subscriber, #7078) [Link]

The web interface is very widely used for lists, I've personally never seen anything that matches thread.gmane.org in terms of ease of use (with the windowed view of the thread structure in a top pane and the message below).

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 18:36 UTC (Thu) by lacos (guest, #70616) [Link] (3 responses)

> But I always found it annoying when LWN linked to it when the primary archive used some more friendly web interface, such as mailman.

Calling mailman a "more friendly" web UI than the threaded view of Gmane bears witness to such a distorted world view that I can't even begin to address it.

Mailman 2 is workable, but severely limited; Gmane beats it hands down.

Mailman 3 / HyperKitty is both a step backwards relative to Mailman 2, and a new absolute bottom in usability (evidenced by the new Fedora mailing list archives, which are atrocious). It doesn't even make sense to compare Mailman 3 to Gmane.

Gmane is the best mailing list archive on the web. Its ability to locate archived message by message-id (regardless of how many lists the message has been cross-posted to!), its threaded view (which integrates discussions across partially cross-posted mailing lists), its permanent links into the threaded view with a specific message focused, and its machine-friendly raw download URLs will be *hugely* missed.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 19:22 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

I concur. However, donating money to Lars is unlikely to help here: his problem is stress and lack of time, and neither are easily donatable.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 22:35 UTC (Thu) by karkhaz (subscriber, #99844) [Link] (1 responses)

I wonder if donating server time/infrastructure might help, though?

gmane.org is down, but IIRC one of the questions on the FAQ was "what hardware does gmane run on" and the answer was a photograph of a dinky server closet using a few ageing Opterons that was running in Lars' day job office.

Maybe some dedicated hosting might be appreciated? gmane was a fantastic asset to the community, it deserves resilient hosting, and Lars deserves a hand with running it...

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 8:57 UTC (Fri) by peake (guest, #109863) [Link]

Gmane looks like it could benefit greatly from Google's Project Shield[1][2]. It doesn't fit their criteria for a qualifying site exactly, but perhaps Google could be persuaded to make an exception for such a valuable service.

[1] https://projectshield.withgoogle.com/public/
[2] https://jigsaw.google.com/products/project-shield/

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 16:31 UTC (Thu) by Kayden (guest, #89093) [Link] (4 responses)

This will definitely be missed.

The fact that you could just link to http://mid.gmane.org/<some-message-id> was fantastically useful. Have an email from some mailing list and want to send a link to someone? Just look up the message-id, paste it it in, good to go. Just sent a patch with git-send-email and want to post a link to bugzilla? Same. To me, this was a killer feature.

It was neat that it worked for any list, too, regardless of where it was hosted...though honestly, if mailman grew support for MessageID-based archive links for all mailing lists hosted at a particular site, that'd probably work just as well in most cases.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 18:46 UTC (Thu) by lacos (guest, #70616) [Link] (1 responses)

You are absolutely right, but it gets better. If you use URLs of the form

http://news.gmane.org/find-root.php?message_id=<some-m...>

it does the same, but it drops you in the threaded view, with the referenced message focused. You don't just get the message itself, you get all its context all at once. That's a killer feature indeed.

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 15:16 UTC (Fri) by darwish (guest, #102479) [Link]

Exactly. I'll really miss gmane's find-root functionality.

Actually I find gmane's find-root + sensible email-thread GUI the only way to properly read mailing lists.

Mailman for example, even if you chose the "list by thread" option, only shows the thread within the boundary of a month. If the thread passed a month, you're out ot luck :-(

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 18:53 UTC (Thu) by amk (subscriber, #19) [Link]

Would there be some organization that could support its maintenance and operations, like the Linux Foundation, Software in the Public Interest, or archive.org?

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 8:03 UTC (Fri) by pabs (subscriber, #43278) [Link]

Some more services for Message-ID lookup are on Wikipedia:

https://en.wikipedia.org/wiki/Message-ID

If anyone knows of more, please add them.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 16:38 UTC (Thu) by linuxjacques (subscriber, #45768) [Link]

The gmane UI is the best I have seen for reading lists.

I use it all the time (when it's up).

I'll be very sad to see it go.

If it stays up as an NNTP server, I'll probably use that.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 16:44 UTC (Thu) by obrakmann (subscriber, #38108) [Link]

> The site [...] is down now

Note that while the website is down, the NNTP service is still running.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 17:21 UTC (Thu) by smoogen (subscriber, #97) [Link]

I have found gmame to be useful for universal links to various emaiuls and searching across different projects who had a linked conversation. It like all other plumbing of the universe it is usually unloved until it flows over and people forget that the entire system was reliant on one person in a back room somewhere.

From all the other sewage treatment places in our Internet culture.. I can heartily agree with his past statement:

I can’t really recommend the job, though. It sucks.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 23:02 UTC (Thu) by luto (guest, #39314) [Link] (10 responses)

I have no idea whether it would help, but if lwn had a checkbox "donate $2/month to Gmane", I would check it.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 23:10 UTC (Thu) by corbet (editor, #1) [Link] (9 responses)

I've been talking a bit with Lars; money doesn't seem to be the issue. He's really just ready to hand it off. I think we might have found a good prospective home for gmane, but it will take a little while to know for sure.

Ingebrigtsen: The End of Gmane?

Posted Jul 28, 2016 23:45 UTC (Thu) by jcm (subscriber, #18262) [Link]

Let us know if there's anything specific that can be done to help - happy to raise it on the RH end

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 9:23 UTC (Fri) by dborkman (guest, #102857) [Link]

Any chance LWN could continue this at least for the vger.kernel.org mailing lists? Would be awesome ... the threaded view was unbeatable.

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 9:44 UTC (Fri) by mtaht (subscriber, #11087) [Link] (4 responses)

I hope this service survives, and that some more resources get put into making nntp modernized.

recently I tried setting up cnews for myself - because I'd like to have my own copy of everything in light of a dark day where something like this happens, or archive.org gets taken down - and failed.

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 9:52 UTC (Fri) by mtaht (subscriber, #11087) [Link] (2 responses)

And I have a great deal of empathy towards lars's burnout and desire to move on. Volunteerism fails in the face of billions of users, and constant adversity. As we get older, our ability to manage disturbances changes, the dull becomes tedious, and the idea of just sitting on a park bench, like bookends, beckons.

Years ago, I wrote about another guy that had burned out, badly, to whom we owe quite a bit:

http://the-edge.blogspot.de/2003/06/wireless-connection.html

and has since vanished from the Internet. I try to remember that story, and retain my essential optimism, but that often gets hard.

I hope more ways emerge to reward, sustain, and recognize people for selfless contributions to software of all sorts, and the internet long before they burn out on it.

To lars, a hug, and a thanks for keeping gmane going for so long. You have long been a hero of mine.

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 10:20 UTC (Fri) by nix (subscriber, #2304) [Link]

Of course, this is more or less an unpaid USENET administrator's job he's doing -- a central point that all the lawyers think is responsible for all the stuff posted on what is really an archive site -- and we *know* that's a job that burns through people like a forest fire through dry dead wood.

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 13:28 UTC (Fri) by Paf (subscriber, #91811) [Link]

Thanks for that link you gave. That's a really striking story.

Ingebrigtsen: The End of Gmane?

Posted Mar 6, 2018 6:43 UTC (Tue) by ceplm (subscriber, #41334) [Link]

CNEWS are from days when men were real men, women were real women and small furry creatures from Alpha Centauri were real small furry creatures from Alpha Centauri (to quote Douglas Adams). In the current days when we are all soft and useless, go for leafnode (https://gitlab.com/leafnode-2/leafnode-2/blob/master/READ...). It is a way easier to use and works fine for small personal sites.

And (a bit of self-propaganda), if you need mail2news bidirectional gateway, I have fixed pyg (originally from an abandoned Debian package) https://gitlab.com/mcepl/pyg

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 9:51 UTC (Fri) by epa (subscriber, #39769) [Link]

In the early days of Gmane there was talk of peering or federating or whatever it is that you do with NNTP. The main site could keep going as a mail-to-news gateway only and then replicate to other sites which may provide web interfaces.

Ingebrigtsen: The End of Gmane?

Posted Aug 8, 2016 6:40 UTC (Mon) by voltagex (guest, #86296) [Link]

Did Jason Scott ever get involved?

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 11:45 UTC (Fri) by mirabilos (subscriber, #84359) [Link] (1 responses)

Please, please, please, someone, save GMane as it “currently is”. Both NNTP and web interfaces in all façets are important tools, and the archive is, too!

Ingebrigtsen: The End of Gmane?

Posted Aug 4, 2016 15:04 UTC (Thu) by wfp5p (subscriber, #56918) [Link]

Yes, the NNTP service is invaluable to me. I follow a lot of mailing lists through it.

Ingebrigtsen: The End of Gmane?

Posted Jul 29, 2016 18:59 UTC (Fri) by alanjwylie (subscriber, #4794) [Link]

Update from Lars:
If the web site was up, it would be at http://permalink.gmane.org/gmane.discuss/17117

From: Lars Ingebrigtsen <larsi@gnus.org>
Subject: Re: Gmane is perhaps ending
Newsgroups: gmane.discuss
Date: Fri, 29 Jul 2016 13:31:55 +0200

After sleeping on it, I'll be continuing the MTA and the NNTP bits for
the time being, at least.

(And there is interest from some serious parties about taking over the
web bits, and hopefully that'll work out, so that all the permalinks out
there will start working again.)

But since I'm changing the hosting for the service, and getting rid of
all the infrastructure for the web servers, I'll be moving the MTA and
the news server to new servers over the weekend. Expect some NNTP
downtime. Well, probably a few hours, at least...

Ingebrigtsen: The End of Gmane?

Posted Jul 30, 2016 1:53 UTC (Sat) by Computator (guest, #109551) [Link] (14 responses)

I understand that he no longer wants to be in charge of the service, but besides that factor, why not use something like Cloudflare? Is there another factor I am missing?

Ingebrigtsen: The End of Gmane?

Posted Jul 30, 2016 21:26 UTC (Sat) by micka (subscriber, #38720) [Link] (13 responses)

I see at least one reason to never put a site behind cloudflare.
Clouflare is obnoxious with its use of captchas.

Ingebrigtsen: The End of Gmane?

Posted Jul 31, 2016 7:22 UTC (Sun) by niner (subscriber, #26151) [Link] (11 responses)

What's worse? A site that's behind a CAPTCHA or a site that's down?

Ingebrigtsen: The End of Gmane?

Posted Jul 31, 2016 8:39 UTC (Sun) by jwilk (subscriber, #63328) [Link]

The worst is false dichotomy.

Ingebrigtsen: The End of Gmane?

Posted Jul 31, 2016 9:01 UTC (Sun) by micka (subscriber, #38720) [Link] (9 responses)

Well, if it's a site you can't go to because there's a captcha, I suppose it has the same value as a site that's down.

Ingebrigtsen: The End of Gmane?

Posted Aug 1, 2016 9:16 UTC (Mon) by gowen (guest, #23914) [Link] (8 responses)

Don't confuse "can't" with "won't". They are not the same thing.

Ingebrigtsen: The End of Gmane?

Posted Aug 1, 2016 9:32 UTC (Mon) by micka (subscriber, #38720) [Link] (7 responses)

So, I amend to "a site where you won't go because you can't".

Ingebrigtsen: The End of Gmane?

Posted Aug 1, 2016 12:31 UTC (Mon) by gowen (guest, #23914) [Link] (6 responses)

Why can you not go to a site with a Captcha?

Ingebrigtsen: The End of Gmane?

Posted Aug 1, 2016 14:09 UTC (Mon) by micka (subscriber, #38720) [Link] (5 responses)

Either because _I can't solve them_. Seriously, some seem to be made so hard that they guess computer from human because only computers can solve them.
Then the alternative is un-understandable synthetized audio speech that I 100% don't understand because it's english and I'm not english speaker.
Or because some of them require you to use a mouse, thinking of recent recaptcha ones. Then I have to go and fetch a mouse. Where did i put it again? Ah crap, I didn't take it with me.
For this same one, I never understand them. It is a river or it is sea? Must I check this square which is quarter-covered with the sign? Is the leg of the sign part of the sign?
And then you get in loops of captchas that never ends. They should just tell you that you failed.
Or because some of them are not usable by disabled people. Note that for this one, I'm not affected but will be as soon as my eyesight deteriorate, like anybody else.

Ingebrigtsen: The End of Gmane?

Posted Aug 1, 2016 14:49 UTC (Mon) by spaetz (guest, #32870) [Link]

I am glad that I am not the only one too stupid to solve recaptchas. Is this gray blob a lake or a river? Is this sign a house number or an Australian street sign? More than once I've simply given up.

Ingebrigtsen: The End of Gmane?

Posted Aug 2, 2016 5:56 UTC (Tue) by karkhaz (subscriber, #99844) [Link] (3 responses)

> Seriously, some seem to be made so hard that they guess computer from human because only computers can solve them

Related: The Arch Linux forum requires that you paste in the output from a shell command in order to register as a new user. At the moment it is date -u +%V$(uname)|sha512sum|sed 's/\W//g'.

I always assumed that it was to stop the hoards of hand-holdy distro users who sign up to post support questions that Archers can't or won't answer, but that particular command is not actually Arch-specific. Would be nice if it involved pacman or something.

Ingebrigtsen: The End of Gmane?

Posted Aug 3, 2016 13:19 UTC (Wed) by ssam (guest, #46587) [Link] (2 responses)

Pasting from websites is a bad idea
https://thejh.net/misc/website-terminal-copy-paste

Ingebrigtsen: The End of Gmane?

Posted Aug 3, 2016 14:04 UTC (Wed) by micka (subscriber, #38720) [Link] (1 responses)

Well, pasting from a website to a terminal is a bad idea right.
This is pasting from a terminal to a website.

Ingebrigtsen: The End of Gmane?

Posted Aug 5, 2016 7:34 UTC (Fri) by k8to (guest, #15413) [Link]

The registration page[0] asks of the user during registration

. . .What is the output of "date -u +%V$(uname)|sha1sum|sed 's/\W//g'"?

How do you think a typical user will answer this question? They could of course try to type it into the terminal themselves, but the quantity of punctuation and whitespace control requirements will lead to many errors. Most likely they will copy and paste it from this question to their terminal. That is what is being questioned, not the second copy-paste of the answer.

0: https://bbs.archlinux.org/register.php?agree=Agree

Ingebrigtsen: The End of Gmane?

Posted Jul 31, 2016 23:01 UTC (Sun) by Computator (guest, #109551) [Link]

Captchas can be turned off for enterprise users, and very close to it for everyone else.

Ingebrigtsen: The End of Gmane?

Posted Apr 11, 2017 8:46 UTC (Tue) by karkhaz (subscriber, #99844) [Link]

Does anybody have any idea about the status of the ``new'' gmane? I'm gmane terribly, and the old functionality doesn't seem to have been restored yet. And the last blog post is from ages ago.