LWN: Comments on "YaCy: A peer-to-peer search engine"

YaCy: A peer-to-peer search engine

gdt — Thu, 08 Dec 2011 06:01:08 +0000

If the program was made more friendly to ISPs then many would be willing to run the harvester for their network and peered networks. Own and peered traffic can be made free by running the harvester outside of peak usage periods.

The major change would be to bias the DHT based upon BGP attributes (community, AS Path length) so that the traffic on transit links is minimised.

Importantly, although ISPs would be happy to run one harvester they would be unhappy to run a second. So there would need to be some interoperable data or search for other FOSS search engines to use.

It would be well worth the authors' time to design an offering to ISPs and to contact the NANOG organisers to give a presentation.

Many ISPs would be delighted to assist a FOSS search engine purely to make a small breach in the current oligopoly, a market situation which carries risks even for the largest ISPs. ISPs are particularly FOSS-friendly, with a long Unix heritage and Linux heavily used and deeply understood. Many directly assist the FOSS community by running software mirrors.

YaCy: A peer-to-peer search engine

k8to — Wed, 07 Dec 2011 04:23:07 +0000

How does the system intend to defend itself against botnet corruption?

YaCy: A peer-to-peer search engine

alogghe — Fri, 02 Dec 2011 00:02:00 +0000

I know right? Why did we outsource this to google?

YaCy: A peer-to-peer search engine

alogghe — Fri, 02 Dec 2011 00:00:23 +0000

Very nice, that might be useful indeed. Thanks.

Specialized crawlers

jnareb — Thu, 01 Dec 2011 20:13:33 +0000

I wonder how hard would be to write specialized crawler for StackExchange network of sites (of StackOverflow)...

YaCy: A peer-to-peer search engine

pcampe — Thu, 01 Dec 2011 13:52:06 +0000

>I think this looks fascinating as a tool for -memory-, I'd love to have it
>index every page I'VE read.

Zotero might be your friend.

YaCy: A peer-to-peer search engine

sumanah — Thu, 01 Dec 2011 12:01:58 +0000

So YaCy could be the memex?

YaCy: A peer-to-peer search engine

alogghe — Thu, 01 Dec 2011 02:41:51 +0000

Everyone is focused on YaCy as a search engine for -discovery- and how it does or does not succeed at this early stage.

I think this looks fascinating as a tool for -memory-, I'd love to have it index every page I'VE read.

I very frequently waste time going -back- to google for some fact or thing I've already read.

This could be really killer for my personal use in this sense.

I could also see doing more interesting stuff with search that -your immediate peers- have read. Working peers that share your goals and interests. What pages come up for searches of things my workgroup has read? Perhaps integrate this with some active hash tagging and active process that people using the system together use.

Cool project, I think the cynicism about it is sad.