GNU/FSF email outage notice (2002-10-16 to 2002-10-18)

On Wednesday, October 16th at 2:43AM EDT, the T1 to the FSF's main
offices went down.  We soon learned that the basement of our office
building had flooded.  The outage was neither the fault of our
equipment, nor our T1 service provider, but rather due to the heavy
rain.  The flood destroyed all of Verizon's equipment in the basement,
leaving everyone in our building with a T1 connection without service.
After much wrangling of various divisions of Verizon on our part,
Verizon finally restored service on Friday, October 18th, at 11:57AM
EDT.

The GNU Project and the Free Software Foundation were without email
service for over 48 hours.  We apologize for Verizon's outage.  Most
of our discussions here at the FSF while we couldn't read email were
about how we might prevent this situation from happening again.
Unfortunately, despite lots of brain-storming, it seems we do not yet
have the resources to prevent this type of outage.

We are asking for your help to keep the FSF's and GNU Project's
services up and running 24/7.  We've identified three main areas where
we need volunteer and donation support to make this happen.

  (0) We are in need of good rack-mountable machines.  We have a good
      colocation facility in the Boston area that donates bandwidth
      and rackspace for the FSF and the GNU Project; however, we are
      only able to put rack-mountable hardware there.  With a
      redundant mail server setup there, it would have been possible
      to continue delivery of mail even with the T1 to our main
      offices down.  Currently we only have three donated
      rack-mountable machines.  They handle savannah.gnu.org,
      www.gnu.org, and ftp.gnu.org, respectively.  Thus, mail.gnu.org
      is a tower case that lives on the "wrong" side of our T1.
 
  (1) We are in need of volunteers to donate their time to work on a
      mail system setup that can scale across multiple machines,
      hopefully throughout the world.  With the help of a few
      volunteers (thanks to Ward Vandewege in particular), we
      implemented an anti-spam system using a combination of
      SpamAssassin and TMDA to reduce the amount of spam sent to GNU
      mailing lists and tag possible spam sent to individual FSF and
      GNU email addresses.  We need expertise and volunteer time to
      build an infrastructure using Free Software that can scale the
      current setup to make use of multiple servers.

      We've had an initial offer from Loic Dachary and his team in
      France to assist, but we will need a large team.  In the end,
      what we'll have is an integration of existing useful Free
      Software that ISPs can use for redundant anti-spam ideas.  We
      have a savannah project called "monty-python" to work on this
      idea.  Please join it if you would like to help and introduce
      yourself on its mailing list, "spam-discuss".  Information on
      subscribing to the mailing list can be found at
      <URL:http://mail.gnu.org/mailman/listinfo/spam-discuss>

  (2) We are in need of redundant Internet connections to our main
      office.  During the outage, only the T1s in the office building
      were down.  If we'd had a backup connection to the Internet
      using a different technology, such as an SDSL connection, our
      mail server almost certainly would have continued to function
      without interruption.  The FSF currently does not have enough
      funds to pay for a redundant connection to our offices; even the
      T1 services we have now are donated.

If you are able to donate rack-mountable hardware, colocation space,
sysadmin/postmaster volunteering time, or additional connectivity for
our main offices, please contact Ravi Khanna at the FSF via
<donate@fsf.org>.

FSF is the primary provider of infrastructure for the GNU Project.  We
have only one system person on staff, and our volunteer ranks are
sadly dwindling, not growing.  We desperately need donations to
continue providing uninterrupted Internet services for the Free
Software Community.

Paul Fisher
Senior System Administrator, Free Software Foundation