LWN: Comments on "Fighting small bugs"

My personal paper cut

bronson — Thu, 30 Jul 2009 16:27:20 +0000

haha, this is brilliant! And obvious -- in all these years of pain (papercut-quality pain) I'm surprised I didn't think of this.

Good call nye.

My personal paper cut

nye — Thu, 30 Jul 2009 15:22:51 +0000

My 'solution' is, whenever Firefox pops up a dialogue asking how to open a file, to choose '/usr/bin/xdg-open'. This allows me to use whatever file handler I define in my DE, neatly bypassing Firefox's hilariously bad 'open with' system.

Fighting massive data loss bugs

Cato — Sat, 25 Jul 2009 20:22:13 +0000

Now that I've rebuilt the PC... I actually lost the contents of two filesystems (root and backup), each on a separate physical disk, and hosted on LVM logical volumes within separate volume groups. There weren't any physical disk errors, or noticeable errors relating to PATA/SATA cables, memory, etc. The only common factor is that both LVM VGs were, well, handled by LVM. It's also suspicious that only one LVM FS was uncorrupted, plus all of the non-LVM FSs.

I suspect that some combination of disk write caching plus LVM and possibly ext3 caused these problems. At least some of the problem was purely at the LVM level, since I couldn't even access the VGs on the backup disk, and got LVM errors.

In the hope that it helps someone else:

- To help avoid integrity problems in future, I used the ext3 'data=journal,barriers=1' options in fstab, and also used tune2fs to set the journal_data option on the root FS (only way that worked for root). I also disabled disk level write caching with "hdparm -W0 /dev/sdX' on both hard disks. This will have some performance cost but this PC is ridiculously fast for light email and web surfing anyway.

- I've dropped SpiderOak for online backup - it didn't back up most of the files (on two PCs, in different ways), generated a corrupt ZIP file on recovering some files via web interface, and the GUI client got stuck recovering files, and generally makes it hard to track backups/restores.

- I have implemented local backups with rsnapshot, which is really outstanding for multi-verson rsync based, and will extend this for online backups, possibly using DAR to encrypt and compress for remote backups.

- Sbackup (Simple Backup) is great for really quick backup setup (literally 2 minutes to install, configure and have first backup running), but I wouldn't rely on that alone.

Also, if you haven't used etckeeper before, it's worth a try - version control for the whole of /etc using git, hg, bzr, or darcs, and also tracks APT package installs that generate /etc changes. Great if you need to replicate some or all of the setup at a later date.

My personal paper cut

cortana — Fri, 24 Jul 2009 23:06:32 +0000

The problem is that you are an expert, whereas most users--for instance, my parents are not. :)

They will be presented with a file selection dialog box and have no idea that they are supposed to go to /usr/bin, and then wait for Firefox to unfreeze, and then pick one of thousands of similarly named items that have absolutely no connection with what they want to do.

For example, opening a PDF document... how is a normal person supposed to know to select evince? :(

As for the freedesktop.org spec... I actually mis-spoke (typed?) earlier. The spec I mentioned allows programs to declare MIME types (that is, provide a mapping from MIME type to human-readable description). It serves a similar purpose to /etc/mime.types, except that it is more modular (it allows applications to define new MIME types) and it allows for the MIME types to have human-readable descriptions, localized to many different languages.

The spec I should have mentioned is the Desktop Entry spec; this is where applications ship .desktop files (in /usr/share/desktop and other places) that specify (among other things) which MIME types an application may handle. It is similar in purpose to the mailcap mechanism, but again it is more modular and allows internationalization, as well as desktop integration (e.g., application menu entries are derived from the .desktop files).

So, Firefox should be reading these .desktop files and offering the user's default handler for a file, along with a selection of other applications that declare that they handle the MIME type. On my system:

$ grep application/pdf /usr/share/applications/mimeinfo.cache
application/pdf=evince.desktop;gimp.desktop;

My personal paper cut

nix — Fri, 24 Jul 2009 22:08:17 +0000

Aha. I just installed KDE4 on that machine (in parallel with KDE3, OK,
yes, I'm reaching):

nix@hades 3 /home/nix% ls -l /usr/bin | wc -l
4102

(is this the single most pointless contest that has ever been carried out
on LWN? I bet I have more symlinks in /usr/bin than you: 4099...)

My personal paper cut

jimparis — Fri, 24 Jul 2009 21:18:48 +0000

> (This is on a system with 3977 binaries in /usr/bin.)

Now it's a challenge :)

Finally!

My personal paper cut

nix — Fri, 24 Jul 2009 20:50:08 +0000

Whatever FF is doing that makes it freeze, it's not statting:

nix@hades 40 /home/nix% /usr/bin/time stat /usr/bin/* >/dev/null
0.21user 0.14system 0:00.78elapsed 46%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (1major+335minor)pagefaults 0swaps

Maybe it's running file(1) or libmagic on every single one? That could
look like a freeze if you didn't notice the disk pounding away:

nix@hades 41 /home/nix% /usr/bin/time file /usr/bin/* >/dev/null
0.17user 0.58system 0:26.16elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+735minor)pagefaults 0swaps

(This is on a system with 3977 binaries in /usr/bin.)

My personal paper cut

jimparis — Fri, 24 Jul 2009 16:52:29 +0000

By the way, that's not to say Firefox can't use improvements in this area. For example, bug https://bugzilla.mozilla.org/show_bug.cgi?id=83305 has been an occasional issue for me for many years now -- with upstream's response being basically "blah, I can't be bothered to fix this". One of these days I'll be motivated to fix it myself, but my brief glances into the Firefox code always scare me...

My personal paper cut

jimparis — Fri, 24 Jul 2009 16:49:13 +0000

Forgive me if I'm dense, but I'm still not seeing what Firefox is doing wrong here. As far as I can tell, it's following the exact same method as everyone else -- it uses /etc/mime.types to map mime-types to file extensions, and it uses /etc/mailcap and ~/.mailcap to choose a default application. If you want to change the default application, then that's the only time you need to open up the file browser -- and I rarely use it to actually browse /usr/bin, I just type the desired executable name directly.

(Incidentally, the Freedesktop.org spec only seems to cover mime types, not mailcap. I don't see which of their specs covers default applications?)

My personal paper cut

cortana — Fri, 24 Jul 2009 16:30:24 +0000

Having to send a file browser to /usr/bin to find an executable to handle a file type is insane. There are 3240 files in that directory on my system. How a non-expert user is supposed to cope is beyond me--if they don't give up after Firefox 'crashes' (freezes while stat'ing all the files in there) first.

Compare with the user experience in Epiphany, which actually makes an effort to integrate with the GNOME desktop; the user is presented with the option to open the file with their default handler, or save it to disk. The default handler is determined via the freedesktop.org MIME spec, which is the same thing used by everything else in GNOME. An administrator may change that for a particular user, or all users, using the standard Unixoid methods outlined in the spec.

Where Epiphany falls down is in letting the user change their preferred handler for a file type; there is no UI for that yet. So the user would have to save the file, then change the handler for it in Nautilus. Eventually, hopefully Epiphany will be enhanced to ask the MIME database who could handle the MIME type it's been served, and give the user a nice list of all the programs that can handle it, in the same way that Nautilus does. Nothing's perfect, eh? :)

Fighting massive data loss bugs

Cato — Thu, 23 Jul 2009 23:27:46 +0000

In this case there was no separate /var FS, and the updates to /var/log/messages have persisted across at least one reboot. So somehow the root FS was mounted readonly (I got errors on trying to write to files so it really was readonly), yet the log files were being updated on /var ...

Fighting massive data loss bugs

dlang — Thu, 23 Jul 2009 22:56:54 +0000

is /var a separate mount? if so it should keep going even if / gets remounted ro

if the underlying device becomes ro, the OS can buffer writes in ram that it wants to get to the filesystem, but can't because it's ro.

this causes more lost data, but not more corruption.

Fighting massive data loss bugs

Cato — Thu, 23 Jul 2009 22:54:06 +0000

I've now had a look at the machine - there are two disks which have mostly LVMed partitions, and one disk is showing classic signs of LVM errors resulting in ext3 corruption (not the one with the root FS, but one with several LVMs for local backup). I used SystemRescueCD initially and the LVM commands showed the LVM state was quite messed up, plus log messages like this:

Jul 23 19:06:57 sysresccd attempt to access beyond end of device
Jul 23 19:06:57 sysresccd sda: rw=0, want=198724030, limit=66055248
Jul 23 19:07:20 sysresccd attempt to access beyond end of device
Jul 23 19:07:20 sysresccd sda: rw=0, want=198723798, limit=66055248

One weird thing is that the LVM on the main disk that hosted the root FS didn't show any LVM related errors, but that was the one with the major corruption. Of course the backup LVM had major corruption but I wasn't focusing on that. In fact a generally odd thing is that the Ubuntu logs didn't show any errors on either disk (i.e. ext3 or LVM type errors), apart from the 'FS remounted' one, yet SystemRescueCD showed them right away.

Another weird thing is that despite the root FS being remounted read-only, the logs in /var were still being written to for 10 days after the first root corruption - surely this is a bug as it can only increase FS corruption.

I haven't yet run a memory test but the system doesn't show any other signs of bad RAM such as randomly crashing applications. The logs also don't show any disk hardware errors.

Anyway, the lesson is simple: never, ever use LVM again. Gparted is pretty good these days for resizing/moving partitions, and the time I have saved on LVM is far less than the hassle of this recovery exercise.

Sorry for going so far off topic, but perhaps LWN would like to write a piece on data loss bugs and how best the community should address them - maybe starting with LVM...

Fighting massive data loss bugs

tialaramex — Thu, 23 Jul 2009 21:42:09 +0000

My #1 guess would be failing RAM. DIMMs don't often go bad, but it does happen and there is nothing in most PCs that will detect it, you just start to see the wrong bits, and of course most of those bits are either coming from or going to files, so it's easy to blame the filesystem.

Filesystem bugs are like any other bugs, they tend to be repeatable, they do something stupid and wrong but not entirely ridiculous (e.g. they don't flip a few bits in the middle of a file, but overwrite an entire block with something else) and so on. If you see weird problems, and especially if you see problems that don't have any clear pattern, that's _much_ more likely to be bad RAM.

My personal paper cut

jimparis — Thu, 23 Jul 2009 18:10:24 +0000

I also did not have "GSM file" on that list, until I clicked one for the first time. And the behavior of clicking on a file (that is not on that list) is to pop up that dialog box where you pick an application, in my experience.

So your problem is that nothing at all happens -- you click the file and it behaves as if you didn't click the file?

Which means we're back to my previous questions -- do you have something like mozplugger installed (check about:plugins), or do you have an action defined in one of the mailcap files (grep gsm /etc/mailcap ~/.mailcap)?

(Are you actually interested in fixing this or just complaining about it? I'm trying to help but you're really not giving enough information for me to provide any useful advice).

Fighting massive data loss bugs

Baylink — Thu, 23 Jul 2009 18:07:50 +0000

I suppose it is, but personally, the only unusual thing I want / and /boot on is "real" hardware RAID 1. When things start to go to hell, the fewer distractions you have, the better off you are.

I'm a sysadmin; I'm paid to be paranoid.

My personal paper cut

Baylink — Thu, 23 Jul 2009 17:58:52 +0000

The problem is "scroll down to GSM file".

I *very often* (nearly always) *do not have* GSM file in that list, on a fresh install of Firefox. And if the type of file you have isn't in the list and you aren't presented with the "open with" option, then you're screwed.

I know that's not supposed to happen, but it does, and not infrequently.

Additionally, the dialog of which you speak -- and this complaint is closer in spirit to the sort of thing they're actually looking for --

1) has 2 columns, with an immovable divider
2) cannot be resized
3) lists the human readable name of the filetype (which is almost always useless) followed by the mimetype (which is very often useless, and is often clipped off partially or completely because of 1 and 2) and does not show the only thing you can actually see -- the file extension -- at all.

My personal paper cut

jimparis — Thu, 23 Jul 2009 17:48:45 +0000

I'm not sure what you mean by the "add" button. I can go into Edit -> Preferences -> Applications, scroll down to "GSM file", and change the Action to anything, including setting it back to "Always ask".

I'm still unclear on what your actual problem is, besides "sometimes it doesn't work". What exactly, on your system, happens when you click on a GSM file? Does it open with an application that you don't like, does it only offer to save to disk, or does it not do anything? Do you have something like mozplugger installed that is taking over the association? Do you have an audio/x-gsm entry in your .mailcap file or in /etc/mailcap?

My personal paper cut

Baylink — Thu, 23 Jul 2009 17:41:12 +0000

The problem I'm having is that, um, it doesn't do that.

That's what *I'd* expect, too. Sometimes, with some point releases, one some platforms, that's what I get. Sometimes not. Doesn't seem deterministic.

And since the very simple "Add" button is missing from whatever version of that association mapping dialogue you have, there's no manual way to fix it if that happens to you... as it happened to us.

My personal paper cut

jimparis — Thu, 23 Jul 2009 17:35:35 +0000

Hmm?
Seems easy.
First I run:
$ sudo apt-get install sox libsox-fmt-gsm
Then I find some random website:
http://fox-den.com/ASTERISK/sounds/2005-07-30-new/gsm/
I click on a file.
It pops up a dialog: "What should Iceweasel do with this file?"
I select "browse"
I type "/usr/bin/play"
I click "Do this automatically for files like this from now on"
I hit OK
I hear the sound.
I click another file.
I hear the sound.
What problem are you having?

Fighting massive data loss bugs

Cato — Thu, 23 Jul 2009 16:55:07 +0000

Fair point, but isn't LVM supposed to be production quality these days? This particular corruption doesn't have the indicators of LVM being involved (such as writing beyond the end of the volume.) Mostly I thought by avoiding RAID and LVM snapshots it was safe enough. However I will remove LVM on the recovered machine, mostly so I can enable ext3 barriers which LVM doesn't permit.

Most of the FS corruption reports I've seen don't mention LVM, so I suspect a bug elsewhere. One report had repeatable corruption on VMware VMs with Ubuntu 8.04 guests, for example, and other corruption reports are on ext3, XFS, JFS, etc, with common factor being an Intel chipset. Quite a few are probably due to dodgy hardware, which makes this hard to pin down. In fact for all I know this is due to bad RAM or a hard disk problem that doesn't appear in the system logs.

Assuming LVM is stable, it's quite easy to recover an LVM machine these days - Knoppix, SystemRescueCD and others support LVM. It's only if there's disk corruption that LVM makes things harder, but that's what backups are for.

I am discovering that SpiderOak is not as good at recovery as it should be - client doesn't work on one machine, and the web download feature generates an invalid ZIP for one directory... So I'm open to recommendations for inexpensive online backup for Linux machines that don't involve rolling your own (I've already done that and want something easier to maintain - but I may go with rsnapshot in future just to avoid the hassles of backup services that don't quite work the way they should).

My personal paper cut

Baylink — Thu, 23 Jul 2009 14:06:13 +0000

is the complete lack of user-level (or even administrator-level) manageability of file associations in Firefox... all the way up to 3.5.

Go ahead: make your firefox use some program to play .gsm audio files; I dare you.

Fighting massive data loss bugs

Baylink — Thu, 23 Jul 2009 14:04:36 +0000

I could ask why the root FS was on an LVM, but I probably shouldn't. :-)

Fighting small bugs

jake — Thu, 23 Jul 2009 13:14:11 +0000

> Yeah, I'd like to have my C back, please :-)

Fixed now, sorry about that!

jake

Fighting small bugs

mclasen@redhat.com — Thu, 23 Jul 2009 13:08:30 +0000

Yeah, I'd like to have my C back, please :-)

Fighting massive data loss bugs

mjthayer — Thu, 23 Jul 2009 11:09:45 +0000

Slightly (but only slightly) off-topic, I feel that it is a shame that people suggesting ways non-programmers can contribute to free software don't emphasise this sort of collaboration enough. It can be done with minimal (though not non-existent) technical skills and no knowledge of programming, and I'm pretty sure that seeing bugs fixed as a result of this sort of work is the sort of rewarding experience that makes people want to use free software.

Fighting massive data loss bugs

mjthayer — Thu, 23 Jul 2009 11:06:00 +0000

Quite agree, although I don't see a contradiction (in my own experience, the "paper cut" sort of bugs are the ones that you fix while you are taking a break from tracking down the serious ones). My own priority here would be, on the one hand bugs with serious consequences (see your example), and on the other, easily fixed bugs which have a reasonable chance of being noticed by more than one or two people. Again, my experience is that you can often quickly tell the last category when a group of users start collaborating over the bugtracker and successfully isolate the bug enough to make it trivial to fix.

Fighting massive data loss bugs

Cato — Thu, 23 Jul 2009 07:16:46 +0000

Correction: the Ubuntu Jaunty data loss bug mentioned in last paragraph is not ext3 related - most likely it's in the ata_piix module.

Fighting massive data loss bugs

Cato — Thu, 23 Jul 2009 06:59:28 +0000

This is important stuff, and well worth supporting. Right now, however, I'm far more concerned with the really huge bugs - the ones that lose large amounts of data. Here's a recent example...

A relative's PC runs Ubuntu 8.04 (stable version) - a couple of weeks ago there was a filesystem corruption on the root FS, which uses ext3 on LVM. The kernel remounted the FS as readonly, but without notifying the user. I noticed this only recently, so I did a remote login - there weren't any hardware or kernel errors in the logs, only the remount message. The system was still usable and bootable at that point.

I ran e2fsck to fix the block device (FS was still read-only) - thousands of errors were found. One of these must have been in a key library, so that executing any command failed, although most files were still there. I now have to spend at least a day driving over there, re-installing Ubuntu, recovering from backups, etc.

This PC was built a year ago, with only high quality components (robust PSU, good motherboard capacitors, etc) and a conservative setup, including a UPS, and I chose Linux largely so I could maintain it remotely without hassles. Clearly this has not worked...

There are several things wrong here:

- the data loss bug itself - given the reports on Launchpad I strongly suspect a kernel or e2fsck bug, probably the latter. This is the second time I've had data loss due to an ext3 corruption without hardware errors - both times on LVM, perhaps that's a factor. I don't think I've ever lost a whole filesystem on Windows with FAT or NTFS other than with hardware errors.

- the fact that a stable version of a major Linux distro can have such major data loss problems over a year after its release

- (somewhat on topic) lack of an unmissable and persistent notification to the user (or ideally a remote administrator) that a significant error has happened (kernel noticing FS corruption and remounting the FS read-only. In fact, https://bugs.launchpad.net/bugs/28622 which covers this could be called a 'papercut' bug if the consequences weren't more serious

- the lack of really good and low cost online remote backup for Linux - I use SpiderOak which worked well in this case, and I find better than JungleDisk or Dropbox, but on another PC it has silently not done any backups for over a month.

- lack of continuous fscks (with meaningful notification to user or administrator) for PCs that are left switched on most of the time.

I use Linux and ext3 because I like reliability - I'm really stunned to find a massive data loss bug like this in 2009 on such a mature filesystem in a stable distro. Obviously such bugs are hard to reproduce but they are reported a lot.

Ubuntu 9.04 Jaunty apparently has a post-release kernel update that *introduces* a new ext3 data loss bug, yet this update has not been pulled... https://bugs.launchpad.net/ubuntu/+source/linux/+bug/346691 has the details. I really like Ubuntu but the non-handling of this data loss regression is rather horrifying - simply removing the updated kernel would be enough. I just installed Ubuntu 9.04 on a friend's PC last weekend - fortunately it was only as a recovery OS alongside Windows, in light of this.

Fighting small bugs

xav — Thu, 23 Jul 2009 04:51:47 +0000

s/Klasen/Clasen/ ?