Definition of software bloat - see "Emacs" (veering off-topic) [LWN.net]

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 4:35 UTC (Mon) by pr1268 (guest, #24648)
In reply to: Emacs news: new maintainer, version 22 pretest by djabsolut
Parent article: Emacs news: new maintainer, version 22 pretest

A friend was recently lamenting that he couldn't edit a 750-megabyte text file (don't ask) in Emacs - it simply refused to open the file. I smirked that Emacs was so incredibly bloated that there was no more room in RAM to open even part of the file. To which he replied, "Yes, but doggone it, I can play Tetris!" :-)

to post comments

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 4:36 UTC (Mon) by atai (subscriber, #10977) [Link] (16 responses)

well can notepad handle it?  gedit? nedit? vi?

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 4:43 UTC (Mon) by pr1268 (guest, #24648) [Link] (11 responses)

Vi (actually, Vim) opened the file, albeit slowly. The weird part was searching down about 96% of the entire file to find the part he was trying edit. Needless to say the hard disk was quite active for some time.

I'm unsure what kind of success other editors would have - I prefer Kwrite in KDE as my preferred text editor (assuming I've got X and KDE running), but even it hesitates noticeably when opening a text file larger than e.g., 5 megabytes.

Posted Feb 25, 2008 6:05 UTC (Mon) by joey (guest, #328) [Link] (10 responses)

I sometimes fear I'll never find
an editor that uses rewind(3).

Is that the system call I need?
Perhaps it should lseek(2) and read(2).

Why jumping around in text is hard

Posted Feb 25, 2008 6:30 UTC (Mon) by zlynx (guest, #2285) [Link] (9 responses)

It would be great if more editors used lseek or mmap to read the files.

But, it is difficult to do this.

If you wish to jump to line 9997, where is it?  How many characters are in a line?  The editor
cannot know unless it scans for each newline from position 0.

Text encodings like UTF-8 complicate this even more because now characters can be variable
sizes as well.

Why jumping around in text is hard

Posted Feb 25, 2008 7:54 UTC (Mon) by nix (subscriber, #2304) [Link] (5 responses)

Even that wouldn't help much, at least on 32-bit systems. You've got a 
choice there between reading everything into memory as (X)Emacs does, or 
mmap()ing with MMAP_PRIVATE and having a well-under-4Gb maximum file size 
limit *and* a requirement to configure huge amounts of swap space, or a 
loooong wait as the file gets copied to new storage at startup (vim of big 
files on slow NFS storage is *painful*, much worse than Emacs), or even an 
editor that essentially implements its own VM management and a sliding 
file access window, copying changed bits to a new file, which would 
necessarily crap out if the file was changed out from underneath it. (I've 
never seen anyone implement the latter, and no surprise: it would be 
really difficult and have no real advantages).

I'd like to find an ideal middle road, but I'm not sure there is one.

Why jumping around in text is hard

Posted Feb 25, 2008 9:02 UTC (Mon) by eru (subscriber, #2753) [Link] (4 responses)

or even an editor that essentially implements its own VM management and a sliding file access window, copying changed bits to a new file, which would necessarily crap out if the file was changed out from underneath it. (I've never seen anyone implement the latter, and no surprise: it would be really difficult and have no real advantages).

Remember the days of 8-bit and earlier 16-bit computers? Text editors at the time normally used techniques like this to be able to edit larger files. Otherwise editing would be limited to something like 30 K (remember that the OS, the ROM, the editor and the data would all have to fit into a 64k address space - if you were lucky to even have that much memory!).

I guess writing an editor that does not keep all its data in memory now qualifies as forgotten tech.

Why jumping around in text is hard

Posted Feb 25, 2008 21:39 UTC (Mon) by nix (subscriber, #2304) [Link] (3 responses)

Waste-of-time tech, more like. It adds significant complexity, several new 
failure modes (other processes changing the file, not something many older 
systems really had to consider), and doesn't even provide anything you 
can't get by privately mmap()ing it and relying on the OS to do 
everything.

`If you want to edit truly vast files in a text editor, get a 64-bit box' 
seems a fairly reasonable requirement: I mean, how often do you want to 
edit multigigabyte text files anyway?

The older systems had to implement something like their fake-VM 
sliding-window things because editing a book is an entirely reasonable 
thing to want to do. But how many people want to edit the entire British 
Library, as one unstructured lump of text? It's hardly a common need.

Why jumping around in text is hard

Posted Feb 26, 2008 6:04 UTC (Tue) by eru (subscriber, #2753) [Link] (2 responses)

I don't really disagree with you in general. But I too have occasionally encountered Emacs (or other text editor) limitations when wanting to browse a really big log file or collected debugging output from some misbehaving program. Actually a pretty common need in software development. Probably this can be seen as misusing the tool, as in those cases I usually have no real desire to change the file. On the other hand it is natural, since Emacs is the way I normally interact with text files. Perhaps for those cases there should be a quick read-only viewing mode that makes no attempt to read anything but the region being viewed, and this mode would be used automatically for files larger than some threshold.

Why jumping around in text is hard

Posted Feb 26, 2008 15:25 UTC (Tue) by anand21 (guest, #38076) [Link] (1 responses)

If you do need to browse a large log file, less is your friend.

Editing is a hard problem. Modern editors do a lot of formatting and storing that information
takes a lot of space. Concurrent access could be solved by locking but it wouldn't help
because the editor must still load the whole file, and then parse it and store temporary
information, for providing a consistent view.

Why jumping around in text is hard

Posted Feb 28, 2008 20:03 UTC (Thu) by nix (subscriber, #2304) [Link]

Locking text files is *really unfriendly*, too. Shades of Windows.

(Note that less also sucks the whole file into RAM, so if it's huge less 
won't do the trick either.)

(Emacs doesn't do a lot of *formatting*, as such, but both the Emacsen 
*can* attach properties/extents/whatever to regions of buffers, and both 
also have to handle multibyte support to the extent of having single 
buffers with many encodings in. All of these things push up the memory 
consumption somewhat, especially the latter.)

Why jumping around in text is hard

Posted Feb 25, 2008 8:04 UTC (Mon) by nix (subscriber, #2304) [Link]

XEmacs at least has a cache mapping (line number -> newline location), 
invalidated on significant buffer changes. Whether it's worthwhile in this 
day and age I'm not sure: it may be faster to scan for nearby newlines 
(probably both in the L2 cache if referenced recently) than to maintain 
the line-start cache; and distant lines won't be in that cache anyway.

Why jumping around in text is hard

Posted Feb 25, 2008 10:20 UTC (Mon) by rsidd (subscriber, #2582) [Link] (1 responses)

qemacs (unfortunately unmaintained) uses mmap(). I tried reading a 450MB file in memory, and it alternately acts very responsively and locks up solid for up to a minute at a time. It also fails to find search strings at the end of the document if I am at the beginning.

emacs takes a few seconds to open the file, but when I was at the top, it found a search string near the bottom quite readily. It does not seem to lock up for more than a second or two. This is an opteron workstation with 2GB RAM, so I guess it's not exercising the swap.

Why jumping around in text is hard

Posted Feb 25, 2008 10:37 UTC (Mon) by tzafrir (subscriber, #11501) [Link]

qemacs. Indeed tester under harsher conditions. And unlike GNU- and X- Emacs it even has
support for bidirectional text rendering.

'nvi -f' also does the trick.

Or, if you only want to view it: less

P.S. Although I'm not a regular Emacs user, this is still a good place to thank the previous
maintainer for the hard work he put in all previous releases and welcome the new maintainers.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 7:15 UTC (Mon) by DG (subscriber, #16978) [Link] (2 responses)

vi[m] should be able to without a problem (I've done similar things with SQL dump files) -
although you'll probably want to skip using your customised .vimrc which adds in colouring and
the like _before_ opening it, else you'll be in for a long wait.

David.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 16:35 UTC (Mon) by kh (guest, #19413) [Link] (1 responses)

I have had problems with vim and large files that had no newlines (but larger files with
newline chars were fine.)

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 18:22 UTC (Mon) by TRS-80 (guest, #1804) [Link]

Yeah, syntax highlighting is only poorly behaved when there are no newlines - if I turn it off before loading a large file vim becomes snappy again.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 7:37 UTC (Mon) by BackSeat (guest, #1886) [Link]

I'm a vim user, but I've found that joe is good for editing the occasional very large file.

New definition (Definition of software bloat - see "Emacs" (veering off-topic))

Posted Feb 25, 2008 9:08 UTC (Mon) by eru (subscriber, #2753) [Link] (10 responses)

I smirked that Emacs was so incredibly bloated that there was no more room in RAM to open even part of the file.

Funny how this old idea about Emacs persists. Compared to other contemporary interactive programs, Emacs is positively light-weight! (I would call OpenOffice.org the new gold standard for bloatedness.)

New definition (Definition of software bloat - see "Emacs" (veering off-topic))

Posted Feb 25, 2008 19:37 UTC (Mon) by jordanb (guest, #45668) [Link] (9 responses)

> (I would call OpenOffice.org the new gold standard for bloatedness.)

Clearly you haven't tried eclipse yet. ;)

New definition (Definition of software bloat - see "Emacs" (veering off-topic))

Posted Feb 25, 2008 20:19 UTC (Mon) by zlynx (guest, #2285) [Link] (8 responses)

It depends on which sort of bloat.

Eclipse wins for run-time memory usage bloat.  I've seen it go to 800 MB.

But Open Office wins for source code bloat I think.  It can take over 8 hours to compile on a
fairly decent machine and GCC 4.2.x.

New definition (Definition of software bloat - see "Emacs" (veering off-topic))

Posted Feb 25, 2008 21:41 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

Agreed.

`Look! There's a wheel!'
`I know, let's reinvent it, and make ours square.'

Iterate for several dozen wheels.

New definition (Definition of software bloat - see "Emacs" (veering off-topic))

Posted Feb 26, 2008 13:05 UTC (Tue) by smitty_one_each (subscriber, #28989) [Link] (1 responses)

The clear goal of any wheel-reinvention project is to rationalize pi, silly rabbit.

New definition (Definition of software bloat - see "Emacs" (veering off-topic))

Posted Feb 28, 2008 19:46 UTC (Thu) by nix (subscriber, #2304) [Link]

OpenOffice tries to have its pi and eat it, but it's eaten so many pis 
that it's got terribly fat.

*puts metaphor into concrete mixer and walks away*

[OT] The real definition of software bloat

Posted Feb 25, 2008 22:50 UTC (Mon) by man_ls (guest, #15091) [Link]

800 MB! Haw haw haw! If you want to see positive bloat, just try IBM's payware versions of Eclipse. WSAD 6 was ludicrous in its memory usage, and (with help from a few custom plugins) I have seen it reach 2 GB. As to the all new RAD 7... the full installer takes something like 11 DVDs.

Of course none of this is free software, but it is based on free software.

Still more software bloat (even further off topic)

Posted Feb 26, 2008 2:01 UTC (Tue) by pr1268 (guest, #24648) [Link] (3 responses)

But Open Office wins for source code bloat I think. It can take over 8 hours to compile on a fairly decent machine and GCC 4.2.x.

While we're on this tangent, I might add that another friend of mine actually attempted to compile Firefox from source. Something like 450 MB of source code alone! After a whole weekend he hadn't gotten very far. (He thought it was convenient that the FF image rendering library is titled "libpr0n" or similar.) :-)

As for my earlier friend trying to edit the 750 MB text file in Emacs, FWIW it was a Dell desktop PC with a (32-bit) P4 HT and 1GB of RAM running RHEL 4.

Still more software bloat (even further off topic)

Posted Feb 26, 2008 21:24 UTC (Tue) by kamil (guest, #3802) [Link] (2 responses)

I find this hard to believe.  Being a Gentoo user, I routinely compile Firefox on my machines.
On the lowly laptop I'm typing this in (Pentium-M 1.7 GHz), Firefox 2.0.0.12 took 32 minutes
to compile.  Maybe your friend simply didn't have enough RAM and GCC had to resort to using
swap -- that would sure slow the process down to a crawl.

Having said that, I never even try compiling OOo.  It is the only open source package I
install from pre-compiled binaries.

Still more software bloat (even further off topic)

Posted Feb 26, 2008 21:42 UTC (Tue) by pr1268 (guest, #24648) [Link] (1 responses)

Maybe your friend simply didn't have enough RAM and GCC had to resort to using swap -- that would sure slow the process down to a crawl.

Oops! I forgot to mention that my friend attempting to compile Firefox from source code was doing so in Windows using Visual Studio .NET. He mentioned that he was able to hobble together some functionality but the running program had "various stability and reliability issues...".

I thought I read somewhere that the Win32 version of FF is compiled on Windows using Cygwin and/or MinGW using CMake, but I might be mistaken.

Still more software bloat (even further off topic)

Posted Feb 28, 2008 19:36 UTC (Thu) by nix (subscriber, #2304) [Link]

Yeah, well, that explains it. Cygwin is very slow at fork()ing, and 
several fork()s are involved in the process of compiling a single .o. 
There are tens of thousands of object files in Firefox.

(The sources for Firefox on my system clock in at a 'mere' 340Mb.)

Text files of that size are absurd

Posted Feb 25, 2008 9:27 UTC (Mon) by flewellyn (subscriber, #5047) [Link] (2 responses)

Good heavens, 750 megs for one text file?  That's simply ridiculous.  I'd have long since
split the file into multiple files.

Text files of that size are absurd

Posted Feb 25, 2008 12:46 UTC (Mon) by smitty_one_each (subscriber, #28989) [Link] (1 responses)

He said "(don't ask)"! ;)

Text files of that size are absurd

Posted Feb 25, 2008 13:09 UTC (Mon) by flewellyn (subscriber, #5047) [Link]

Yes, I know, but...sheesh.  I wouldn't even try using an interactive editor on a file that
size.  That's what sed and awk are made for.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 9:30 UTC (Mon) by debacle (subscriber, #7114) [Link] (2 responses)

I don't believe that failing to read your file is caused by the memory consumption of Emacs
itself. Maybe it's a bug in Emacs or whatever. By nowadays standards Emacs is absolutely not
bloated. Compare it to other IDEs, such as Eclipse, or word processing tools like OpenOffice
and you find Emacs very memory-friendly. I know some developers who have abandoned Emacs in
the last five years and moved to Eclipse. Despite Eclipse being significantly more memory and
CPU hungry - just because Eclipse is more powerful for them. I'm still hanging around with
Emacs...

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 9:37 UTC (Mon) by pr1268 (guest, #24648) [Link]

I don't believe that failing to read your file is caused by the memory consumption of Emacs itself. Maybe it's a bug in Emacs or whatever.

My smirky comment was intentionally facetious; I didn't mean to ignite a whole vi(m) vs. emacs flame war over it. That being said, I don't think my friend's error was caused by a "bug" or software defect; he received an intentional error message from Emacs when he attempted to open the file.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 21:47 UTC (Mon) by edschofield (guest, #39993) [Link]

> Compare it to other IDEs, such as Eclipse, or word processing tools like OpenOffice and you
find Emacs very memory-friendly.

Exactly! And I am taller than the Japanese, richer than the Ethiopians, and have better teeth
than the Molvianians.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 9:32 UTC (Mon) by rise (guest, #5045) [Link] (4 responses)

Interesting, I remember being shocked when I opened a multiple gigabyte log file in GNU emacs
with no issues approximately three years ago.  It took a little while to start up, but it
wasn't bad at all.  I can't recall if this was on a 64-bit processor or not, that might make a
difference.  In my experience emacs handles large files much better than programs like less.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 9:37 UTC (Mon) by rise (guest, #5045) [Link]

Looks like that was probably a compilation option or word-size effect, I just tried it with
the stock openSUSE 10.3 version (22.1.1) on a plain x86 processor and it complains about files
over 1GB.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 21:43 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

64-bitness will definitely make a difference. This all has to do with the 
maximum size of Lisp integers (as used for things like the buffer 
position). Emacs uses a couple of high bits as tag markers (like many Lisp 
implementations), leaving you with enough to open a Gb or so on a 32-bit 
platform (maybe still 512Mb, I can't remember if the limit was ever 
lifted).

Emacs maximum buffer size

Posted Mar 6, 2008 17:34 UTC (Thu) by anton (subscriber, #25547) [Link] (1 responses)

Yes, I also guess that the buffer size limit comes from the limited word size combined with needing some bits for Lisp tags. I just made a few experiments:

Starting a 32-bit Emacs on a 159MB file: "Maximum buffer size exceeded".
Starting a 32-bit Emacs on a 127.1MB file: works, and that emacs then has a VSZ of 137MB and an RSS of 134MB.
Starting a 64-bit Emacs on an 784MB file: works.

Emacs maximum buffer size

Posted Mar 9, 2008 19:25 UTC (Sun) by nix (subscriber, #2304) [Link]

It's not a guess, it's a fact, and even a documented one :)

Perhaps it sucks a bit, but it's essentially irrelevant on 64-bit 
platforms unless Emacs starts to use a whole *lot* more bits for tags...

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 9:37 UTC (Mon) by pdewacht (subscriber, #47633) [Link] (1 responses)

Emacs has the limit that file positions need to fit in an elisp integer. The elisp tagging
scheme requires 4 bits for integers, so on 32-bit systems you're limited to 28 bit ==> 256mb
files. The limit was even lower before Emacs 22 :(

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 14:05 UTC (Mon) by andikleen (guest, #39006) [Link]

You could actually get 256MB files even before 22, you just
had to patch the source.

But the obvious solution: use a 64bit system. 32bit is so 90ies...

Ok emacs gets a fatter because all words are 64bit now
instead of 32bit, but on the other hand it is still really rank'n'lean
now compared to several other popular programs that edit text.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 25, 2008 17:08 UTC (Mon) by mti (subscriber, #5390) [Link]

Works fine with XEmacs on a 64-bit system. It took some time loading but editing works well.
M-x goto-line 1000000 takes 2-3 seconds.

I remember using Emacs to edit files in the 10-20 MB range more than a decade ago.

Btw, I didn't have a 750 MB textfile so I open a iso-image instead
(ubuntu-7.10-desktop-i386.iso).

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 28, 2008 22:12 UTC (Thu) by Xman (guest, #10620) [Link] (1 responses)

Emacs' use of tagging limits its maximum file size pretty significantly on 32-bit systems.
IIRC the limit is like 24MB.

Definition of software bloat - see "Emacs" (veering off-topic)

Posted Feb 29, 2008 1:54 UTC (Fri) by nix (subscriber, #2304) [Link]

Waaaay bigger than that. GNU Emacs uses (IIRC) four bits as tag bits, 
limiting Emacs int size and buffer size to 2^28 (256Mb). XEmacs is 
somewhat less limited in this area, but a limit still applies.