Applications and bundled libraries

Posted Mar 17, 2010 20:30 UTC (Wed) by agl (guest, #4541)
Parent article: Applications and bundled libraries

(Disclosure: I am employed by Google and work on Chrome)

Keep in mind that we build on Windows, so much of the code in our
third_party directory is there because we need that code on Windows. There
are a number of configure time flags to switch between the third_party and
system versions of libraries like zlib, libevent etc.

Having said that we do have a number of forks. Here's an unrepresentative
selection of them covering some of the reasons:

libevent: we needed bug fixes and we needed to be able to run on systems
which didn't have them.

icu: we need a more recent version than was even provided on Karmic.

libjingle: upstream appears to be unmaintained.

sqlite: we added full-text indexing (now upstream) and several performance
improvements which are rather specific to our use case. We don't want to do
without them and upstream aren't too interested.

nss: we push patches upstream, but we are working on this heavily. Even so,
the default Chromium build uses the system version and disables the
features that we've added.

I'm going to avoid making philosophical points here. I only wanted to give
some of the real-life reasons why we carry forks of some of this code.

to post comments

Applications and bundled libraries

Posted Mar 17, 2010 20:37 UTC (Wed) by jospoortvliet (guest, #33164) [Link] (20 responses)

Does the shipping of these 'forks' also mean they have to be loaded in
memory twice? And if all apps would do that, would that increase the memory
load of an average system? Any idea how much, if so?

I don't care much about diskspace or bandwith, but memory is a different
matter...

Applications and bundled libraries

Posted Mar 17, 2010 21:39 UTC (Wed) by agl (guest, #4541) [Link] (19 responses)

Yes. Without the same .so, the common code will consume duplicated pages.

In practice, when just running Firefox or Chromium, the duplication is
pretty minimal. The big stuff (GTK etc) is certainly shared (at least with
Chromium).

Probably ICU's tables are the largest amount of duplication and that might
run to 5MB (to about a factor of two).

Applications and bundled libraries

Posted Mar 17, 2010 22:02 UTC (Wed) by jospoortvliet (guest, #33164) [Link] (18 responses)

Ok, got it. So for one or two apps, as long as they still share the big
stuff like GTK or Qt, not a big issue. If all apps would start doing it, and
if they would require specific versions of Qt/KDElibs/Gnome/GTK, things
would get ugly, I suppose...

Applications and bundled libraries

Posted Mar 17, 2010 22:02 UTC (Wed) by jospoortvliet (guest, #33164) [Link]

(which would be the slippery slope mentioned in the article)

Applications and bundled libraries

Posted Mar 17, 2010 22:25 UTC (Wed) by yokem_55 (subscriber, #10498) [Link] (16 responses)

Does Adobe acroread still ship with its own GTK libraries? Another annoying one is the proprietary version of VirtualBox that ships with its own private qt4 libraries which can be easily replaced with symlinks to the system libs.

Applications and bundled libraries

Posted Mar 18, 2010 4:51 UTC (Thu) by djao (guest, #4263) [Link] (14 responses)

I installed acroread the other day and I was, frankly, shocked at how good it is now. And I say this as one who strongly supports free software.

It still uses enormous amounts of memory (192M), but alternatives such as evince (147M) are not exactly efficient either. There's no embedded copy of GTK (I checked). It feels just as fast as evince, and in certain important objective respects (such as the proportion of screen space wasted by the chrome) it improves on evince by quite a large margin. In terms of stability, it's superior to the free alternatives -- I've never seen acroread crash, whereas I've seen plenty of crashes with the free pdf readers.

Finally, acroread uses system settings for font antialiasing, including subpixel antialiasing on LCD screens, which evince does not do (although my own copy of evince is patched to support this feature, because I really like it).

As I said above, I dislike the idea of proprietary software, but all things being equal, I'd much prefer acroread to be good than to be bad, and I have to admit it's getting good.

Applications and bundled libraries

Posted Mar 18, 2010 5:38 UTC (Thu) by roelofs (guest, #2599) [Link] (5 responses)

I've never seen acroread crash

Hoo boy, I sure have. Of course, I use an old version (37M) in preference to the JS-infested privacy disaster Adobe is currently shipping, so that probably has something to do with it. But after a few weeks or months of use, it frequently either locks up with/on an X grab or just blows itself out of the water altogether. It's not frequent enough to be a showstopper, but it is mildly annoying, and I use both Evince and xpdf as well (especially for forms).

Greg

Applications and bundled libraries

Posted Mar 18, 2010 6:53 UTC (Thu) by djao (guest, #4263) [Link] (4 responses)

Yes, the old versions of acroread were terrible, and that's why I was surprised at the stability of the latest version.

The best software for PDF forms, without question, is flpsed. It allows arbitrary annotations, even on PDFs that don't include embedded forms. You can save your work at any time and edit it later (hardly worth advertising as a feature, except for the fact that Acrobat Reader doesn't allow it), and the resulting output files are small and correct. It's also free software (GPL) and quite robust and stable.

Applications and bundled libraries

Posted Mar 18, 2010 7:40 UTC (Thu) by evgeny (subscriber, #774) [Link] (3 responses)

> The best software for PDF forms, without question, is flpsed.

I suggest you take a look at xournal (<http://xournal.sourceforge.net/>). Although it was primarily intended for use with a tablet input, later versions also have support for entering typeset text from keyboard. I use it exclusively for PDF annotations.

Applications and bundled libraries

Posted Mar 18, 2010 8:10 UTC (Thu) by djao (guest, #4263) [Link] (2 responses)

I was not aware of xournal; thanks for the tip. I just checked it out. I find that flpsed is a better fit for my needs, for the following reasons:

In flpsed, you can move an existing annotation to another location on the page using the arrow keys on the keyboard, with a granularity sufficiently fine for aligning text within form boundaries, but coarser than individual pixels. Xournal requires using the mouse for moving annotations, and only aligns along pixel boundaries, which (besides being meaningless in the context of vector graphics) is too fine a granularity to achieve perfect alignment of columns or rows of text vertically or horizontally. (This is a common requirement in many situations, e.g. tax forms.)
Switching between typing text and moving annotations around requires a mode switch in xournal (you have to move the mouse over to the other tool and click on it), but not in flpsed (just type normally and use the arrow keys to move the block of text you just typed, no mousing required), making xournal less efficient than flpsed for the case of filling in large numbers of extremely short text entries, such as tax forms. Again, this use case is quite common, and (for me) is the most important one.

I may be wrong about xournal's capabilities, as I only tried it out for a few minutes, so please correct me if I am wrong about xournal. I'm happy to hear about it, since it is more useful than flpsed in some situations (mainly when one has to annotate a graphical page of some kind, rather than a data form).

Applications and bundled libraries

Posted Mar 18, 2010 8:33 UTC (Thu) by evgeny (subscriber, #774) [Link] (1 responses)

Your points are correct, but my typical use is different from yours. I really use it for _annotations_, e.g. while refereeing docs of others or proofreading. The alignment is not very important then, but highlighting and underlining really are. I also use my tablet sometimes with handwriting (that's the intended use of xournal, of course). When I do use xournal for filling in forms, I usually type the text for the whole page without worrying much about alignment and then switch to the other mode and arrange the positions of different chunks of text. I agree that using the arrow keys while entering the text would be a convenient feature.

Applications and bundled libraries

Posted Mar 18, 2010 9:06 UTC (Thu) by djao (guest, #4263) [Link]

Wow, it literally never occurred to me to use xournal for refereeing. That's something that I actually do, and I can see how xournal would be ideal for that task. Thanks!

I still believe, however, that flpsed is better for PDF forms, which invariably consist largely of data entry.

Applications and bundled libraries

Posted Mar 18, 2010 7:43 UTC (Thu) by jospoortvliet (guest, #33164) [Link] (7 responses)

I use acrobat reader at work - as Okular can't be installed that easily on
windows if you don't have admin rights ;-)

But I find Okular far superior to adobe's product, mostly in the UI
department. Acrobat is just horrible, while Okular has 95% of the features
yet a clean and efficient interface. Just try using your mouse to scroll
through a page - when you hit the bottom of the page, things stop in
Acrobat. Okular just goes on, you won't even notice such features but they
matter once you have to go back to Acrobat.

Applications and bundled libraries

Posted Mar 18, 2010 8:15 UTC (Thu) by djao (guest, #4263) [Link] (6 responses)

When I use Acrobat Reader, scrolling past the end of a page brings the top of the next page within view, and it displays both the bottom of the previous page and the top of the next page in a continuous scroll, similar to evince and okular. It certainly does not just "stop", unless the page in question happens to be the last page in the document, in which case all the other PDF readers also stop. The scrolling behavior of Acrobat Reader is configurable: Edit -> Preferences -> Page Display -> Automatic (or Single Page Continuous).

Perhaps you're using an old or misconfigured version of Acrobat Reader?

Applications and bundled libraries

Posted Mar 18, 2010 9:46 UTC (Thu) by jospoortvliet (guest, #33164) [Link] (5 responses)

I didn't mean using the scrollwheel but the dragging function. Click, drag - you'll have to let go at the bottom of the page, go up again and drag again. RSI, here I come!

In Okular you can just scroll on, no extra movements needed. It's a small touch, extremely intuitive and I only figured out I was using it when someone pointed it out ;-)

Applications and bundled libraries

Posted Mar 18, 2010 18:33 UTC (Thu) by djao (guest, #4263) [Link] (4 responses)

In my version of Acrobat Reader (9.3.1 Linux), I cannot reproduce the behavior you describe. No matter how I scroll the document (scroll wheel, dragging the scrollbar, dragging the main page with the hand tool), the program scrolls continuously past the end of a page to the beginning of the next, without stopping.

Applications and bundled libraries

Posted Mar 18, 2010 21:34 UTC (Thu) by jospoortvliet (guest, #33164) [Link] (3 responses)

It's not about the page, it is about the mouse. If you grab the page with
the mouse (keep left mouse button pressed) you can drag it to the bottom
of the screen. But then the dragging stops, doesn't it, as you've reached
the bottom of the screen... But in Okular, you can continue to drag
because it 'wraps' the screen by moving the mouse to the top of the screen
and you just drag on.

It's tiny, hard to understand (clearly) if you haven't seen it, yet
completely intuitive and unobtrusive. Just a nice touch.

Similarly nice is finding stuff in Okular, btw. I find the search bar on
the left, which only shows the pages where the search results show up so
you get a quick overview of where the term you were looking for is, far
superior to Acrobat's approach. Acrobat lacks such a simple yet effective
search - you have to go through everything with F3.

Also the automatic scrolling (shift-arrow down) is very nice, I've used
Okular a lot to read from the screen, adjusting speed with the arrow keys
(shift-arrow, again). Press shift to stop scrolling, shift again to
continue. Space to move one page further, shift-space to go back. Sure,
Acrobat offers auto scrolling with the mouse, like word and most
webbrowsers, but it's far less nice imho.

Again, tiny differences, but as I don't use any of the advanced stuff but
just read and search for stuff (and annotate sometimes) Okular is perfect.

Applications and bundled libraries

Posted Mar 18, 2010 22:17 UTC (Thu) by dlang (guest, #313) [Link]

what I've seen is that when you try to drag outside the window, it gets interpreted as a scroll request and the window starts scrolling until you move the mouse back into the window area.

I don't know if this is implemented by the window manager or by the individual app.

Applications and bundled libraries

Posted Mar 18, 2010 22:22 UTC (Thu) by djao (guest, #4263) [Link] (1 responses)

I have okular installed here (Fedora 12) and I cannot reproduce the behavior you describe.

One problem is that, most of the time, when I'm scrolling through a PDF, I want to read the pages from the top down (i.e. scroll forward through the file). However, in order to scroll forward by dragging the main page with the left mouse button, the mouse cursor itself actually has to move up in order for the page content to move down. So my mouse cursor never hits the bottom of the screen like you describe, unless I'm scrolling backwards, which happens very rarely. When I scroll forward, the cursor hits the top of the screen, and when it hits the top, it certainly doesn't automatically wrap the cursor to the bottom.

Since I cannot reproduce this behavior, I have to make certain assumptions about what you mean. Assuming you meant that the mouse cursor wraps from top to bottom, I can see how it would be a worthwhile option, but I would never use it myself. Most of my pdf reading occurs on a laptop, with a touchpad, in which case dragging the page is worse than useless -- it requires holding down a button as well as moving a finger along the touchpad, whereas the scrollwheel is built in to the touchpad and only requires moving a finger along the touchpad, and thus involves strictly less work. The only time I use dragging is for fine (pixel-level) scrolling control that cannot be achieved with the scroll wheel, but in such cases wraparound is unnecessary.

In addition to the lack of utility, my own opinion is that the bottom of the screen should be an absolute boundary to movement, not an invitation to wrap the cursor around to the top of the screen, no matter how worthy the justification may be. Moreover, if the PDF is displayed in a window, rather than full screen, then automatic cursor wraparound would be even more jarring, as it would jump from the top of the window to the bottom of the window rather than the top of the screen to the bottom of the screen.

Applications and bundled libraries

Posted Mar 19, 2010 17:06 UTC (Fri) by jospoortvliet (guest, #33164) [Link]

Interesting. Yes, you understood what I meant, it's strange that you can't reproduce it. Maybe it does depend on the windowmanager used. Either way, I haven't used it in a while now - I did use it when reading lots of PDF's while doing research for a paper. I vividly recall how I disliked Acrobat, as it was far less nice to use for reading. I actually went home behind my own computer screen to read, rather than doing it at work (where I only had windows available).

But I guess everyone's habits are different, as are preferences ;-)

I just wanted to illustrate a very small yet nice feature Okular has which makes it (to me) nicer than Acrobat. It has more of those, of course ;-)

Applications and bundled libraries

Posted Mar 18, 2010 13:06 UTC (Thu) by mjthayer (guest, #39183) [Link]

>Another annoying one is the proprietary version of VirtualBox that ships with its own private
>qt4 libraries which can be easily replaced with symlinks to the system libs.
VirtualBox provides packages for several distributions, as well as a script-based installer
which is supposed to run "anywhere". Only the packages which have to run on older
distributions ship their own copies of Qt4.

Applications and bundled libraries

Posted Mar 17, 2010 20:45 UTC (Wed) by jspaleta (subscriber, #50639) [Link] (3 responses)

Are you keeping a detailed accounting of individual patches such that its easy to see which patches have been submitted to upstream and then whether each patch has been rejected/approved?

Can things be engineered such that functionality from rejected/yet-to-be-approved patches to upstream can be disabled cleanly in rebuilds? You've alluded that his is the case for nss can that also be the case for sqlite and others where there is an active upstream?

For libjingle.. if the upstream project is verifiably dead... why doesn't google spin up their libjingle as a separate project for distributors to pull releases from.

-jef

Applications and bundled libraries

Posted Mar 17, 2010 21:38 UTC (Wed) by tzafrir (subscriber, #11501) [Link] (2 responses)

The upstream for libjingle is actually Google.

Applications and bundled libraries

Posted Mar 17, 2010 21:41 UTC (Wed) by jspaleta (subscriber, #50639) [Link] (1 responses)

The gods of irony are please. A good omen for the maintainability of chromium. Well played Google... well played.

-jef

Applications and bundled libraries

Posted Mar 20, 2010 12:51 UTC (Sat) by man_ls (guest, #15091) [Link]

It may seem obvious, but I have to ask -- why not revive libjingle and push the patches upstream?

Applications and bundled libraries

Posted Mar 18, 2010 0:15 UTC (Thu) by shahms (subscriber, #8877) [Link] (1 responses)

(Disclaimer: Also a Google employee, although I don't work on Chrome).

As much griping as goes on about Google's forks and patches I generally
find their approach to be responsible and practical. The focus is on
getting something working well, even if that means patching and bundling
system libraries, followed by pushing those patches upstream and
unbundling, if possible. Or, occasionally, becoming upstream.

For open source projects, it's usually not an issue. You publish the
source of your application. You publish patches to or patched versions of
the necessary dependencies. Packagers/maintainers then have the option of:

1) Updating to the upstream version that has your patches
2) Patching the shipped version
3) Bundling the library with the application (in some manner)

Yes, it's a little extra work for the repository maintainer, but not much.
It's also a (very) little extra work for the application developer to make
sure they can be built and linked against the system version of the
library. As a former Fedora maintainer, I've had to resort to some version
of that several times. Including coordinating with the maintainers of
dependent packages. Of course, Ubuntu has kind of shot themselves in the
foot on this issue with the LTS stuff, but that's (kind of) beside the
point.

Now, for closed source apps, the burden is on the application developer
and, frankly, I don't want them trying to use the more change-prone system
libraries. Bundle. Install into /opt/<application> and go away. That's
all they usually do anyway.

I'm not entirely up to speed on the Firefox/Mozilla situation, but the last
time I checked at least part of the problem was distributions packaging
mostly-internal unstable libraries as "system" libraries and then linking
other packages against them. Oh, and failing to understand/use ELF
versioning. Most distributions are perfectly capable of having multiple
versions of libraries installed and linked appropriately, provided the
library authors follow the .so versioning rules.

Applications and bundled libraries

Posted Mar 22, 2010 18:39 UTC (Mon) by lkundrak (subscriber, #43452) [Link]

I stopped reading your comment at "focused on getting something working well" point. Are you really sure avoiding distributions shipping the code qualifies as working well? When the software can't even be installed via standard methods?

Applications and bundled libraries

Posted Mar 18, 2010 3:59 UTC (Thu) by blitzkrieg3 (guest, #57873) [Link]

Keep in mind that we build on Windows, so much of the code in our third_party directory is there because we need that code on Windows. There are a number of configure time flags to switch between the third_party and system versions of libraries like zlib, libevent etc.

No one is decrying your use of a third_party directory. Mozilla does this too.

However, it's a bit harder to use system libraries than you make it seem. For example, looking through spot's src.rpm, I see no less than 7 patches to hardcode stuff to use system libs. Take this commit for example. Pretty much every file references ../third_party/icu, but when you look for the actual directory, it isn't there!

Having said that we do have a number of forks. Here's an unrepresentative selection of them covering some of the reasons: libevent: we needed bug fixes and we needed to be able to run on systems which didn't have them. icu: we need a more recent version than was even provided on Karmic.

These are the distributors problem. Either they configure --with-system-libevent and get the libevent maintainer to update their part or build and ship with the library.

libjingle: upstream appears to be unmaintained.

Several have mentioned Google is the upstream maintainer. Either whip the talk guys into shape or get commit access to their tree.

sqlite: we added full-text indexing (now upstream) and several performance improvements which are rather specific to our use case. We don't want to do without them and upstream aren't too interested.

This sounds like an excuse.

I don't think that Mozilla is that much better, and spot is incorrect that they don't bundle libraries (--with-system-png is commented out in F13 alpha and I know mozilla has their own brand of cairo), but with a few changes we can duplicate less work and simultaneously get a better product

Applications and bundled libraries

Posted Mar 18, 2010 9:48 UTC (Thu) by jengelh (subscriber, #33263) [Link] (5 responses)

>Having said that we do have a number of forks. Here's an unrepresentative selection of them covering some of the reasons:
>
>libevent: need new version for bugfixes
>icu: we need a more recent version than was even provided on Karmic.
>[and so on]

So what? If $distro ships libevent2-1.4.9 for example, there are two choices:
1. you need libevent.so.2-1.4.10: fine, make it a dependency of chromium and let the distro update their libevent2.
2. you need libevent.so.3-1.5.0: fine, make it a dependency of chromium and let the distro add a package libevent3.

sqlite: if upstream is not interested, rename it, and let the distro have a sqlite-google package created (this also requires that you make it use a different SONAME than the pristine sqlite), or, if it's API-compatible, have the distro replace the original sqlite/import your patch.

Applications and bundled libraries

Posted Mar 18, 2010 15:04 UTC (Thu) by NAR (subscriber, #1313) [Link] (4 responses)

If $distro ships libevent2-1.4.9 for example, there are two choices:
1. you need libevent.so.2-1.4.10: fine, make it a dependency of chromium and let the distro update their libevent2.

What happens if libevent.so.2-1.4.10 breaks some other application? There are surprising number of applications that actually depend even on specific bugs being present in a library (because e.g. the bug was there years ago and they've implemented a workaround which will be broken by the proper fix)...

Applications and bundled libraries

Posted Mar 18, 2010 15:11 UTC (Thu) by jengelh (subscriber, #33263) [Link]

A responsible distro tries to coordinate one or more fixes being available to them and upstream, because ultimately, the distro will ship a newer version of the library at some point and the programs.
The world would probably have a lot more compile/runtime failures and churn if distros did not do what they are already doing.

Applications and bundled libraries

Posted Mar 18, 2010 15:28 UTC (Thu) by cortana (subscriber, #24596) [Link] (2 responses)

If a new release of a library alters behaviour in a backwards-incompatible way, then the SONAME
is bumped (that is, going from libevent.so.2 to libevent.so.3 in the example).

Applications and bundled libraries

Posted Mar 19, 2010 11:30 UTC (Fri) by smurf (subscriber, #17840) [Link] (1 responses)

Yeah, for major changes. For minor ones (like the aforementioned bug workarounds), symbol versioning is a much better choice.

Applications and bundled libraries

Posted Mar 19, 2010 11:36 UTC (Fri) by jengelh (subscriber, #33263) [Link]

Symbol versioning would not work in this case, I think, because as you recompile $broken_program (which can easily happen in things like rawhide/factory), you are linking it against $new_symbol.

Help for the source built systems?

Posted Mar 18, 2010 14:25 UTC (Thu) by alex (subscriber, #1355) [Link]

Does the Chromium build system use system libraries if they are new enough?
If the autoconf (or whatever build system chromium uses) could do a version
check on the system library at build time this might reduce the pain
somewhat.

One of the reasons I run a source based distro is so I can run with recent
libs on my system which is handy as a developer.

Of course this would involve keeping careful track of the divergence of
patches between the bundled libs and upstream. I'm guessing this is a wider
problem in free software that needs a decent solution.

Applications and bundled libraries

Posted Mar 18, 2010 15:42 UTC (Thu) by judas_iscariote (guest, #47386) [Link] (1 responses)

> libevent: we needed bug fixes and we needed to be able to run on systems
>which didn't have them.

Simple, abort build saying libevent x.y.z is needed, distributors will
figure it out.

>icu: we need a more recent version than was even provided on Karmic.

Same here.

>libjingle: upstream appears to be unmaintained.

Huh, what about asking your own company to support its own code ?

> sqlite: we added full-text indexing (now upstream) and several
>performance improvements which are rather specific to our use case. We
>don't want to do without them and upstream aren't too interested.

Contact distributors and ask them to add your patches to the system
libraries.

Applications and bundled libraries

Posted Mar 22, 2010 18:43 UTC (Mon) by lkundrak (subscriber, #43452) [Link]

So, how about v8?