|
|
Subscribe / Log in / New account

This is the heart of the problem

This is the heart of the problem

Posted Feb 13, 2014 20:40 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)
In reply to: This is the heart of the problem by madscientist
Parent article: The Debian technical committee vote concludes

> You can have your own strlcpy() in a trivial number of lines of code.
Look at glibc's implementation of string functions. They are not trivial.

> And I see absolutely no point in having glibc support compiling for older versions (and I've done embedded and cross development work for most of my career).
I have Ubuntu 14.04

How do I compile a binary for RHEL5?

> It would add a lot of maintenance headache, AND it's completely trivial to get the same results without all that complexity and _much_ more reliably: you simply have to provide the --sysroot flag to your builds and point it to an older set of libraries and headers. Done.
No, not done. It requires to create a completely separate build environment with all the required libraries. It's easier to install a complete OS in a chroot at that point.


to post comments

This is the heart of the problem

Posted Feb 13, 2014 20:55 UTC (Thu) by dlang (guest, #313) [Link] (3 responses)

> It's easier to install a complete OS in a chroot at that point.

This is your answer to the RHEL 5 question above. glibc is the least of your worries, every other library you use is a problem as well.

the thing is that you don't have to do this for every distro you want to use, you just do it for the oldest one and newer ones almost always work.

this is backwards compatibility of the systems, things build for older systems keep working on newer systems.

I know that you praise the ability of the windows monoculture to set a flag and have the resulting binary work for that old version, but with multiple competing vendors it's not that easy to point at an arbitrary 'old' version of some other companies work.

This is the heart of the problem

Posted Feb 13, 2014 21:26 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

I can realistically avoid using every other library or in a pinch simply package them along with my application.

But not glibc. It can't be packaged or avoided.

This is the heart of the problem

Posted Feb 16, 2014 2:55 UTC (Sun) by dlang (guest, #313) [Link] (1 responses)

sure you can. It may not be trivial, but there are several other libc implementations, and some people do use them instead of glibc

but even if you use glibc, the glibc developers can't implement policy in your apps by implementing some new function, they can only do it if they change an existing function.

so while they can break or change something they currently do, they can't take over other functions at a whim

This is the heart of the problem

Posted Feb 16, 2014 5:11 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

Nope. It sounds easy but when you try to mix uclibc and glibc things go downhill. Fast. And it's extremely easy to pull in glibc accidentally. By depending on libgl, for example.

This is the heart of the problem

Posted Feb 13, 2014 22:42 UTC (Thu) by madscientist (subscriber, #16861) [Link] (25 responses)

> Look at glibc's implementation of string functions. They are not trivial.

Nevertheless, writing a strlcpy() is trivial. There are even lots of free implementations available for you. If you want versions that take advantage of all the performance tweaks for your particular hardware platform, that is much more work, sure.

> I have Ubuntu 14.04. How do I compile a binary for RHEL5?

Very simple. I've done it many times. You go get copies the RPMs or an installation ISO (if you don't want to pay for RHEL licensing you can get CentOS instead: it's binary compatible). You extract the RPMs into some directory on your system (this is simple because rpm is packaged for Debian and Ubuntu). Then you add "--sysroot=<extractdir>" to your compile and link lines.

Done and done.

> No, not done. It requires to create a completely separate build environment with all the required libraries. It's easier to install a complete OS in a chroot at that point.

That's just not true. In fact, I think chroots are a pain the ass because they require root privileges etc. I never use them. Even for creating RPMs (which ostensibly do require root access) I use fakeroot which is just the bees knees.

I have a complete build environment with my own GCC/binutils/make/flex/bison/m4/gdb, all the latest versions, and various different sysroots, and I can build my code soup to nuts including generating .deb and .rpm files with no special privileges (sudo etc.) and zero reliance on the underlying distribution: I may use Mint 14, someone else uses CentOS 6.4, someone else uses Ubuntu 13.10, and they all generate the same code and it works on any distribution RHEL 5.5 or up (or earlier, if I wanted it to). The entire thing can be tarred up and sent around and requires NO special packages installed on or configuration added to the build systems. And a further advantage is that I always know exactly what my system dependencies are (because I have to put them into the sysroot, which, since it's just for compiling/linking, doesn't need any other libraries to run a real system): no opportunities for accidentally pulling other libraries.

Now. My code is more system-level and so doesn't rely on GNOME libraries, etc. However, glibc is actually the LEAST problematic library, because while the glibc folks don't add the ability to generate code that runs on older releases, they DO very carefully and strictly maintain the ability to run code that was compiled for older releases on newer ones. This is a lot of effort on their part but it's hugely important. I can build a single binary that runs on versions of glibc that shipped with distributions 8 years old, or more, and all the ones in between.

If only the OpenSSL people cared so much :-/.

This is the heart of the problem

Posted Feb 13, 2014 22:45 UTC (Thu) by madscientist (subscriber, #16861) [Link]

(when I say "extract the RPMs", above, I only mean the lib and devel RPMs for the libraries you need to link with, of course, not all the RPMs in the distro!! So, for example, glibc, glibc-common, glibc-devel, glibc-headers, zlib, zlib-devel, libtermcap, libtermcap-devel, etc.)

This is the heart of the problem

Posted Feb 14, 2014 0:38 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (17 responses)

> Nevertheless, writing a strlcpy() is trivial. There are even lots of free implementations available for you. If you want versions that take advantage of all the performance tweaks for your particular hardware platform, that is much more work, sure.
We are not talking about 2-line version of strlcpy(), we're talking about optimized SSE-using version with automatic CPU detection. Like the other stuff in glibc.

> Very simple. I've done it many times. You go get copies the RPMs or an installation ISO (if you don't want to pay for RHEL licensing you can get CentOS instead: it's binary compatible).
Now try to compile the recent Boost library with this. I know because I did - it's not trivial.

Then there's CMake which helpfully tries to find packages from the usual include/library directories. At least, the recent versions of pkg-config got better than they used to be.

It's easier to simply do a complete chroot installation (perhaps with fakeroot).

> Now. My code is more system-level and so doesn't rely on GNOME libraries, etc. However, glibc is actually the LEAST problematic library
Wrong. glibc is THE most problematic library in the Linux stack. Bar none.

I can trivially package all other libraries along with my application. ALL of them (well, except maybe some libcrazy that does something perverted with ld).

Let me tell you how I'm developing applications for Windows XP on Win 8.1 (that's 12 years distance in time). I take MSVS 2012, setup a project, set the compatibility level to WinXP and that's it. All the unavailable system functions are automatically masked by WINVER guards and I can trivially bundle the new MSVCRT libraries along with my application.

This is the heart of the problem

Posted Feb 14, 2014 8:55 UTC (Fri) by khim (subscriber, #9252) [Link] (10 responses)

I can trivially package all other libraries along with my application. ALL of them (well, except maybe some libcrazy that does something perverted with ld).

Really? How well libGL packaged with your application works? How about libasound? Or may be libcups? How well they work on different distributions if bundled?

GLibC is one of the least problematic libraries: if you exclude purely-computatiuonal libraries which you can bundle with your app then you'll find out that GLibC is in fact one of the most stable and most well-supported libraries.

Then there's CMake which helpfully tries to find packages from the usual include/library directories.

Then it must be fixed, right? It's hard to blame GLibC for bugs in some separate, totally unrelated package.

I admit that Microsoft does better job, but then again: it, too, recommented to use old version of it's product to target old versions of Windows not too long ago, right? In reality to support XP “under the hood” MSVC does the same thing GLibC recommends to do BTW. It's called multitargeting there. The only advantage is that in Windows world these things are well-integrated and rarely bite while in Linux world everyone does this in their own separate fashion and there are rough edges everywhere. But this is problem if Linux fragmentation, not GlibC design.

This is the heart of the problem

Posted Feb 14, 2014 10:12 UTC (Fri) by mchapman (subscriber, #66589) [Link] (6 responses)

> Really? How well libGL packaged with your application works? How about libasound?

Bad examples. Plenty of proprietary Linux games are packaged exactly like this.

This is the heart of the problem

Posted Feb 14, 2014 17:51 UTC (Fri) by khim (subscriber, #9252) [Link] (5 responses)

Really? I've seen quite a few of them and they never include libGL. libGLU? Sure. libGLEW? Yes, sometimes. libGL? Never.

This is the heart of the problem

Posted Feb 14, 2014 18:32 UTC (Fri) by mjg59 (subscriber, #23239) [Link]

…and which libGL would you ship? Mesa? Nvidia's? fglrx's?

This is the heart of the problem

Posted Feb 14, 2014 23:52 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

Yes, really. I wrote an application (about 10 years ago) that used Mesa's libGL for high-quality (like 4096x4096 printer-quality images) software rendering.

Of course, packaging libGL that depends on hardware is not really a good idea. But you certainly _can_ do it.

This is the heart of the problem

Posted Feb 15, 2014 11:58 UTC (Sat) by khim (subscriber, #9252) [Link] (1 responses)

If you want to play such a crazy tricks then yoy can distribute GLibC, too. We don't distribute GLibC with our programs here, but we do have a GLibC packages here which are distributed independently from distributions we are using them on and which are used by most of the software developed in house thus I know it's quite possible to do.

This is the heart of the problem

Posted Feb 15, 2014 19:01 UTC (Sat) by nix (subscriber, #2304) [Link]

btw, it's called 'glibc' or 'the GNU C Library". It isn't called 'GLibC'.

This is the heart of the problem

Posted Feb 15, 2014 2:07 UTC (Sat) by mchapman (subscriber, #66589) [Link]

> Really? I've seen quite a few of them and they never include libGL. libGLU? Sure. libGLEW? Yes, sometimes. libGL? Never.

You are correct. I got mixed up between my GL libraries.

This is the heart of the problem

Posted Feb 14, 2014 12:18 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

>Really? How well libGL packaged with your application works?
Works just fine.

>How about libasound?
Ditto.

>Or may be libcups? How well they work on different distributions if bundled?
Haven't tried that.

>I admit that Microsoft does better job, but then again: it, too, recommented to use old version of it's product to target old versions of Windows not too long ago
AFAIR, that was a beta bug. I distinctly remember using the newest MSVS to compile binaries for WinXP SP2. Ah, here's a blog post that confirms this: http://blogs.msdn.com/b/vcblog/archive/2012/06/15/1032064...

> In reality to support XP “under the hood” MSVC does the same thing GLibC recommends to do BTW.
Nope. Multitarteting is --sysroot analog, but it's rarely necessary.

And I stand by my statement - glibc is literally the worst library in Linux. Bar none.

This is the heart of the problem

Posted Feb 14, 2014 17:45 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

AFAIR, that was a beta bug.

Not even close. MSVC 2012 RTM had no support for XP at all.

Ah, here's a blog post that confirms this: http://blogs.msdn.com/b/vcblog/archive/2012/06/15/1032064...

Have you actually tried to read said blog post? Later this fall, Microsoft will provide an update to Visual Studio 2012 that will enable C++ applications to target Windows XP means “you can target Windows XP just yet”, you know.

Nope. Multitarteting is --sysroot analog, but it's rarely necessary.
Well, Microsoft claims that it is necessary if you want to target WIndows XP. You can use some clever tricks to target Windows XP without doing that but they are even more fragile than multitargeting.

And I stand by my statement - glibc is literally the worst library in Linux. Bar none.

Well, Ok. Since by now we know that you are all too ready call black white and even ready to “stand by your statements” by adding links which state that black is indeed black and white is indeed white further discussion is completely useless. Because it's quite hard to discuss anything with someone who blatantly ignores logic.

This is the heart of the problem

Posted Feb 14, 2014 23:48 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

>Have you actually tried to read said blog post? Later this fall, Microsoft will provide an update to Visual Studio 2012 that will enable C++ applications to target Windows XP means “you can target Windows XP just yet”, you know.
Look at the date. It was in 2012 and the update with XP support was issued in 2013. It was true that initially MSVC12 did not support XP.

>Well, Ok. Since by now we know that you are all too ready call black white and even ready to “stand by your statements” by adding links which state that black is indeed black and white is indeed white further discussion is completely useless. Because it's quite hard to discuss anything with someone who blatantly ignores logic.
It's a matter for another thread, but I have other arguments (like schizophrenic API).

This is the heart of the problem

Posted Feb 14, 2014 16:46 UTC (Fri) by madscientist (subscriber, #16861) [Link] (2 responses)

> We are not talking about 2-line version of strlcpy(), we're talking about optimized SSE-using version with automatic CPU detection. Like the other stuff in glibc.

I don't understand. Why do you think that the glibc implementation of strlcpy() would have all that stuff in it? Did someone write that implementation and contribute it and it was rejected? If so then you can just use that yourself. If not, then most likely if strlcpy() _was_ added to glibc it would just be the simple, unoptimized version you would write yourself. The _fact_ is that there's absolutely nothing preventing anyone from using strlcpy() in their program, any time they want. There's no need for glibc to include it. And even if some newer version of glibc DID include it, how many years would it be before you could rely on it being present on your target systems and remove your own implementation?

> Now try to compile the recent Boost library with this. I know because I did - it's not trivial.

We use Boost 1.53. I don't know if you consider that "recent" or what might have broken in Boost since then, but I don't remember having any particular problem when I built it (for third party stuff like this we build it once then use the results). We also use ICU, MPIR, google coredumper, unixODBC, etc.: we also compile all our build tools like this so we don't have to worry about the version of Linux on our build systems, so GCC, gdb, binutils, bison, flex, make, m4, cmake, git, python, etc.

> I can trivially package all other libraries along with my application. ALL of them (well, except maybe some libcrazy that does something perverted with ld).

I never pack system libraries with my software. If I did, then I'd be on the hook for tracking and providing security updates any time _any_ of those libraries has a flaw. That's too much work, and no matter how good we are at it, it's still not as good as using the upstream distro's security updates.

I admit, as I said before, I rely on a relatively modest set of low-level system libraries that are fairly stable: no Qt, GTK, etc. But we're talking about glibc here anyway. Glibc uses weak binding, versioned symbols, etc. to avoid having to change the major soversion which makes it completely painless to support. Libraries like libcrypto, libssl, libcurl, etc. just bump the major soversion and so are a SERIOUS pain in the rear end.

> Let me tell you how I'm developing applications for Windows XP on Win 8.1 (that's 12 years distance in time).

Lord. Windows is by FAR our most difficult port. For a DLL we build that we need to work with older versions of PHP we have to actually install MSVC 2008 and perform a completely different build with that. Not to mention all the effort needed to install software onto every Windows build machine by hand, set obscure system configuration options to deal with crashes on the headless/unattended build systems, etc. etc. And of course there's all the frustration, expense, and annoyance with getting licensing for all of it.

> Wrong. glibc is THE most problematic library in the Linux stack. Bar none.

You're wrong. Simple as that.

This will be my last post on the matter since we obviously cannot make progress here. I'm going back to using my straightforward environment that requires no chroot, su, or sudo, is 100% relocatable wherever you want to install it, can be bundled into a tarball and copied to a different system without extra packages or configuration, and generates the same result, where that result runs on every Linux system from Red Hat EL 5 or newer and does not require any statically-linked libraries and does not ship any system libraries and still works fine, and which I've been using to ship enterprise products to real customers for years.

This is the heart of the problem

Posted Feb 14, 2014 20:20 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

> I don't understand. Why do you think that the glibc implementation of strlcpy() would have all that stuff in it?

glibc does load-time resolution of symbols based on CPU capabilities. I call memcmp in my code, but it is resolved to __memcmp_sse4_1 at runtime. Sure, a compat/strlcpy.h works, but it wouldn't be as fast as __strlcpy_ssse3. Looking at my glibc, it seems that there would be optimized versions of strlcpy:

0000000000103cf0 T __strcpy_chk
00000000000930d0 T __strcpy_small
0000000000086070 t __strcpy_sse2
0000000000098ae0 t __strcpy_sse2_unaligned
0000000000153250 t __strcpy_ssse3

This is the heart of the problem

Posted Feb 14, 2014 20:34 UTC (Fri) by madscientist (subscriber, #16861) [Link]

> glibc does load-time resolution of symbols based on CPU capabilities.

Yes, I definitely know that. But my point is that the existence of these highly optimized versions of strcpy() doesn't magically get you a highly optimized version of strlcpy(). There are plenty of functions in glibc which have not been optimized in this way, and there is no particular reason to believe that strlcpy() would magically get this support just by being included.

Someone needs to do that work, and even if the glibc developers could be convinced to add strlcpy() etc. to the library that's no guarantee they'd be motivated to generate highly-optimized versions.

And if someone WAS motivated to generate highly-optimized versions, then those could be used external to glibc just as easily (in fact, possibly more easily since you wouldn't have to worry about which glibc version you were using).

This is the heart of the problem

Posted Feb 14, 2014 21:35 UTC (Fri) by jimparis (guest, #38647) [Link] (2 responses)

> Wrong. glibc is THE most problematic library in the Linux stack. Bar none.
>
> I can trivially package all other libraries along with my application. ALL of them (well, except maybe some libcrazy that does something perverted with ld).

Joey Hess seems to claim that shipping glibc with your application is also not hard: https://joeyh.name/blog/entry/completely_linux_distributi...

This is the heart of the problem

Posted Feb 14, 2014 23:50 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

You can realistically do it only by using Docker-like containers or some deep linker magic.

This is the heart of the problem

Posted Feb 17, 2014 21:52 UTC (Mon) by jimparis (guest, #38647) [Link]

Does that mean that you didn't read the link, that his approach doesn't work, or that you just don't consider it realistic? It seems pretty straightfoward to me: ship glibc and ensure that you execute your binaries through the shipped dynamic linker. I'm curious to know why you don't consider that acceptable?

This is the heart of the problem

Posted Feb 14, 2014 17:28 UTC (Fri) by hummassa (subscriber, #307) [Link] (5 responses)

> I have a complete build environment with my own GCC/binutils/make/flex/bison/m4/gdb, all the latest versions, and various different sysroots, and I can build my code soup to nuts including generating .deb and .rpm files with no special privileges (sudo etc.) and zero reliance on the underlying distribution

would you please be so kind and teach us how you do that?

This is the heart of the problem

Posted Feb 14, 2014 23:00 UTC (Fri) by madscientist (subscriber, #16861) [Link] (4 responses)

I'm not sure what "that" you are referring to?

I build GCC et.al. by running configure and make, as per the README. I have a fancy makefile I wrote that does this. I create the sysroots as described in my previous posts: I get the RPMs and use rpm2cpio to unpack them into a directory, then when I compile and link my code I add the --sysroot=<dir> flag pointing to that directory. You might have to go through the sysroot and replace absolute symlinks with relative symlinks; I have a little Perl script that does this but it's easy enough to do by hand. You only need to do it once (or anyway, once every time you change your base operating system).

This is the heart of the problem

Posted Feb 15, 2014 13:22 UTC (Sat) by hummassa (subscriber, #307) [Link] (3 responses)

I think the "that" I referred to was "a complete build environment with my own GCC/binutils/make/flex/bison/m4/gdb, all the latest versions, and various different sysroots" etc.

IOW, do you already, and if not, would you/ could you PLEASE (pretty please with chantilly and a cherry on top) publish your specialized makefile and perl script that enables you to do all those things authomagically? :D

I can throw in the promise of a beer (or other appeasing beverage/ edible with one hour of my arguably agreeable companionship and conversation) whether and whenever we are geographically close...

This is the heart of the problem

Posted Feb 15, 2014 14:44 UTC (Sat) by madscientist (subscriber, #16861) [Link]

Since I did this as part of my work I'm not free to just publish it all without asking. But I'll check and see what can be done.

This is the heart of the problem

Posted Feb 15, 2014 19:03 UTC (Sat) by nix (subscriber, #2304) [Link]

You might want to look at the buildroot project.

This is the heart of the problem

Posted Feb 16, 2014 18:36 UTC (Sun) by liw (subscriber, #6379) [Link]

I have something vaguely similar as my personal CI system.
http://git.liw.fi/cgi-bin/cgit/cgit.cgi/jenkinstool/ if you want to
have look.

Beware: I wrote this for myself. It's not designed to be clean or
easy to setup, and I don't particularly want to support it, but you
can have a look.

Basically: a) a VM that runs Jenkins b) a VM for each environment (stable vs unstable, i386 vs amd64) where builds happen c) an apt repository (using reprepro) to hold packages built by CI d) successfully built packages get uploaded to the repo automatically e) builds use the repo in addition the Debian once and so groups of dependent packages can be built within the CI system.

Some day I may have time (or someone to pay me to hvae time) to make
that easier for others to do.


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds