| |||||||||||||
Wheeler: Insecure open source software libraries?Wheeler: Insecure open source software libraries?Posted Apr 7, 2012 12:33 UTC (Sat) by khim (subscriber, #9252)In reply to: Wheeler: Insecure open source software libraries? by rqosa Parent article: Wheeler: Insecure open source software libraries?
Oh, and in the other thread you were complaining that the library APIs change too often, but those "current" library versions you're saying here that Ubuntu ought to include probably have backwards-incompatible API changes. You can't have it both ways. Of course not! The biggest problem with distro-supplied libraries is not even the fact that they provide obsolete version of libraries. This is similarly to what other, saner, platforms are doing. The biggest problem is that you have to work with unknown version of library. One distribution will have libpng 1.2, another one will have libpng 1.5, one will include some fixes backported from older version of library, another will have bugs. WTF? Why should I support all that zoo? It's often easier and simpler to bundle library with the application. OS should mostly provide facilities I can not easily bundle with the application, not provide bazillion of libraries with constantly changing ABI.
(Log in to post comments)
Wheeler: Insecure open source software libraries? Posted Apr 7, 2012 12:55 UTC (Sat) by rqosa (subscriber, #24136) [Link] > One distribution will have libpng 1.2, another one will have libpng 1.5, one will include some fixes backported from older version of library, another will have bugs. If a distro-provided library has bugs, then that's not the application developer's fault, and shouldn't be treated as such. > WTF? Why should I support all that zoo? You shouldn't. Instead, you should just pick an API-version of the library that's still maintained, and support only that one. The distributions will generally include any still-maintained version of a library as long as there is other software in the distribution that depends on it, and when they drop it, it's time for any remaining apps to either migrate away from the obsolete library or else get dropped by the distributions. > OS should mostly provide facilities I can not easily bundle with the application That sounds like Mac OS X, where you often have to update the whole OS to the next version in order to run the one program you want. No thanks!
Wheeler: Insecure open source software libraries? Posted Apr 7, 2012 13:20 UTC (Sat) by khim (subscriber, #9252) [Link] If a distro-provided library has bugs, then that's not the application developer's fault, and shouldn't be treated as such. As user I don't care if the root cause of problem lies with Intel, Debian or Google. If application does not work then it's author is to blame. Unless application explicitly says that my hardware or software is deficient I'll blame application author. You may say it's not fair, but this is how Joe Average thinks. That sounds like Mac OS X, where you often have to update the whole OS to the next version in order to run the one program you want. No thanks! Sorry, but no. In Mac OS X you need to update the whole version of OS to run newer version of application 1 time out of 10, on Linux it's more like 5 times out of 10 (and 4 times out of 10 you can try to run an application yet it'll exhibit some strange errors which will be blamed on the distribution and the solution offered will be “upgrade your OS, then try again”… 2 times out of 10 said solution will not help). Thnx, but no, thnx.
Wheeler: Insecure open source software libraries? Posted Apr 7, 2012 14:54 UTC (Sat) by rqosa (subscriber, #24136) [Link] > this is how Joe Average thinks. So what? Let them blame whoever they want to — it doesn't mean they're right. > on Linux it's more like 5 times out of 10 Not in my experience. On Linux, one can usually track down the dependencies for a particular program and install them in an out-of-the-way location (e.g. not in /usr) in order to get it to run. Good luck trying the same thing on Mac OS X…
Wheeler: Insecure open source software libraries? Posted Apr 7, 2012 21:40 UTC (Sat) by drag (subscriber, #31333) [Link]
> That sounds like Mac OS X, where you often have to update the whole OS to the next version in order to run the one program you want. No thanks!
The model to compare Linux package management solution is other, more effective, solutions like with Android and iOS.
The entire userland is versioned with a static API between minor versions. Not just individual libraries, but the application API is presented as a complete set.
Application developers are then free to drop their application, built the way they want to support it, into the ready-to-install pool of applications presented to the user via the package management system.
Then the package management system presents packages to users based on what versions of APIs that their OS supports. So you could have applications built for Linux versions years ago that are still installable as long as the API version they need is still supported.
With Linux desktop things need to be a bit more fined grained then that that, but I think it should be very possible if you present standard APIs in blocks.
For example a Gnome desktop may have these levels of API support:
Linux Kernel --> userland = 'set in stone API' (as per kernel current policy).
Core plumbing ---> The init system, low level libaries, system utilities, udev, server stuff. Distros can use whatever they want as long as it supports a 'common denominator' set of APIs that people can depend on. These would be free to update, break compatibility when necessary, every 3-4 years or so. Which is on par with competing proprietary desktops.
Generic desktop personality ---> Stuff that can be ignored if you are doing a server install. X libraries, Alsa, common media libraries, CUPS etc. Versioned in parallel with 'Core plumbing' with 3-4 years between breaking things.
Gnome personality ---> Ultimately up to project consideration how they want to support the API and what APIs need to be supported, but it must be versioned.
Ideally you want each distro manage the 'kernel', 'core plumbing', and 'generic desktop personality' how they see fit.
'Gnome/KDE/XFCE/etc personalities' should be managed and be compiled by upstream. That is Fedora and Debian would use the same binaries and same libraries built by Gnome.
No compiling and customization should be done on the distribution ideally. If distributions want branding and incorporate their trademarks into the desktop then they need to work with upstream on how to do that through themeing packages. That way the distributions can be kept honest.
That way we can go along way to making things a lot friendlier and would be a big first step into making Linux a lot easier for people to install their games and such.
Wheeler: Insecure open source software libraries? Posted Apr 8, 2012 8:18 UTC (Sun) by rqosa (subscriber, #24136) [Link] > The model to compare Linux package management solution is other, more effective, solutions like with Android and iOS. "More effective", yeah right. The insecure-bundled-libraries problem is endemic to those systems, just like it has traditionally been on that other "standard API" platform that's called Java SE. > I think it should be very possible if you present standard APIs in blocks. Every platform with "standard APIs" that's ever been tried has caused the exact sort of problem that this article describes. The platform APIs will never be enough for all apps, so the developers must turn to third-party libraries — and there's no way to use these without bundling them.
Wheeler: Insecure open source software libraries? Posted Apr 8, 2012 10:36 UTC (Sun) by rqosa (subscriber, #24136) [Link] > just like it has traditionally been on that other "standard API" platform that's called Java SE. And furthermore, more recently some Java developers are adopting something that's very similar to Linux-style package management (complete with transitive dependency resolution), because the standard API just isn't enough.
Wheeler: Insecure open source software libraries? Posted Apr 8, 2012 11:03 UTC (Sun) by alankila (subscriber, #47141) [Link]
I doubt this is actually very similar. Maven makes dependency management a breeze, but these libraries are versioned and people do specify the exact versions of the libraries they want in their pom.xml files.
Wheeler: Insecure open source software libraries? Posted Apr 8, 2012 19:00 UTC (Sun) by rqosa (subscriber, #24136) [Link] It's not all that different. In package management systems, packages also can specify a particular version (or minimum/maximum version) to depend on. (Though it's true that often they don't actually do this — likely because it's becoming common for library packages to include an "API major version number" right in the package name, e.g. "freetype2" or "kdelibs3" or "allegro4". This way allows for installing multiple major versions of a library in parallel, and also ensures that the most recent API-compatible versions of all dependencies for all installed apps are present.)
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 2:03 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
>It's not all that different. In package management systems, packages also can specify a particular version (or minimum/maximum version) to depend on.
In Maven you _must_ specify a particular version (or version range). Your POM won't be valid without it.
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 8:08 UTC (Mon) by micka (subscriber, #38720) [Link]
It's possible to specify a version range for a dependecy, like [1.0,1.5] or even [5.6,], though that's not exactly common practice.
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 8:12 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
I've seen it used with stuff like log4j or Spring where binary compatibility is kept between major versions.
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 2:01 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
Nope, it's most definitely NOT very similar.
Maven has two modes of operation for libraries: SNAPSHOT and fixed versions. In fixed version mode Maven simply downloads the referenced libraries. They are immutable, they are _not_ updated automatically and can be cached in a local repository (like http://www.jfrog.com/products.php ).
In the SNAPSHOT mode Maven tracks the most recent version of referenced libraries. This mode IS similar to what Linux does. However, this mode is used by all real project only for development, and quite explicitly NOT for releases.
I.e. you develop with SNAPSHOT versions, tracking the newest library versions. Then you fix the library versions (there's even a plugin for that: http://maven.apache.org/plugins/maven-release-plugin/inde... ) and push it for QA.
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 21:26 UTC (Mon) by cmccabe (guest, #60281) [Link]
In the real world, nobody uses Maven's ability to download the most recent versions of referenced libraries-- except maybe with libraries that are part of their own project. The problem is that Java has nothing like an API level. So if you tell Maven "just get the latest version" you don't know if you're getting something that's API-compatible with your code. So you have mysterious build breakages on random developer machines, and eventually someone edits the .pom file to use a fixed version. This is why in Hadoop, we use fixed versions for all external libraries.
This problem was solved with shared libraries in Linux a long time ago by introducing the concept of an API level. New versions of shared libraries can describe what previous revisions they are API-compatible with, This is the funny three-tuple you see at the end of shared library names. However, nothing similar to this exists in Java-- at least to my knowledge; I'd be happy to be corrected here.
This endemic "pinning" of dependencies to specific version numbers in Java can cause unexpected problems. For example, if your project tries to rely on version 1.23 of library A and version 4.56 of library B, but library B itself tries to pin version 7.89 of library A-- what do you do? Sometimes including two different versions of the same library in the project can cause problems at runtime. For an example of this, see https://issues.apache.org/jira/browse/HADOOP-8104
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 22:16 UTC (Mon) by rqosa (subscriber, #24136) [Link] > However, this mode is used by all real project only for development, and quite explicitly NOT for releases. I'm aware that when building a release with Maven, it makes a .jar which contains all dependencies. But that's beside the point — what I meant was that even though Java SE has a fairly large standard "platform API", it's still not enough for all apps, and that's why Java developers now use a package repository with a constantly-changing selection of libraries (unlike the set-in-stone "platform API") and also a tree of dependencies together with dependency resolution in the package manager (unlike the supposedly "more effective" Android Market and iOS App Store, which lack dependency resolution and therefore require all packages to depend on the "platform API" only).
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 22:27 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
>I'm aware that when building a release with Maven, it makes a .jar which contains all dependencies.
No it doesn't.
>But that's beside the point — what I meant was that even though Java SE has a fairly large standard "platform API", it's still not enough for all apps, and that's why Java developers now use a package repository with a constantly-changing selection of libraries (unlike the set-in-stone "platform API")
Most libraries in Java have excellent ABI compatibility. Besides, changes in Maven packages DO NOT affect dependent projects. I.e. right now my Clementine player on Ubuntu doesn't work because of some DBUS accessibility-related changes (and I'm too tired to check why).
Maven doesn't work that way. It emphasizes repeatable builds, i.e. I can checkout a release from 2 years ago from our repository and build a byte-perfect copy with all the dependencies preserved. In fact, Maven actually complains if you do something that breaks repeatable builds.
Debian packages provide nothing like this.
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 19:54 UTC (Tue) by rqosa (subscriber, #24136) [Link] > No it doesn't. According to here, > Debian packages provide nothing like this. That's because old package versions don't remain in the repository, and it doesn't allow installing multiple versions of the same package in parallel. (However, like I already said, different API-versions of the same dependency will often have different package names from each other, thus allowing them to be installed in parallel, and also allowing a package to depend on "the most recent compatible version" of each of its dependencies.) But again, this is all beside the point. drag was saying that there should be a "versioned […] static API" for the entire set of userland APIs on Linux; I responded that Java already tried this approach, which is what's to blame for the problem (insecure bundled libraries) that the original article is about, and the existence of Maven shows that Java developers are migrating away from the notion of a single "static API" to a system of multiple separately-versioned library APIs with a tree of dependencies among them (i.e. exactly what isn't done on iOS and Android, the OSes drag says have "more effective" package management).
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 20:05 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
>According to here, mvn package does bundle dependencies when building a package that has <packaging>war</packaging> in its POM.
Nope, maven-war-plugin does it. By itself Maven doesn't do any packaging.
Besides, even maven-war-plugin doesn't pack all dependencies into one JAR file, it simply copies them into WEB-INF/lib directory of a web application.
>That's because old package versions don't remain in the repository, and it doesn't allow installing multiple versions of the same package in parallel.
Yup. That's why it's different from Maven.
>I responded that Java already tried this approach, which is what's to blame for the problem (insecure bundled libraries) that the original article is about, and the existence of Maven shows that Java developers are migrating away from the notion of a single "static API" to a system of multiple separately-versioned library APIs with a tree of dependencies among them
Nope they don't. Maven is a freaking _build_ tool, it's NOT a package management tool. So it HAS to work with versions and dependencies.
You mean it's similar because it has a notion of dependency tree? Duh. Even 'make' has it.
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 21:17 UTC (Tue) by rqosa (subscriber, #24136) [Link] > Nope, maven-war-plugin does it. That fine distinction doesn't matter here. > Besides, even maven-war-plugin doesn't pack all dependencies into one JAR file, it simply copies them into WEB-INF/lib directory of a web application. But the WEB-INF/lib directory is in the WAR file — therefore the applications and its dependencies are bundled into a single archive file. > Maven is a freaking _build_ tool, it's NOT a package management tool. It's obviously more than just a build tool; there are plenty of build tools (probably most of them) that have neither transitive dependency resolution for external dependencies nor the ability to download said external dependencies from a central repository. > You mean it's similar because it has a notion of dependency tree? No. It's similar because it uses transitive dependency resolution to determine what packages from a central package repository (one that's hosted on a single public server on the Internet and used by default on each Maven installation, that is; not just something that's on the local HDD) are required by the project to be built, then downloads them from there and uses them to build and/or run the project. > Duh. Even 'make' has it. Make's dependency tree doesn't extend beyond the files that are part of the project to be built; it doesn't have any real notion of external dependencies. Higher-level build tools like cmake and autoconf/automake have a notion of immediate external dependencies, but no tree of external dependencies (and therefore no transitive dependency resolution for external dependencies) and no ability to retrieve dependencies from a central repository. (Though it's possible to build an external-dependency-resolution system on top of make, as the FreeBSD ports tree shows — but I consider the FreeBSD ports tree to be a package manager too.)
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 22:07 UTC (Tue) by khim (subscriber, #9252) [Link] I still don't understand what you are trying to discuss here. Yes, maven is similar to Linux distribution - so what? Linux distributions are quite good for development. They are abysmal for deplayment. And here Maven offers the solution: prepare package suitable for use with a given version of J2EE ABI. Administrator does not need to know what dependencies are there in a given package - s/he only need to know version of J2EE needed by a given WAR and version of J2EE supported by given server. Easy, simple, unambiguous. In a sense this approach nicely combines Linux distribution development advantages with Appstore-style deployment advantages. The only problem: it's tied to Java which by itself does not form suitable desktop environment.
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 23:02 UTC (Tue) by rqosa (subscriber, #24136) [Link] > I still don't understand what you are trying to discuss here. I've explained this twice already, here: even though Javaand here: drag was saying that there should be a "versioned […] static API" for the entire set of userland APIs on Linux; I responded that Java already tried this approach, which is what's to blame for the problem (insecure bundled libraries) that the original article is about, and the existence of Maven shows that Java developers are migrating away from the notion of a single "static API" to a system of multiple separately-versioned library APIs with a tree of dependencies among them (i.e. exactly what isn't done on iOS and Android, the OSes drag says have "more effective" package management).TL;DR version: Any "standard platform API" will never be enough for all apps on the platform; this is why developers using Java (which has a very large "standard platform API") now use a central package repository with an ever-increasing selection of libraries and a dependency tree. > In a sense this approach nicely combines Linux distribution development advantages with Appstore-style deployment advantages. I suppose it does help solve the problem of insecure-bundled-libraries, because it makes it easier to track the newest library versions. But still, it seems that the insecure-bundled-libraries problem persists, because Java library developers tend to follow the convention that "release version number == API version number" and break API compatibility often (leaving the old API versions unmaintained), which leads to "endemic "pinning" of dependencies to specific version numbers".
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 23:56 UTC (Tue) by rqosa (subscriber, #24136) [Link] And also, when using Maven to build apps as WARs (i.e. "Appstore-style deployment"), there's still the issue of multiple copies of the same library (leading to wasted storage space and RAM, and to a single library bugfix requiring an update for all apps using it).
Wheeler: Insecure open source software libraries? Posted Apr 11, 2012 9:56 UTC (Wed) by alankila (subscriber, #47141) [Link]
As an application developer these are acceptable tradeoffs. Disk space is very cheap, and jar files can not truly be shared between jvm processes anyway because the jar file is not useful per se, the class files are usually deflate-compressed, and they contain symbolic references that are resolved at class loading time to speed up interpretation. Then there's the whole jit compilation step whose result is affected by the surrounding code that uses the library.
|
|||||||||||||