| |||||||||||||
Wheeler: Insecure open source software libraries?Wheeler: Insecure open source software libraries?Posted Apr 8, 2012 8:18 UTC (Sun) by rqosa (subscriber, #24136)In reply to: Wheeler: Insecure open source software libraries? by drag Parent article: Wheeler: Insecure open source software libraries?
> The model to compare Linux package management solution is other, more effective, solutions like with Android and iOS. "More effective", yeah right. The insecure-bundled-libraries problem is endemic to those systems, just like it has traditionally been on that other "standard API" platform that's called Java SE. > I think it should be very possible if you present standard APIs in blocks. Every platform with "standard APIs" that's ever been tried has caused the exact sort of problem that this article describes. The platform APIs will never be enough for all apps, so the developers must turn to third-party libraries — and there's no way to use these without bundling them.
(Log in to post comments)
Wheeler: Insecure open source software libraries? Posted Apr 8, 2012 10:36 UTC (Sun) by rqosa (subscriber, #24136) [Link] > just like it has traditionally been on that other "standard API" platform that's called Java SE. And furthermore, more recently some Java developers are adopting something that's very similar to Linux-style package management (complete with transitive dependency resolution), because the standard API just isn't enough.
Wheeler: Insecure open source software libraries? Posted Apr 8, 2012 11:03 UTC (Sun) by alankila (subscriber, #47141) [Link]
I doubt this is actually very similar. Maven makes dependency management a breeze, but these libraries are versioned and people do specify the exact versions of the libraries they want in their pom.xml files.
Wheeler: Insecure open source software libraries? Posted Apr 8, 2012 19:00 UTC (Sun) by rqosa (subscriber, #24136) [Link] It's not all that different. In package management systems, packages also can specify a particular version (or minimum/maximum version) to depend on. (Though it's true that often they don't actually do this — likely because it's becoming common for library packages to include an "API major version number" right in the package name, e.g. "freetype2" or "kdelibs3" or "allegro4". This way allows for installing multiple major versions of a library in parallel, and also ensures that the most recent API-compatible versions of all dependencies for all installed apps are present.)
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 2:03 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
>It's not all that different. In package management systems, packages also can specify a particular version (or minimum/maximum version) to depend on.
In Maven you _must_ specify a particular version (or version range). Your POM won't be valid without it.
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 8:08 UTC (Mon) by micka (subscriber, #38720) [Link]
It's possible to specify a version range for a dependecy, like [1.0,1.5] or even [5.6,], though that's not exactly common practice.
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 8:12 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
I've seen it used with stuff like log4j or Spring where binary compatibility is kept between major versions.
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 2:01 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
Nope, it's most definitely NOT very similar.
Maven has two modes of operation for libraries: SNAPSHOT and fixed versions. In fixed version mode Maven simply downloads the referenced libraries. They are immutable, they are _not_ updated automatically and can be cached in a local repository (like http://www.jfrog.com/products.php ).
In the SNAPSHOT mode Maven tracks the most recent version of referenced libraries. This mode IS similar to what Linux does. However, this mode is used by all real project only for development, and quite explicitly NOT for releases.
I.e. you develop with SNAPSHOT versions, tracking the newest library versions. Then you fix the library versions (there's even a plugin for that: http://maven.apache.org/plugins/maven-release-plugin/inde... ) and push it for QA.
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 21:26 UTC (Mon) by cmccabe (guest, #60281) [Link]
In the real world, nobody uses Maven's ability to download the most recent versions of referenced libraries-- except maybe with libraries that are part of their own project. The problem is that Java has nothing like an API level. So if you tell Maven "just get the latest version" you don't know if you're getting something that's API-compatible with your code. So you have mysterious build breakages on random developer machines, and eventually someone edits the .pom file to use a fixed version. This is why in Hadoop, we use fixed versions for all external libraries.
This problem was solved with shared libraries in Linux a long time ago by introducing the concept of an API level. New versions of shared libraries can describe what previous revisions they are API-compatible with, This is the funny three-tuple you see at the end of shared library names. However, nothing similar to this exists in Java-- at least to my knowledge; I'd be happy to be corrected here.
This endemic "pinning" of dependencies to specific version numbers in Java can cause unexpected problems. For example, if your project tries to rely on version 1.23 of library A and version 4.56 of library B, but library B itself tries to pin version 7.89 of library A-- what do you do? Sometimes including two different versions of the same library in the project can cause problems at runtime. For an example of this, see https://issues.apache.org/jira/browse/HADOOP-8104
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 22:16 UTC (Mon) by rqosa (subscriber, #24136) [Link] > However, this mode is used by all real project only for development, and quite explicitly NOT for releases. I'm aware that when building a release with Maven, it makes a .jar which contains all dependencies. But that's beside the point — what I meant was that even though Java SE has a fairly large standard "platform API", it's still not enough for all apps, and that's why Java developers now use a package repository with a constantly-changing selection of libraries (unlike the set-in-stone "platform API") and also a tree of dependencies together with dependency resolution in the package manager (unlike the supposedly "more effective" Android Market and iOS App Store, which lack dependency resolution and therefore require all packages to depend on the "platform API" only).
Wheeler: Insecure open source software libraries? Posted Apr 9, 2012 22:27 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]
>I'm aware that when building a release with Maven, it makes a .jar which contains all dependencies.
No it doesn't.
>But that's beside the point — what I meant was that even though Java SE has a fairly large standard "platform API", it's still not enough for all apps, and that's why Java developers now use a package repository with a constantly-changing selection of libraries (unlike the set-in-stone "platform API")
Most libraries in Java have excellent ABI compatibility. Besides, changes in Maven packages DO NOT affect dependent projects. I.e. right now my Clementine player on Ubuntu doesn't work because of some DBUS accessibility-related changes (and I'm too tired to check why).
Maven doesn't work that way. It emphasizes repeatable builds, i.e. I can checkout a release from 2 years ago from our repository and build a byte-perfect copy with all the dependencies preserved. In fact, Maven actually complains if you do something that breaks repeatable builds.
Debian packages provide nothing like this.
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 19:54 UTC (Tue) by rqosa (subscriber, #24136) [Link] > No it doesn't. According to here, > Debian packages provide nothing like this. That's because old package versions don't remain in the repository, and it doesn't allow installing multiple versions of the same package in parallel. (However, like I already said, different API-versions of the same dependency will often have different package names from each other, thus allowing them to be installed in parallel, and also allowing a package to depend on "the most recent compatible version" of each of its dependencies.) But again, this is all beside the point. drag was saying that there should be a "versioned […] static API" for the entire set of userland APIs on Linux; I responded that Java already tried this approach, which is what's to blame for the problem (insecure bundled libraries) that the original article is about, and the existence of Maven shows that Java developers are migrating away from the notion of a single "static API" to a system of multiple separately-versioned library APIs with a tree of dependencies among them (i.e. exactly what isn't done on iOS and Android, the OSes drag says have "more effective" package management).
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 20:05 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
>According to here, mvn package does bundle dependencies when building a package that has <packaging>war</packaging> in its POM.
Nope, maven-war-plugin does it. By itself Maven doesn't do any packaging.
Besides, even maven-war-plugin doesn't pack all dependencies into one JAR file, it simply copies them into WEB-INF/lib directory of a web application.
>That's because old package versions don't remain in the repository, and it doesn't allow installing multiple versions of the same package in parallel.
Yup. That's why it's different from Maven.
>I responded that Java already tried this approach, which is what's to blame for the problem (insecure bundled libraries) that the original article is about, and the existence of Maven shows that Java developers are migrating away from the notion of a single "static API" to a system of multiple separately-versioned library APIs with a tree of dependencies among them
Nope they don't. Maven is a freaking _build_ tool, it's NOT a package management tool. So it HAS to work with versions and dependencies.
You mean it's similar because it has a notion of dependency tree? Duh. Even 'make' has it.
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 21:17 UTC (Tue) by rqosa (subscriber, #24136) [Link] > Nope, maven-war-plugin does it. That fine distinction doesn't matter here. > Besides, even maven-war-plugin doesn't pack all dependencies into one JAR file, it simply copies them into WEB-INF/lib directory of a web application. But the WEB-INF/lib directory is in the WAR file — therefore the applications and its dependencies are bundled into a single archive file. > Maven is a freaking _build_ tool, it's NOT a package management tool. It's obviously more than just a build tool; there are plenty of build tools (probably most of them) that have neither transitive dependency resolution for external dependencies nor the ability to download said external dependencies from a central repository. > You mean it's similar because it has a notion of dependency tree? No. It's similar because it uses transitive dependency resolution to determine what packages from a central package repository (one that's hosted on a single public server on the Internet and used by default on each Maven installation, that is; not just something that's on the local HDD) are required by the project to be built, then downloads them from there and uses them to build and/or run the project. > Duh. Even 'make' has it. Make's dependency tree doesn't extend beyond the files that are part of the project to be built; it doesn't have any real notion of external dependencies. Higher-level build tools like cmake and autoconf/automake have a notion of immediate external dependencies, but no tree of external dependencies (and therefore no transitive dependency resolution for external dependencies) and no ability to retrieve dependencies from a central repository. (Though it's possible to build an external-dependency-resolution system on top of make, as the FreeBSD ports tree shows — but I consider the FreeBSD ports tree to be a package manager too.)
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 22:07 UTC (Tue) by khim (subscriber, #9252) [Link] I still don't understand what you are trying to discuss here. Yes, maven is similar to Linux distribution - so what? Linux distributions are quite good for development. They are abysmal for deplayment. And here Maven offers the solution: prepare package suitable for use with a given version of J2EE ABI. Administrator does not need to know what dependencies are there in a given package - s/he only need to know version of J2EE needed by a given WAR and version of J2EE supported by given server. Easy, simple, unambiguous. In a sense this approach nicely combines Linux distribution development advantages with Appstore-style deployment advantages. The only problem: it's tied to Java which by itself does not form suitable desktop environment.
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 23:02 UTC (Tue) by rqosa (subscriber, #24136) [Link] > I still don't understand what you are trying to discuss here. I've explained this twice already, here: even though Javaand here: drag was saying that there should be a "versioned […] static API" for the entire set of userland APIs on Linux; I responded that Java already tried this approach, which is what's to blame for the problem (insecure bundled libraries) that the original article is about, and the existence of Maven shows that Java developers are migrating away from the notion of a single "static API" to a system of multiple separately-versioned library APIs with a tree of dependencies among them (i.e. exactly what isn't done on iOS and Android, the OSes drag says have "more effective" package management).TL;DR version: Any "standard platform API" will never be enough for all apps on the platform; this is why developers using Java (which has a very large "standard platform API") now use a central package repository with an ever-increasing selection of libraries and a dependency tree. > In a sense this approach nicely combines Linux distribution development advantages with Appstore-style deployment advantages. I suppose it does help solve the problem of insecure-bundled-libraries, because it makes it easier to track the newest library versions. But still, it seems that the insecure-bundled-libraries problem persists, because Java library developers tend to follow the convention that "release version number == API version number" and break API compatibility often (leaving the old API versions unmaintained), which leads to "endemic "pinning" of dependencies to specific version numbers".
Wheeler: Insecure open source software libraries? Posted Apr 10, 2012 23:56 UTC (Tue) by rqosa (subscriber, #24136) [Link] And also, when using Maven to build apps as WARs (i.e. "Appstore-style deployment"), there's still the issue of multiple copies of the same library (leading to wasted storage space and RAM, and to a single library bugfix requiring an update for all apps using it).
Wheeler: Insecure open source software libraries? Posted Apr 11, 2012 9:56 UTC (Wed) by alankila (subscriber, #47141) [Link]
As an application developer these are acceptable tradeoffs. Disk space is very cheap, and jar files can not truly be shared between jvm processes anyway because the jar file is not useful per se, the class files are usually deflate-compressed, and they contain symbolic references that are resolved at class loading time to speed up interpretation. Then there's the whole jit compilation step whose result is affected by the surrounding code that uses the library.
|
|||||||||||||