LWN: Comments on "Carrez: The real problem with Java in Linux distros"

OT: dependency and ABI mismanagement

nix — Sat, 09 Oct 2010 22:14:58 +0000

glibc can do it *better* than Solaris, as the GNU linkers have default symbol versions, which Solaris does not (or didn't last I checked, but that was way back in the Solaris 9 days, maybe they added it in 10 or OpenSolaris?)

OT: dependency and ABI mismanagement

jamesh — Thu, 07 Oct 2010 09:58:19 +0000

My point about applications was when they make use of multiple libraries providing GTK widgets.

Since this thread started on evolution-data-server, consider an application using one of the widgets from the libedataserverui library. If GTK broke the ABI of GtkWidget, you would need a new version of the libedataserverui widgets to use with the new GtkWidget ABI. If that was not available, then your app would need to use the old GTK ABI.

As I said previously these sort of ABI breakages are quite painful, so effort is made to avoid them. For GTK itself we've maintained compatibility for 8 years, so it certainly is possible (although is a bit painful at times).

Would it be nice if evolution-data-server went through fewer ABI breakages? Sure, but I don't think symbol versioning would solve the problem.

OT: dependency and ABI mismanagement

paulj — Wed, 06 Oct 2010 19:32:24 +0000

For that kind of wide-ranging change, I agree it might be classed as one of the "few legit reasons to break compatibility". Though, you could take the approach of making a new library; compiling old and new separately; then combining 2 objects for each library into one appropriately symboled one (I think you might have to write your own ELF tool to remap the symbols of one library though, if such doesn't already exist).

However, you're mistaken that the applications must be updated. You can retain *source* compatibility even if binary compatibility is broken in some way. I.e. you're assuming the old GtkWidget definition retains that name and the new one gets a new name. However, you can also rename the _old_ definition (GtkWidgetOld or GtkWidget2_2) and have the new definition use the well-known source-level name, presuming it is still source compatible. With linker maps you can direct old apps (compiled with the old GtkWidget definition, i.e. GtkWidgetOld when it was still called GtkWidget) to functions that expect GtkWidgetOld. There is no requirement at all that the name of the structure be the same in the caller and the function, it's not part of the ABI.

Solaris made heavy use of this kind of stuff to preserve runtime compatibility even as data types could be changed incompatibly without changing source-level name (be it changed by default, or changed in the presence of whatever feature selection defines). Glibc probably does too.

Carrez: The real problem with Java in Linux distros

marcH — Wed, 06 Oct 2010 09:33:03 +0000

> This is not the same.

Please elaborate?

> You could also argue any random network protocol is documented because one can use network analysis tools on it to find its innards.

No, this is not the same :-)

You are not doing any kind of justice to DependencyFinder with this poor analogy. Unlike DependencyFinger, a network analysis tool:
- requires you to run the software
- will provide very incomplete "documentation"

I think I might guess what your point is but please elaborate better.

OT: dependency and ABI mismanagement

jamesh — Wed, 06 Oct 2010 09:31:33 +0000

While providing two versions of the function is certainly possible, I think you are misjudging the effort to do so.

Using GTK as an example, if we made an incompatible change to the GtkWidget structure, there are 179 gtk_widget_* symbols that we'd need two versions for.

Now every widget in the library (and every library built on top of GTK) embeds the GtkWidget structure, so we would need two versions in order to support both the old and new API. There is more than 3800 symbols in GTK alone, so this is not a small job. If my application uses any libraries built on top of GTK, they will need to be updated in a similar way to support the new GtkWidget data type if I am to use the new version of the API.

Granted the problems are smaller if the incompatible change is made further down the class hierarchy, but I hope this explains why symbol versioning isn't the first tool developers reach for in these cases.

Carrez: The real problem with Java in Linux distros

nix — Tue, 05 Oct 2010 22:18:58 +0000

An example with a missing bracket so it doesn't compile ;P

OT: dependency and ABI mismanagement

nix — Tue, 05 Oct 2010 22:00:45 +0000

Another exception: projects where ABI breaks would be hell to fix, and where everyone knows that. (e.g. glibc.)

I suspect the X libraries don't use it because *introducing* symbol versioning would itself break the ABI, and the current ABI of libX11 et al predates symbol versioning by years.

Carrez: The real problem with Java in Linux distros

nix — Mon, 04 Oct 2010 11:30:11 +0000

Well, when libraries gain new facilities, those facilities have to be used by something or they might as well not be there. So even if everything is backward-compatible, you have to version the libraries to indicate the addition of new facilities, *and* you have to version the applications to indicate that they require a library with those new facilities. Giving both the same version number and releasing them simultaneously seems like a not-too-confusing-to-users way to do that.

Carrez: The real problem with Java in Linux distros

HelloWorld — Mon, 04 Oct 2010 01:13:41 +0000

You asked for arguments, so here's one. You're "stupid" example is inefficient, yes. But it's also buggy. Consider what would happen if there were two even numbers in a row.

That's actually a good point, that wouldn't have happened with an iterator.

And that's a good reason to use commonly used libraries for these kinds of things, of course. I'm sure you know that, but you don't appear to be a Java developer, so I wouldn't expect you to know what those libraries are. The parent post mentioned Apache Commons. That would work, but their Collections framework doesn't do generics. These days Guava would be used.

This kind of stuff makes sense in theory, but as I pointed out elsewhere in this thread, the Java language makes this kind of stuff way too hard and verbose. Programmers will just continue writing loops as long as it stays that way (It works perfectly well in more expressive languages though). Perhaps Java 8 will remedy this situation.

Carrez: The real problem with Java in Linux distros

sbishop — Mon, 04 Oct 2010 00:18:42 +0000

You asked for arguments, so here's one. You're "stupid" example is inefficient, yes. But it's also buggy. Consider what would happen if there were two even numbers in a row.

And that's a good reason to use commonly used libraries for these kinds of things, of course. I'm sure you know that, but you don't appear to be a Java developer, so I wouldn't expect you to know what those libraries are. The parent post mentioned Apache Commons. That would work, but their Collections framework doesn't do generics. These days Guava would be used.

Carrez: The real problem with Java in Linux distros

marcH — Sun, 03 Oct 2010 10:20:04 +0000

> which makes it so verbose that it's pointless.

You really have no idea how much money boilerplate code can make.

If someone don't want to believe something then he'll not believe something

khim — Sun, 03 Oct 2010 09:07:52 +0000

You'll have to give reasons for this if you want anybody to believe this who doesn't already.

Sorry, but no. You can not disprove honest belief so easily: that's why we hava Judaism, Christianity, Islam, Buddhism, etc. I've already said why Java is "better": instead of small number of constructs which can be combined in many ways you have lots and lots of stuff which can be combined in limited way. This is easier to use if you try to randomly connect things and see what will finally work.

Those developers shouldn't have been hired in the first place. After all, this is what the probation time is good for: keeping lunkheads out of your team.

Welcome to real world. Yes, probation time helps, but not absolutely: if you reject candidates one after another then HR department will eventually decide that it's better to fire you (citing unaccommodating as reason) rather then tolerate problems you cause (they lose bonuses if you reject lunkheads, you know).

So in the end only absolutely hopeless imbeciles are fired. Everyone else stays and you must accommodate them somehow. Especially if they have good recommendations and finished prestigious universities.

This is like letting a would-be carpenter use a hand saw instead of a power saw since he might cut his fingers off instead of telling him that he's simply not the right person for the job.

Yes and no. It's more like giving him power saw but only asking him to produce identical straight planks - and this is exactly what happens in real life. If you take a look on what is in stores then you'll see that work of genuine carpenters-designers are rare and expensive. Most furniture are mass-produced using the same design for thousands of items. Why? It's easy: it's cheaper this way. People who can design exist, often they can even use power saw, but they are rare and expensive. So furniture production is optimized for lunkheads: they know how to operate power saw or even more complex equipment, but they don't know how to design things and how to cope with timber blemishes (unless they've gotten explicit instructions). So other people (designers) are doing all planning work and low-lever people are just following the rules.

Java world is designed for just such a use. And the main ingredients are not even language capabilities but IDE capabilities - but they are enabled by language capabilities. I've conducted hundred of interviews and "Java developers" are worst bunch: often they can not even write simple programs like you've shown above without help of IDE! And they will happily write O(N^2) solution you've presented - if it'll work then Ok, if it'll not work - some higher level person will speedup the thing by factor ten and will get separate bonuses for that work.

Business world is not about good code. It's about strict adherence to specifications - and Java helps immensely here.

That's true, but you don't have a choice.

HelloWorld — Sun, 03 Oct 2010 01:23:23 +0000

Yes - and it means most "business programs" are horrible. But with Java they work. With other, less restricting languages, they don't.

You'll have to give reasons for this if you want anybody to believe this who doesn't already.

The problem here is the fact that it's usually very hard to fire bad developer from "big business" so they must be used somehow - and Java is the best language for that.

Those developers shouldn't have been hired in the first place. After all, this is what the probation time is good for: keeping lunkheads out of your team. Instead, you're even trying to accommodate those lunkheads. This is like letting a would-be carpenter use a hand saw instead of a power saw since he might cut his fingers off instead of telling him that he's simply not the right person for the job.

That's true, but you don't have a choice.

khim — Sat, 02 Oct 2010 22:50:06 +0000

You could say that smart people could figure out the design and let grunt programmers implement it. But I don't think that works well in practice. The devil is in the details, and for a bad programmer, it's easy to write a horrible program from a design he didn't really understand.

Yes - and it means most "business programs" are horrible. But with Java they work. With other, less restricting languages, they don't. The problem here is the fact that it's usually very hard to fire bad developer from "big business" so they must be used somehow - and Java is the best language for that. C# is not because it's nicer language and actually includes many higher-level features!

Sorry, but this is NOT Turing-complete VM

khim — Sat, 02 Oct 2010 22:35:46 +0000

For Turing-completeness you need some kind of ENDLESS memory. It may be tape (as in Turing or Post machines), memory (well, most CPUs have finite memory so they are fully not Turing-complete - yet 2^64 is usually "good enough") or just a few counters (three is enough - but they must be unbound!).

DVD VM has very limited memory so it's not Turing-complete...

Carrez: The real problem with Java in Linux distros

marcH — Sat, 02 Oct 2010 15:12:38 +0000

Sorry if you felt "trolled" but I actually never meant to discuss this *at all*. I just meant to briefly share a number of converging experiences, and such a forum is the appropriate place for that. Formally demonstrating the conclusions of these experiences with hard facts is not possible in such a forum, anything serious would require an entire book. We all understood that your experience differs, thanks.

Carrez: The real problem with Java in Linux distros

HelloWorld — Fri, 01 Oct 2010 14:55:06 +0000

And how does Java help with all that? Does Java magically create good database designs? I don't think so. And it probably won't magically make your program scalable and all that. So what it boils down to is that to make fast software, you need smart developers.

You could say that smart people could figure out the design and let grunt programmers implement it. But I don't think that works well in practice. The devil is in the details, and for a bad programmer, it's easy to write a horrible program from a design he didn't really understand.

Carrez: The real problem with Java in Linux distros

HelloWorld — Fri, 01 Oct 2010 14:11:43 +0000

IMO, both are not good. Iterator.remove() should be used.

Please, if you make such statements, provide arguments for them. Otherwise, it's just a waste of time for both of us. Also, as far as I can see, there is no analogon to the removeRange method that takes iterators instead of indices.

Or, if there is such a utility function in Apache commons, it should be used.

I don't think there is. It's basically impossible to abstract this kind of things in Java enough to make it useful. A method that removes even numbers from a list isn't really that useful, since one day you may want to remove odd numbers or primes or negatives or whatever. So what you need to do is pass a small chunk of code that determines whether an element will be removed. The best Java will give you is an anonymous inner class, which makes it so verbose that it's pointless.

filter(new Predicate<Integer>() { public boolean doIt(Integer i) { return i % 2 == 0; } }, someArrayList);

Carrez: The real problem with Java in Linux distros

HelloWorld — Fri, 01 Oct 2010 11:38:01 +0000

This discussion is going nowhere. Making claims and not providing any arguments has a name. It's called trolling.

Carrez: The real problem with Java in Linux distros

HelloWorld — Fri, 01 Oct 2010 11:19:41 +0000

Given that this is a discussion about poor developers writing poor code in Java, an example in Java seemed fit.

Anyway, if you want an example in a functional language, here is how you'd do it in Haskell:

removeEvenNumbers = filter odd

Carrez: The real problem with Java in Linux distros

marcH — Fri, 01 Oct 2010 10:29:20 +0000

Without really looking at your example in detail, I like the fact that you first show off with some functional-speak, and then demonstrate an example with for loops.

Carrez: The real problem with Java in Linux distros

marcH — Fri, 01 Oct 2010 10:22:38 +0000

> Oh by the way, you still haven't told why you think that Java makes stupid things harder.

Yes I left this as an exercise (hint: cheat and use the Internet).

Carrez: The real problem with Java in Linux distros

jschrod — Fri, 01 Oct 2010 07:14:54 +0000

> See above for a trivial O(n^2) algorithm that is being written all the time.

See my answer there.

> Also, if the problem size is small anyway, why worry about performance at
> all and not just write the stuff in some other, more productive language?

First of all, a large problem size does by no means equate with complex algorithms. E.g., in a current project of us, phone events of 40,000,000 customers must be rated and billed. Processing each event is simple, no algorithmic complexity at all. The challenge is an architectural design that supports or even furthers horizontal scalibity, i.e., the ability to distribute the processing over many servers.

Second, in business applications performance issues are more often caused by bad interface decisions, I/O problems, and bad database designs than by algorithms. Business applications are not about computing, they are about moving data from A to B.

As I wrote, complexity and challenges for business applications are found in the requirement and design phases, not in the programming phases. Actually, also in the testing phase if one goes beyond module unit tests and tests if a system really delivers the functionality and serviceability the customer *needs* (that's most of the time not the same he's ordered).

Third, I have yet to meet a non-toy project -- let's say 50+ people working on it -- where one is allowed to select its implementation language on purely technical grounds. I hope I don't have to spell out all the reasons for that. In the real world, implementation language(s) selection is as much a management and political decision as it is a technical decision, for better and worse.

Carrez: The real problem with Java in Linux distros

jschrod — Fri, 01 Oct 2010 07:00:46 +0000

IMO, both are not good. Iterator.remove() should be used.
Or, if there is such a utility function in Apache commons, it should be used.

Carrez: The real problem with Java in Linux distros

HelloWorld — Thu, 30 Sep 2010 19:15:37 +0000

Oh by the way, you still haven't told why you think that Java makes stupid things harder. I think you're underestimating the dumb developers' imagination when it comes to doing dumb things.

Carrez: The real problem with Java in Linux distros

HelloWorld — Thu, 30 Sep 2010 18:54:07 +0000

See above for a trivial O(n^2) algorithm that is being written all the time.

Also, if the problem size is small anyway, why worry about performance at all and not just write the stuff in some other, more productive language?

Carrez: The real problem with Java in Linux distros

HelloWorld — Thu, 30 Sep 2010 18:51:58 +0000

Algorithms are being written all the time. It's about trivial stuff like removing even numbers from a list. Don't tell me this doesn't happen in the real world, _especially_ when dumb programmers (who don't know about higher-order functions like filter) and dumb languages (like Java, which also doesn't know about higher-order functions) are involved.

A dumb programmer would do this:

void removeEvenNumbers(ArrayList<Integer> l) {
    for(int i = 0; i < l.size(); ++i)
        if (l.get(i) % 2 == 0)
            l.remove(i);
}

A smart one would do this:

void removeEvenNumbers(ArrayList<Integer> l) {
    int j = 0;
    for(int i = 0; i < l.size(); ++i) 
        if (l.get(i) % 2 != 0)
            l.set(j++, l.get(i))
    l.removeRange(j, l.size()
}

Now please don't tell me that this kind of stuff isn't done in the real world.

Carrez: The real problem with Java in Linux distros

jackb — Thu, 30 Sep 2010 15:25:28 +0000

It wants to be able to auto-update itself.

Carrez: The real problem with Java in Linux distros

rwmj — Thu, 30 Sep 2010 15:15:40 +0000

"write access to its own executable". WTF? What on earth for???

Carrez: The real problem with Java in Linux distros

BenHutchings — Thu, 30 Sep 2010 15:09:23 +0000

DVD menus are also controlled by a program running on a virtual machine. The VM is Turing-complete but is limited to 128 instructions per menu, 16 16-bit general-purpose registers and no memory.

Carrez: The real problem with Linux distros

nim-nim — Thu, 30 Sep 2010 13:53:04 +0000

Why should I take that personally ?
I described a mode of development I don't agree with (piling up components with random versions and minimal code tracking and then using many testers to try identify the exposed bugs and fix them before the customer notices).

And if you don't think that's how most big Java projects are "qualified", you've never looked below the shiny presentation veneer. They rely heavily on brute-force testing (either cheap testers or the same simulated via loadrunner and friends). That works to limit the number of obvious problems (and customer complains) on the exact configuration tested. But that's never been a good way to validate a design is solid enough to withstand the test of time, and can evolve later when requirements change. As soon as the environment changes, even minimally, everything needs to be redone (write once, run once, not everywhere)

Carrez: The real problem with Java in Linux distros

jschrod — Thu, 30 Sep 2010 10:38:03 +0000

In the Real World(tm), a.k.a. business programs, programmers don't have the chance to select between an O(n^2) and an O(n) algorithm[*], they use libraries and frameworks that supply such algorithms.

Obviously you don't develop in a business context. There, the interesting parts of software development are in the requirement, architectural, and testing parts of a project, not in any algorithm designs which are 99.999% of the time *VERY* boring, being `get data, change some attribute, move it somewhere else, finished'.

[*] Being pedantic, there are few algorithms anyhow, where one *could* make such a choice -- most of the time, it's O(n^2) vs. O(n \log n). Equally well, it often doesn't matter at all, because the factors that O-notation abstracts away, and also $n$ that needs to be handled are too small anyhow. There's a reason why DEK does full analysis in his books, and not just O notation.

Carrez: The real problem with Linux distros

jschrod — Thu, 30 Sep 2010 09:49:54 +0000

> Testing is prohibitively expensive but it also does not require very
> qualified people.

Don't take that personally, but I hope I won't have to use a product where you were responsible for test resources' allocation.

With a perhaps better understood analogy, your statement reads like the same mindset as `UI design changes don't require very qualified people, everybody can change some dialogs.'. *shudder*...

Carrez: The real problem with Java in Linux distros

jschrod — Thu, 30 Sep 2010 09:41:39 +0000

NB: AFAIR, current SUSE Linux has no roots in Slackware. While S.u.S.E. GmbH originally distributed German translations of Slackware, base of their own distribution was Jurix from Florian. This was 4.2, in 1996 or so, and was my 1st SUSE installation. (I used original Slackware before.)

Carrez: The real problem with Java in Linux distros

nim-nim — Thu, 30 Sep 2010 09:15:16 +0000

>> With the java way a user of 100 java apps will have 100 versions

> But every one of those 100 apps probably works with respect to the
> features that the app developer tested. Nobody else has gone in and
> changed something and screwed it up.

And what happens when the app needs to be updated to add new customer-requested features? It can't because the local fork of Java deps is missing the features that were added in the real upstream version. So everything needs to be re-done.

The Java way does not solve anything it just delays the payback (That's one reason big Java apps cost astronomical prices, and have slow release cycles. The language is nice and should enable cheaper/faster development, but the abysmal release practices add up in multiple ways in the long term)

Carrez: The real problem with Java in Linux distros

nim-nim — Thu, 30 Sep 2010 09:04:00 +0000

This is not the same.

You could also argue any random network protocol is documented because one can use network analysis tools on it to find its innards.

Carrez: The real problem with Java in Linux distros

marcH — Thu, 30 Sep 2010 06:24:46 +0000

> (Why do I even have to explain this?)

Because writing "enterprise" software, of which you apparently know little (lucky you), is almost not about algorithms but mainly about large amounts of boring boilerplate code for persistence, serialization and user interfaces. Have a look at what J2EE is (now JEE).

In the real world, grunts do not get to spend time reading Knuth and carefully select algorithms.

Carrez: The real problem with Java in Linux distros

HelloWorld — Wed, 29 Sep 2010 22:07:13 +0000

Writing fast software requires some understanding of smart datastructures, algorithms, complexity theory etc., and poor developers generally don't have that kind of understanding. Since to my knowledge Java can't magically turn an O(n^2) algorithm into a O(n) one, Java programs written by poor developers will perform poorly.

(Why do I even have to explain this?)

Carrez: The real problem with Java in Linux distros

marcH — Wed, 29 Sep 2010 21:33:56 +0000

> AFAIK, java doesn't support any sane dependency extraction, without which it is more difficult to provide good runtime dependencies.

http://depfind.sourceforge.net/
Not good enough?

Carrez: The real problem with Java in Linux distros

marcH — Wed, 29 Sep 2010 21:20:06 +0000

> It's easy to make such bold statements.

It is easy but only after years of experience.

> And given the complete lack of rational arguments, it's just as easy not to believe them.

Granted.

Note: for an equally bold, but much more simplistic statement see the grand-grand-parent post.