OT: dependency and ABI mismanagement

Posted Sep 27, 2010 13:25 UTC (Mon) by paulj (subscriber, #341)
In reply to: OT: dependency and ABI mismanagement by cmccabe
Parent article: Carrez: The real problem with Java in Linux distros

Soname versioning is pretty crude.

ELF formats have more fine-grained versioning systems now that can be granular at a symbol level, and even allow 1 library to support multiple *incompatible* versions of a symbol at the same time. This is used, e.g., by Glibc. The symbol versions are specified in a linker map. There are very few reasons to break compatibility with previous, stable interface once you use symbol versioning.

It's "just" a question of spending a little extra time on paying attention to the compatibility issues.

OT: dependency and ABI mismanagement

Posted Sep 29, 2010 6:22 UTC (Wed) by jamesh (guest, #1159) [Link] (6 responses)

ELF symbol versioning only really works well when the entry points you are versioning use simple types as inputs and outputs.

As the types get more complex, it becomes harder to support multiple versions of those data structures within the same library. And object oriented designs that make use of inheritance are probably the most difficult (as found on most C++ projects and glib GObject based projects).

It is possible to design the data structures so they can be extended without breaking compatibility (e.g. GTK has maintained ABI compatibility for quite a long time, despite extensive changes to some widgets), but people don't always follow those guidelines, or get things wrong the first time. If the library is high enough in the stack with few users, the developers might not even feel it worth while to plan for future changes and just bump the soname when needed.

OT: dependency and ABI mismanagement

Posted Sep 29, 2010 11:01 UTC (Wed) by paulj (subscriber, #341) [Link] (5 responses)

Yeah, data type compatibility takes a little bit of care.

Still though, even if you must introduce a new, incompatible data type, there's still no reason why your library can not support the same (runtime) call using both old and new data types. The old symbol, to which old binaries bind, simply expects the old data type - and the new symbol the new data type.

Compile time backward compatibility may require a little extra work again, of course, but its not rocket science.

Have a read of the Solaris and GNU linker documentation on symbol version scripts/maps. It's a pretty powerful mechanism. Solaris makes heavy use of them, given Suns' strong desires to have binary compatibility as a feature (also requires carefully documenting what guarantees you make for the stability of interfaces, and testing). It's a pretty old feature too...

The trouble is, this is effort and work that benefits unknown users - it doesn't immediately benefit the developer much and its not much fun. So it usually simply doesn't get done in the free software world, other than exceptions like, e.g., projects where there's a corporate sponsor to provide a focus on customer experience.

From a quick look with readelf at my local GTK+ library, it doesn't look like GTK+ uses symbol versioning.

OT: dependency and ABI mismanagement

Posted Oct 5, 2010 22:00 UTC (Tue) by nix (subscriber, #2304) [Link]

Another exception: projects where ABI breaks would be hell to fix, and where everyone knows that. (e.g. glibc.)

I suspect the X libraries don't use it because *introducing* symbol versioning would itself break the ABI, and the current ABI of libX11 et al predates symbol versioning by years.

OT: dependency and ABI mismanagement

Posted Oct 6, 2010 9:31 UTC (Wed) by jamesh (guest, #1159) [Link] (3 responses)

While providing two versions of the function is certainly possible, I think you are misjudging the effort to do so.

Using GTK as an example, if we made an incompatible change to the GtkWidget structure, there are 179 gtk_widget_* symbols that we'd need two versions for.

Now every widget in the library (and every library built on top of GTK) embeds the GtkWidget structure, so we would need two versions in order to support both the old and new API. There is more than 3800 symbols in GTK alone, so this is not a small job. If my application uses any libraries built on top of GTK, they will need to be updated in a similar way to support the new GtkWidget data type if I am to use the new version of the API.

Granted the problems are smaller if the incompatible change is made further down the class hierarchy, but I hope this explains why symbol versioning isn't the first tool developers reach for in these cases.

OT: dependency and ABI mismanagement

Posted Oct 6, 2010 19:32 UTC (Wed) by paulj (subscriber, #341) [Link] (2 responses)

For that kind of wide-ranging change, I agree it might be classed as one of the "few legit reasons to break compatibility". Though, you could take the approach of making a new library; compiling old and new separately; then combining 2 objects for each library into one appropriately symboled one (I think you might have to write your own ELF tool to remap the symbols of one library though, if such doesn't already exist).

However, you're mistaken that the applications must be updated. You can retain *source* compatibility even if binary compatibility is broken in some way. I.e. you're assuming the old GtkWidget definition retains that name and the new one gets a new name. However, you can also rename the _old_ definition (GtkWidgetOld or GtkWidget2_2) and have the new definition use the well-known source-level name, presuming it is still source compatible. With linker maps you can direct old apps (compiled with the old GtkWidget definition, i.e. GtkWidgetOld when it was still called GtkWidget) to functions that expect GtkWidgetOld. There is no requirement at all that the name of the structure be the same in the caller and the function, it's not part of the ABI.

Solaris made heavy use of this kind of stuff to preserve runtime compatibility even as data types could be changed incompatibly without changing source-level name (be it changed by default, or changed in the presence of whatever feature selection defines). Glibc probably does too.

OT: dependency and ABI mismanagement

Posted Oct 7, 2010 9:58 UTC (Thu) by jamesh (guest, #1159) [Link]

My point about applications was when they make use of multiple libraries providing GTK widgets.

Since this thread started on evolution-data-server, consider an application using one of the widgets from the libedataserverui library. If GTK broke the ABI of GtkWidget, you would need a new version of the libedataserverui widgets to use with the new GtkWidget ABI. If that was not available, then your app would need to use the old GTK ABI.

As I said previously these sort of ABI breakages are quite painful, so effort is made to avoid them. For GTK itself we've maintained compatibility for 8 years, so it certainly is possible (although is a bit painful at times).

Would it be nice if evolution-data-server went through fewer ABI breakages? Sure, but I don't think symbol versioning would solve the problem.

OT: dependency and ABI mismanagement

Posted Oct 9, 2010 22:14 UTC (Sat) by nix (subscriber, #2304) [Link]

glibc can do it *better* than Solaris, as the GNU linkers have default symbol versions, which Solaris does not (or didn't last I checked, but that was way back in the Solaris 9 days, maybe they added it in 10 or OpenSolaris?)