automated testing

Posted Jun 15, 2012 10:56 UTC (Fri) by Wol (subscriber, #4433)
In reply to: automated testing by rwst
Parent article: Calling for a new openSUSE development model

You miss something that's bitten me on several occasions ...

Running the latest version of SuSE at the time (and this is why I switched to gentoo ...) I needed the latest version of lilypond. Wouldn't build.

They'd upgraded their dependencies - lily does tend to be a bit bleeding edge on occasion. So you get nasty circular loops where updating a package means you need to upgrade the libraries - which breaks other packages. So you upgrade them, rinse and repeat.

Individuals and rolling distros can handle this (fairly) easily. Enterprise, hardened, distros it's a lot more difficult.

Cheers,
Wol

to post comments

automated testing

Posted Jun 15, 2012 16:21 UTC (Fri) by k3ninho (subscriber, #50375) [Link] (21 responses)

Surely this one's bitten enough people that versioned libraries are available somewhere in the Linux or Posix specs? No? Oh well.

K3n.

automated testing

Posted Jun 15, 2012 16:29 UTC (Fri) by cmccabe (guest, #60281) [Link] (20 responses)

Yes, Linux has shared library versioning. That doesn't mean the distro is going to actually use it.

automated testing

Posted Jun 15, 2012 17:08 UTC (Fri) by nix (subscriber, #2304) [Link] (18 responses)

More to the point, you need not just shared library versioning but pervasive use of versioned symbols and no ABI breaks for this. This is so far from being true that it is almost laughable: hardly any upstreams bother with versioned symbols in any real sense (using them to avoid ABI breaks), and when distros try to version their own libraries' symbols despite upstream not doing so, it causes compatibility problems with upstream and half the time they have to remove the versioning again.

automated testing

Posted Jun 15, 2012 17:30 UTC (Fri) by drag (guest, #31333) [Link]

This is why it's useful for application developers and users to ignore the whole mess completely and just make sure to bundle everything they need that has a significant chance of breaking with the application.

automated testing

Posted Jun 15, 2012 21:06 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (14 responses)

> hardly any upstreams bother with versioned symbols in any real sense (using them to avoid ABI breaks)

I'd like to use it, but AFAICT, there's darn near 0 support for C++ and it makes the headers not work with other compilers in the same way. For C++, sure, I can type out the mangled name, but that's a pain and should be unnecessary. I'd really just like to decorate functions and methods with a VERSION_SYMBOL("real_symbol", VERSION_STRING) right on the function. I was also unable to find out how method support works with it. Not to mention that I also need to support MSVC for this project which isn't going to handle headers with versioned symbols in it.

Of course, my search-fu may have just come up with nothing for these problems.

automated testing

Posted Jun 16, 2012 13:40 UTC (Sat) by nix (subscriber, #2304) [Link] (13 responses)

Yeah, you're mightily screwed with C++ as well. I forgot that case.

There should indeed be an __attribute__((__symbol_version__,"...")), though you'd need to consider the case of symbols having more than one version too (e.g. default versions, though possibly you could have a __default_symbol_version__ attribute for this). But there isn't.

You don't need anything like this for C++

Posted Jun 19, 2012 20:12 UTC (Tue) by khim (subscriber, #9252) [Link] (12 responses)

C++ has it's own portable way of doing this:

namespace my_super_duper_library {

  inline namespace version_0 {

    func foo(int);

  }

}

in next version

namespace my_super_duper_library {

  inline namespace version_1 {

    func foo(long);

  }

}

Users always use my_super_duper_library::foo and everyone is happy.

You don't need anything like this for C++

Posted Jun 19, 2012 20:28 UTC (Tue) by nix (subscriber, #2304) [Link] (11 responses)

And use 'using' instead of a default symbol version? I suppose that would work. If you put a using inside each symbol namespace, you can have the effect of symbol nodes depending on other symbol nodes as well.

(In any case, mathstuf was wrong and so was I: GNU ld supports extern "C++" in version scripts to force appropriate C++ mangling of names, using the name mangler in libbfd.)

You don't need anything like this for C++

Posted Jun 19, 2012 21:08 UTC (Tue) by joib (subscriber, #8541) [Link] (9 responses)

No need for 'using' tricks. I guess you're missing the point, which is that in C++11 there is a new feature called "inline namespaces" which allows one to designate a default namespace. So in the provided example, my_super_duper_library::foo generates a symbol reference to my_super_duper_library::version_1::foo (mangled, obviously). Code which was compiled against the old version of the library will keep on using my_super_duper_library::version_0::foo. Thus providing a way for the library developer to enhance the API while keeping ABI compatibility.

You don't need anything like this for C++

Posted Jun 19, 2012 21:41 UTC (Tue) by nix (subscriber, #2304) [Link]

Aaah! Nifty. Yes, I did miss the 'inline'.

You don't need anything like this for C++

Posted Jun 20, 2012 0:58 UTC (Wed) by mathstuf (subscriber, #69389) [Link] (2 responses)

For another project, I had come up with a namespacing strategy for it (it was API compatible via a namespace protocol = protocol_v1 or something similar for whenever things bumped), but I had thought for some reason that it wouldn't work similarly to symbol versioning. Since the compiler would, even in that case, expand the namespaces out so that things are always the same, it would work. It'd be nice if the symbol versioning documentation would mention this as a solution for C++ (the inline namespace is nice, but the stuff I've been writing tends to need to work on RHEL5 and MSVC still).

You don't need anything like this for C++

Posted Jun 20, 2012 1:08 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

You can sort of emulate inline namespaces by adding lots of "using" statements to outer namespace, but it's not pretty.

You don't need anything like this for C++

Posted Jun 20, 2012 14:54 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Well, the idea is that protocol_v2 would do the using stuff from protocol_v1 for the compatible parts, then replace the functions that need replacing. The default protocol would be updated with the namespace protocol = ; statement.

You don't need anything like this for C++

Posted Jul 3, 2012 19:49 UTC (Tue) by cmccabe (guest, #60281) [Link] (4 responses)

One thing I'm curious about. Are you intended to duplicate all of your declarations in both namespaces, or are you intended to do something like this:

namespace version1 {
  void foo(int64_t a);
  void bar(int);
}

namespace version2 {
  void foo(float b);
  using namespace version1;
}

If you are intended to do something like what I've just described, there are obvious problems with implicit type conversions. For example, foo(0x123456789LL) will give you the version1 function, whereas foo(0x1234) will give you the version2 function, which is probably not what you want.

Ultimately, this seems like just another variant of "new version, new datatypes / functions," which you can easily do in C or older versions of C++. You can even hide the old version declarations in the header file by default using macros. I think FUSE does this-- if you don't set FUSE_VERSION to an "old" number, you can't see the old deprecated APIs in the header file.

You don't need anything like this for C++

Posted Jul 4, 2012 18:16 UTC (Wed) by jwakely (subscriber, #60262) [Link] (3 responses)

foo(0x123456789LL) will give you the version1 function

Are you sure? I don't understand your example, as there isn't actually an inline namespace there, but if version2 is meant to be inline, then both calls to foo would get the float version. i.e. adding your version2 would have change the API in incompatible ways, as the original foo would no longer be used by anyone. You could do that, but it's probably not what you want.

If you use a using declaration instead of a using directive then both overloads are found and overload resolution picks the best one.

Using declarations work better with inline namespaces. Consider version 1 of the API:

namespace api {
  inline namespace v1 {
    void foo(int) {
      // implementation
    }
    void bar() {
      // implementation
    }
  }
}

Code that calls api::foo(1) links to api::v1::foo(int).

Later, version 2 is released:

namespace api {
  namespace v1 {  // not inline now
    // as before
  }
  inline namespace v2 {
    void foo(int i) {
      // improved v2 implementation
    }
    void foo(float) {
      // new impl for float
    }
    using v1::bar;
  }
}

Code that was linked to api::v1:foo still finds that symbol, with the same implementation. Code that is compiled against the new version and calls api::foo(1) links to api::v2::foo(int). Both symbols can coexist in the library (just as with ELF symbol versioning.) Code that calls api::bar() gets api::v1::bar() in both cases, so there's no duplication (except for adding a using declaration to the v2 namespace.)

You don't need anything like this for C++

Posted Jul 5, 2012 22:20 UTC (Thu) by cmccabe (guest, #60281) [Link] (2 responses)

> Are you sure?

Well, let's try it.

#include <stdint.h>
#include <stdio.h>
namespace version1 {
    void foo(int64_t a) {
        printf("version1 foo\n");
    }
}
inline namespace version2 {
    void foo(float b) {
        printf("version2 foo\n");
    }
    using namespace version1;
}
int main(void) {
    foo(0x123);
    return 0;
}

gcc 4.6.2 gives me "foo.cpp:22:14: error: call of overloaded ‘foo(int)’ is ambiguous". So it's not as bad as I thought-- just a compile-time failure, not silently doing the wrong thing.

[snip example]
Yes, you could add an explicit "using" line for each version1 symbol, to avoid this problem.

This all seems very similar to just adding a "using namespace version2" to your header file. I guess some people are worried about the potential for conflicts with whatever is already in your namespace, though.

Overall it I give it a "meh." It's not that useful, but it's not seriously harmful either. Which means it's doing better than a lot of language extensions. On the other hand, I do think the copy constructor stuff will be useful (so you see, I'm not just a complete curmudgeon) :)

You don't need anything like this for C++

Posted Jul 9, 2012 10:46 UTC (Mon) by jwakely (subscriber, #60262) [Link] (1 responses)

It's not really "very similar" to using namespace version2, an inline namespace is a lot more transparent so users never need know it exists (unless they look at the mangled symbols in their objects.)

A using-directive just makes names visible, but they are not treated as first-class members of the namespace containing the using-directive and will not be found by qualified name lookup if there are declarations of the same name in the namespace containing the using-directive. That's not the case for inline namespaces.

Templates defined in an inline namespace can be instantiated and specialized as though they are members of the enclosing namespace, not possible via a using-directive.

An inline namespace is an associated namespace of types in its enclosing namespace and vice versa, so it plays nicely with ADL.

Don't dismiss the feature because you don't understand it yet.

You don't need anything like this for C++

Posted Jul 10, 2012 21:31 UTC (Tue) by cmccabe (guest, #60281) [Link]

It seems like the best practice is to put all of your C++ library declarations into one namespace-- for the sake of argument, call it foolib.

It seems like what you're recommending then is that you have something like this:

namespace foolib {
   namespace version1 {
       ...
   }
   inline namespace version2 {
       ...
   }
}

Given that use-case, I don't think we need to care about the scenario where there are declarations of the same name in the namespace containing the using-directive-- neither the library user nor the library implementor should be doing that.

Templates defined in an inline namespace can be instantiated and specialized as though they are members of the enclosing namespace, not possible via a using-directive.

That's fair. I suppose template specialization is where the "using directive" approach really breaks down.

The cost of this feature is that debugging becomes (even more) difficult, since you search the symbol table of your application for foolib::doit and you don't find it. Instead you find N versions and have to decide which one you're really using. But of course, none of us here writes bugs, so that shouldn't be a problem :)

You don't need anything like this for C++

Posted Jun 20, 2012 0:54 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

I saw the scripts do support C++, but I'd like the versions to live next to the code (just like visibility is done). The external scripts will just drift relative to the code.

automated testing

Posted Jun 17, 2012 0:22 UTC (Sun) by cmccabe (guest, #60281) [Link] (1 responses)

> More to the point, you need not just shared library versioning but
> pervasive use of versioned symbols and no ABI breaks for this. This is so
> far from being true that it is almost laughable: hardly any upstreams
> bother with versioned symbols in any real sense (using them to avoid ABI
> breaks), and when distros try to version their own libraries' symbols
> despite upstream not doing so, it causes compatibility problems with
> upstream and half the time they have to remove the versioning again.

You don't need versioned symbols. You just need to be sane about not breaking backward compatibility. So if you have a struct foo that you exposed to the world in version 1.0, in the new version you create a struct foo2 that has your new stuff in it. Or better yet, don't expose foo to the world-- expose an opaque pointer and accessor functions instead. If worst comes to worst, you can bump up the major API number and allow the old and the new version to be installed simultaneously. A lot of libraries have gone down this path, and it's not that bad.

Of course, versioned symbols are available for those who really, really want them. But each platform has its own subtly different implementation, and it's really difficult to actually get it right. As usual, simple is better.

Binary compatibility is do-able in C++ too. See http://techbase.kde.org/Policies/Binary_Compatibility_Iss...

automated testing

Posted Jun 17, 2012 13:22 UTC (Sun) by nix (subscriber, #2304) [Link]

You don't need versioned symbols. You just need to be sane about not breaking backward compatibility.

Yeah, but if you want real backward compatibility, which includes not breaking people merely because you did need to change an API, then you need versioned symbols.

But most people these days don't seem to give a damn about ABI or API compatibility. Wild change and close coupling is the order of the day. Hallelujah :((

(Hm, when did I become a grumpy old man?)

(As for the C++ binary compatibility constraints: they're horrendous, and I remain amazed that KDE can stick to them.)

automated testing

Posted Jun 21, 2012 16:28 UTC (Thu) by dgm (subscriber, #49227) [Link]

> Yes, Linux has shared library versioning. That doesn't mean the distro is going to actually use it.

I cannot believe you. Is really the bar _that_ low? We should expect MUCH more from distros. And developers.