C library system-call wrappers, or the lack thereof

Posted Nov 16, 2018 21:53 UTC (Fri) by lgerbarg (guest, #57988)
In reply to: C library system-call wrappers, or the lack thereof by mathstuf
Parent article: C library system-call wrappers, or the lack thereof

No problem. You are pointing -bundle_loader at a dylib, it should be pointed at an MH_EXECUTABLE. Maybe I misunderstood, but I thought you said the symbols were export from the python executable itself?

C library system-call wrappers, or the lack thereof

Posted Nov 16, 2018 22:06 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (3 responses)

The symbols are provided to the module by loading a libpython.dylib first. The executable has barely any symbols in it at all. Giving the executable complains that the symbols aren't defined.

C library system-call wrappers, or the lack thereof

Posted Nov 16, 2018 22:08 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (2 responses)

And it is libpython.dylib that does the dlopen, not the executable. I guess the flag is meant for static compilation?

C library system-call wrappers, or the lack thereof

Posted Nov 18, 2018 21:12 UTC (Sun) by lgerbarg (guest, #57988) [Link] (1 responses)

The flag is meant to allow plugins to refer back to symbols exported by an application in order to support a plugin API. Where the dlopen() happens doesn't really impact it, but the fact that theAPI is exported by a dylib rather than the interpreter itself is what complicates the situation. I could go into a bunch of historical reasons for why it behaves that way, but the short answer is that it dates back to classic macOS and the plugin mechanisms used there.

Now that I understand what you are trying to do a bit more clearly (sorry about the confusion) -bundle_loader is not appropriate unless you can re-export the symbols from the main executable. I think that would be the best option from a technical perspective, but it is probably unreasonable since it would require changes and back ports to all of the python interpreters.

Assuming that you need to get this to work without making changes to python itself. I think you should pass ld "-undefined dynamic_lookup." It is kind of gross, but it should work for your use case. If there is ever a desire to improve this behavior for future pythons there changes to the interpreter that would make modules work better:

1) Reexporting the symbols from libpython out of the python interpreter itself and using -bundle_loader
or
2) Renaming or symlinking the library exporting the API to have an unversioned name (like "libpython.dylib") and then having python set an LC_RPATH to the directory containing the dylib/symlink. Then modules could link to them via "@rpath/libpython.dylib" .

C library system-call wrappers, or the lack thereof

Posted Nov 20, 2018 15:16 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

Hmm. OK. So how is this meant to work with something like ParaView which supports different applications (for different use cases) and uses shared libraries to hold the actual API (and I mean that there can be 100+ of them)? What's the best way to reexport symbols of all linked libraries from an executable?

> I think you should pass ld "-undefined dynamic_lookup." It is kind of gross

That works for today. And projects which don't only target macOS are almost never going to do what seems like a bad legacy behavior. The "gross" part of this solution is what led to the patch to ld64.

> Renaming or symlinking the library exporting the API to have an unversioned name (like "libpython.dylib") and then having python set an LC_RPATH to the directory containing the dylib/symlink. Then modules could link to them via "@rpath/libpython.dylib" .

The name of the file doesn't matter. The binary must be edited using `install_name_tool -id` to change what gets embedded in the linking binary. And RPATH on macOS is awful. It only applies if explicitly requested and if something says "use me via rpath", no tools (I know of) add the paths automatically and they must be manually added.