[HN Gopher] Rpath, or why lld doesn't work on NixOS
___________________________________________________________________
Rpath, or why lld doesn't work on NixOS
Author : ingve
Score : 44 points
Date : 2022-03-15 18:11 UTC (4 hours ago)
(HTM) web link (matklad.github.io)
(TXT) w3m dump (matklad.github.io)
| slabity wrote:
| How does an article on NixOS talk about the `rpath` issue without
| also mentioning the `patchelf` utility that NixOS developers
| created to solve this issue? It's a small tool that lets you
| modify ELF executables and binaries. It's also the recommended
| way for NixOS users to modify binaries to work properly.
|
| https://github.com/NixOS/patchelf
| matklad wrote:
| `patchelf` solves a different problem: it fixes up an already
| built binary. Here, I am building the binary myself, so I'd
| rather make that just work without any extra build steps.
| slabity wrote:
| Ah, my bad. I was confused because I never run into linker
| issues when I build my rust (or any other type of) binaries
| on a NixOS system.
|
| In fact, I am running the `evdev` example and I don't get any
| linker errors at all even when I change the linker to LLVM. I
| am using a nightly version of rust though.
| kazinator wrote:
| I'm pretty sure NixOS isn't the only one doing this hack. The
| Yocto build system does something similar. It contains its own
| build-time binary glibc, and patches its tools to point to its
| own internal library installation. Or something like that. In
| effect, Yocto has its own build-time distro, which has to run
| from any filesystem location.
| pvtmert wrote:
| interestingly macOS has a really nice solution called @rpath and
| install_name_tool
|
| basically you build your binary and set a rpath, let's say it is
| /usr/local/lib in your machine
|
| mach-o binary stores: @rpath = /usr/local/lib libxyz =
| @rpath/libxyz.dylib.1
|
| when I want to install those to /opt/something only thing I need
| to do is install_name_tool -add_rpath /opt/something
|
| This will add search directories to binary itself. There are some
| DYLD_* environment variables too but I'm not sure about them...
| (Some are SIP protected by the way)
|
| PS: It may invalidate signed binaries. Again, not tested such use
| cases.
| kazinator wrote:
| That's a pretty poor solution compared to just finding
| libraries relative to the root of the program's installation.
| whateveracct wrote:
| Yep - Windows of all things handles this the best.
|
| You can try to do this with rpath of $ORIGIN. But then you'll
| probably still run into libc issues -.-
|
| Not hard to see why a lot of games focus on Windows and let
| wine/proton handle Linux.
| elikoga wrote:
| Wow, great investigation article
| fanf2 wrote:
| Because I am stubborn, when I am linking with a library installed
| in a nonstandard place, I usually try to get the configure script
| to do the right thing, even though it is not always easy. But,
| just in case I lose the battle, I keep in mind the existence of
| _chrpath_ , a little utility for changing the rpath in an ELF
| binary. Because you use it on the final build artefact, there is
| no way for autoconf or libtool to screw it up.
| kazinator wrote:
| rpath is braindamaged; no program should be using that, and a
| distro build should almost never be inserting such a thing into
| executables.
|
| I suspect that NixOS is playing with this in order to have a
| relocatable install: so that is to say, so that user can install
| NixOS in some subdirectory of a system running some existing
| distro. Any subdirectory, yet so that programs can find their
| libraries.
|
| If I were in this predicament, rather than perpetrating hacks to
| patch the rpaths in binaries, I'd fix the dynamic linker to have
| a better way of locating shared libraries. The linker would
| determine the path from which the executable is being run,
| calculate the sysroot location dynamically, then look for
| libraries in that tree. E.g. /path/to/usr/bin/program would look
| under /path/to/usr/lib and related places.
|
| A possibly nice hack would be to extend the meaning of the rpath
| variable. Give it a syntax, like say that if it starts with @,
| then the rest of it denotes a relative sysroot path fragment.
|
| E.g. the program that gets installed as /path/to/usr/bin/program
| would be built with an rpath of "@/usr/bin". So then the dynamic
| linker sees the @ and does a sysroot calculation. First it strips
| off the basename to get just the directory part
| "/path/to/usr/bin". Then it sees, hey, the suffix of
| "/path/to/usr/bin" matches the "/usr/bin" in the rpath. The
| suffix is stripped to produce "/path/to" and that path is then
| used as the root for the library searching. Instead of searching
| literally in /lib or /usr/lib or whatnot, the "/path/to" part is
| prefixed to every search place to look in /path/to/lib and
| /path/to/usr/lib.
|
| Patching binaries is very poor; it changes their cryptographic
| hash like SHA-256. You want your distro to be installing bit-
| exact stuff from the packages, and treating it as immutable.
| eptcyka wrote:
| nixOS is built around hashing the outputs of its builds, so you
| can verify if a build produces the expected output given the
| same inputs. The reason nixOS patches binaries is so that they
| can actually find the shared objects they expect when they are
| not stored in /usr/lib. Since every build output gets its own
| unique path, this allows for a binary to link against two
| slightly different versions of the same library, which in
| practice is never an issue one needs to resolve. However, a
| more practical issue this solves is that you can have a single
| sysroot with two binaries that need two slightly different
| versions of the same library.
| matklad wrote:
| > I suspect that NixOS is playing with this in order to have a
| relocatable install
|
| Kinda the opposite: NixOS doesn't really have a sysroot.
| geraldcombs wrote:
| > A possibly nice hack would be to extend the meaning of the
| rpath variable. Give it a syntax, like say that if it starts
| with @, then the rest of it denotes a relative sysroot path
| fragment.
|
| This sounds a lot like dyld's @executable_path variable on
| macOS.
| stabbles wrote:
| Linux usually (by convention) provides 1 file and 2 symlinks per
| lib: liba.so -> liba.so.x -> liba.so.j.k.l.
|
| The first one is to make the linker (ld) happy: -la will look for
| liba.so. The linker puts the SONAME (liba.so.x) in DT_NEEDED.
|
| The second symlink's filename corresponds to the SONAME, so that
| the runtime linker (ld.so) can locate the library by SONAME in
| rpaths.
|
| The third one is the actual library, which can be updated while
| keeping the same soname & same abi.
|
| Now, it would be great if the linker had an option to not only
| copy the SONAME into DT_NEEDED, but also register the path in
| which the library was located as an rpath.
|
| Cause the situation on Linux is absurd! You pass some flags -L
| and -l to the compiler/linker, the linker links _something_ and
| nobody knows what. Then when you run your executable it has to
| locate this _something_ again, and you can only pray that your
| libc and binutils /llvm agree on search paths & order. In most
| cases this does not work, and you must manually pass
| -Wl,-rpath,/some/path to add a search path. Nobody guarantees
| that what the linker links is what the runtime linker uses.
|
| Of course there are many edge cases:
|
| - linking during make without relinking during make install will
| make your executables register rpaths to build directories
| instead of install dirs
|
| - sometimes you link to a stub lib that should not be used at
| runtime
|
| But still, some guarantee that what you build with is what you
| run with would be a major user experience improvement for linux.
| jcranmer wrote:
| > Cause the situation on Linux is absurd! You pass some flags
| -L and -l to the compiler/linker, the linker links something
| and nobody knows what. Then when you run your executable it has
| to locate this something again, and you can only pray that your
| libc and binutils/llvm agree on search paths & order. In most
| cases this does not work, and you must manually pass
| -Wl,-rpath,/some/path to add a search path. Nobody guarantees
| that what the linker links is what the runtime linker uses.
|
| The issue you're missing is that the build directory is usually
| not the location of the final binary objects. The actual
| absolute path of libfoo.so may well be in /builds/runner/foo-
| package-4df78af0/build/prefix/lib/libfoo.so, which is unlikely
| to exist on anyone other than the CI's machine (and even on the
| CI machine itself for too much longer). The actual location
| will usually be /usr/lib64/libfoo.so, but the library that is
| linked against may well not be there at the time of linking
| (particularly in the case where a package is building both a
| library and an executable that depends on said library in the
| same package).
|
| What you really want is for the _relative_ path to the library
| to stored in the executable. Unless what you want is to
| actually use the globally-installed library and not one you 're
| building at the same time. There's no single solution that fits
| every use case!
| SAI_Peregrinus wrote:
| Of course you can't guarantee that the library will get
| installed to the directory you think it'll get installed to,
| since there's no unified installer system for all Linux
| distributions. So even a relative path doesn't necessarily
| work. The best you can do is hope that the Freedesktop
| filesystem hierarchy is being followed, but that forces
| installing software for all users at once instead of per
| user, despite Linux supposedly being a multiuser OS.
| setheron wrote:
| I actually wrote somethijg recently related to RPATH in Nix you
| might find interesting
|
| https://fzakaria.com/2022/03/14/shrinkwrap-taming-dynamic-sh...
| higherhalf wrote:
| Speaking of significantly faster linkers, I personally use
| mold[1], including when writing in Rust.
|
| [1]: https://github.com/rui314/mold
___________________________________________________________________
(page generated 2022-03-15 23:01 UTC)