BACKWARDS BUILDS A nice warm day today [yesterday, by the time I finished writing this], at least compared against the endless cold dreary winter's days that have set the mood for months. Great for working on one of my outdoor projects, but I was rather unfocused and distractable. Ended up on my chair in the nice warm sun room, took off all my clothes, and grabbed my laptop for some hardcore... Linux! Ages ago I talked about battling with getting OpenSSH to build for the old version of Linux that I run on the old PCs like the one I'm posting from now. The problem was actually with the configure script because I wanted it to link against OpenSSL libraries in a custom directory to avoid messing with the old OpenSSL lib used by other programs, and the script just didn't want to play ball. I couldn't even manage to hack the configure script into submission, so I eventually gave up. Then I found PuTTY, which turned out to support Linux as well as Windows, didn't require any extra libraries, and had all the extras that come with openSSH, particularly an SFTP client. But that ran afoul of Tilde.Club when I signed up earlier this year, because it didn't support the latest RSA key magic using SHA-2 instead of SHA-1. Yes, it seems SSH software might only last a couple of years these days before becoming obsolete. Bloody internet. No problem though, PuTTY added support for that later, so I can just compile the latest version. But when I tried that about a week ago things took a turn full-circle. PuTTY has switched to using CMake as a build system. Whereas configure scripts and Makefiles at least try to run everywhere, CMake builds require a cmake program that's only a few years old at the latest. Building the latest CMake requires a C++11 compiler, but I found an earlier version that could still compile with old GCC, but the build failed early looking for a system header file that didn't exist. It didn't look hopeful. Like OpenSSH before, I've been roadblocked by the build system. So I gave up and built the last version of PuTTY before it switched to CMake, PuTTY 0.76, which Tilde.Club does like, for now. But clearly that solution is only going to last so long as well. What I need is to build from a modern system, with a modern CMake. But Linux, frustratingly, isn't kind to those attempting binary compatibility between systems of different ages. Actually that blame is largely misplaced. It's not actually Linux that's the problem, it's Glibc. Glibc, for the uninitiated, is the library that implements the standard C function routines (open, printf, etc.) of the C programming language. It talks between programs and the operating system kernel, and as a GNU project it's probably the overall most common GNU part of "GNU/Linux". Unfortunately programs compiled for new versions of Glibc generally won't run with older Glibc versions (although the Glibc developers do a great job at keeping old binaries working with newser Glibc versions). In theory static linking (GCC's "-static" option) solves this problem by bundling Glibc in with the program's binary, but in practice it doesn't because Glibc has some features which cause it to load extra files from the system. Glibc's not compatible with old versions of some of these files, particulary some which are actually object files (like mini libraries themselves) and don't get included in the static binary. Glibc's developers don't really like static linking, and they've basically willfully broken it, especially for building networking-related software like PuTTY. AppImages give the impression of having solved this, but they haven't. I built an AppImage for a complicated program a while ago. While in the (very slow) process of creating a monsterously long command line of manually-added rules to make the AppImage work successfully, I discovered that the standard solution to the Glibc compatibility problem is that developers simply pick the oldest Linux distro that they expect anyone to use, and run that to build the AppImage. As such, most users of the AppImage run later Glibc versions, which are designed to be compatible with such 'older' binaries, but it doesn't solve anything for going the other way and running 'new' binaries on old Glibc. It also means developers are tempted to bundle old support libraries, from the old Linux distros that they use, into the AppImages, which isn't good (there are a few solutions for building compatible AppImages on newer systems, but they're quite complicated and still require picking a relatively-recent minimum Glibc version). On top of that, Glibc since version 2.26 simply won't run on Linux kernel versions older than 3.2, which also rules it out for building a static binary to work for me on older Linux. So today's "hardcore Linux" task was to try an alternative to Glibc: Musl. This is designed as a more lightweight C library than Glibc, and has been adoped by some Linux distros instead of Glibc. Alpine Linux is particularly popular amongst these, and has amassed a surprisingly large number of programs in their package repo apparantly built against Musl. Even PuTTY! The Musl docs also don't mention a Linux kernel requirement beyond version 2.4 for "simple single-threaded applications" which _might_ include PuTTY? Plus it doesn't do any 'tricks' with object files like Glibc, so static builds work like they should. Musl _can_ be installed on Debian 10 Buster (or in my case the matching version of Devuan) too. Packages will still use Glibc because that's what they were compiled for, but you can compile your own software against Musl by setting the compiler to "musl-gcc" after installing "musl-tools". My Devuan system is actually x86_64 though, and I want to build for x86. I _could_ set up a separate x86 build system, but in theory GCC's x86_64 build target also supports building x86 binaries, even when it's not running on x86. Debian also has "multiarch" for installing 32bit libraries on 64-bit installations, which I already set up for running Wine. In theory this all goes together to mean that I can compile static, 32bit, Musl libC, builds of software on x86_64 Devuan! In practice everything fought against me on this, but after a couple of hours I'd worked out these steps to success (for C programs at least): Install these packages with their dependencies on multiarch-enabled Debian/Devuan: gcc-multiarch musl-tools musl-dev:i386 You need a GCC "specs" file for building against the 32-bit Musl library, but the Debian package system doesn't want to install it from musl-tools:i386 without replacing GCC with its 32-bit equivalent (damn obstinate package systems!), so make it from the 64-bit version (run as root): sed 's/x86_64/i386/g' /usr/lib/x86_64-linux-musl/musl-gcc.specs > \ /usr/lib/i386-linux-musl/musl-gcc.specs Set the CFLAGS environment variable with all the right magic words for CMake (or a 'configure' script) to whisper to GCC (Debian packages, presumably including Musl, are currently compiled for i686 arch.): export CFLAGS='-m32 -march=i686 -specs /usr/lib/i386-linux-musl/musl-gcc.specs -static -Xlinker -melf_i386' Run CMake (or ./configure): cmake -DCMAKE_BUILD_TYPE=Release . Edit the newly created CMakeCache.txt file as required (or run one of the GUI CMake interfaces).* I built just the command-line programs, not "putty" itself which uses GTK. Build with CMake (or "make"): cmake --build . Now you should have 32-bit i686 static binaries which will run on the x86_64 build system, and on older Linux distros, either x86_64 or x86. To be sure that it's really working, you can try building a test program described on this webpage (which otherwise stops a bit short of describing a complete solution for the task): https://www.geeksforgeeks.org/compile-32-bit-program-64-bit-gcc-c-c/ ------------------------------------------------------------------ // C program to demonstrate difference // in output in 32-bit and 64-bit gcc // File name: geek.c   #include int main() { printf("Size = %lu\n", sizeof(size_t)); } ------------------------------------------------------------------ When built as a 64-bit binary it prints "Size = 8" to the terminal when executed, and as a 32-bit binary it prints "Size = 4". Build it with dynamic linking first to confirm that the GCC command arguments are really causing it to link to Musl instead of Glibc: gcc -m32 -march=i686 \ -specs /usr/lib/i386-linux-musl/musl-gcc.specs \ -Xlinker -melf_i386 -o geek geek.c Now check with "file" to see the arch and "interpreter" used by the created binary: file geek geek: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-i386.so.1, with debug_info, not stripped So it's an x86 binary, and running it will therefore print "Speed = 4", plus "interpreter /lib/ld-musl-i386.so.1" means it's linked to Musl instead of Glibc! Now for the static binary: gcc -m32 -march=i686 \ -specs /usr/lib/i386-linux-musl/musl-gcc.specs -static \ -Xlinker -melf_i386 -o geek geek.c file geek geek: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, with debug_info, not stripped So after all that, did it make a PuTTY binary that worked on my old Linux distros on my old PCs? No. They just give me "segmentation fault". It does work on older 32-bit Linux distros. It worked on Debian 7 "Wheezy". But it doesn't go as far back as I needed it to. Musl isn't as compatible with old Linux kernels as I'd hoped, or maybe the problem is something else. Anyway it turned out to be pretty much a waste of time for me, but the technique might come in handy later on. I've also uploaded my i686 Musl static binaries of the PuTTY command-line tools here, for anyone who might want to give them a try. You're welcome to report back the earliest Linux that you can run them on. I'm too lazy to test with lots of old Linux kernels to find out where the cut-off point actually is: gopher://aussies.space/5/~freet/cupboard/putty_0.78_musl_static.tar.bz2 The next step for me might be to try and chroot into a copy of the root filesystem from my target Linux distro on a modern distro with modern CMake (accessed via a bind mount to the 'real' root FS and run with LD_LIBRARY_PATH). Then in theory I could run the CMake command where it thinks it's running in the old distro and generates corresponding makefiles. After that I could copy that PuTTY build directory to the real copy of the old distro and run "make" in there. Or maybe it's time to go back and have another go at OpenSSH? That's really getting desperate after the agony of last time though. - The Free Thinker * I really can't believe that the CMake developers haven't implemented an equivalent to "./configure --help", which has always been my starting point for building anything with a configure script. CMakeCache.txt is full of jumbled up options which makes it very hard to read (bare hand-edited makefiles are more readable!). Then, if you're trying to create a repeatable command to build later versions, you've got to then delete the CMakeCache.txt file and run the "cmake" command again with the changed options set as "-D[option]=[value]" arguments. What an awkward design! Neither PuTTY, nor the one other program I've compiled using CMake, bothered to document custom build options in their documentation either.