Inanity 2: From RAGS to Riches ------------------------------ This is the second post in my planned epic three part tale. The following is poorly written, contains loads of spelling and gramatical errors, and is likely of interest to almost nobody. Proceed at your own risk!! * * * After being asked several times by various people whether I had my own gemlog, I finally spent some time last weekend doing something about this. Of course, in the spirit of this The Lambda Lab this meant implementing a new server in Scheme. :-) The goal was to have something like Scratchy [1] (my gopher server) but for gemini. Honestly I wasn't expecting this to be very hard at all. The only thing I was unsure of was the TLS aspect. I've never had anything to do with TLS, besides the experience getting elpher to initiate TLS connections to supporting gopher servers and then gemini servers. Some searching revealed that there already was an egg (that's what Chicken calls its packages :-) ) providing openssl bindings. Sadly, although it is nicely documented, the documentation lacks any examples. While this isn't usually a problem with scheme libraries, it was a show-stopper for me as - honestly - I didn't know what half of the options to the procedures used to listen for incomming connections actually meant. Eventually I went hunting through the source code of the one package that seemed to use this library: the HTTP library, Spiffy. Therein I was able to find the necessary examples, which honestly seemed too simple - most of the options to the listener-creation procedures being apparently unnecessary. I was able to use this to cobble together a simple test that would accept TLS connections on a port and send a "Hello, World!\n" when a client connected. After managing to get this working, replaced the hello world with a gemini header string and successfully managed to get av98 to connect to the port and process the result. At this point I figured I was home free. I spent an enjoyable hour or two writing the document delivery code. This was a wonderful experience which again drove home how much nicer it is to deal with gemini, both on the server and the client side, than gopher. The resulting server is at least as capable as Scratchy, but both shorter and less complex. (The ability of servers to indicate the mime-type of the content they produce makes so many things easier.) Now I must digress for a moment to gush about how lisps make the creation of extensible programs just incredibly trivial. Here's the definition of the procedure which reads a a scheme function from a file, then applies that function to the URI requested by the client: (define (serve-script config uri) (let* ((path (document-path config uri)) (proc (eval (with-input-from-file path read)))) (with-current-working-directory (pathname-directory (document-path config uri)) (lambda () (proc uri))))) If you're not used to lispy languages, this might just look like a mess of parentheses, but just look at the let binding up the top. The expression (eval (with-input-from-file path read)) is responsible for opening the file and parsing the contents as lisp expression, then evaluating the result. That file can have comments, the lisp expression can be arbitrarily long: it can have it's own imports, procedure definitions, etc etc. Talk about concise! All due to the power of `read'. There's a document [2] on Richard Stallman's web page where he discusses his various opinions on programming languages, among other things, and where he states that "Most other [non-lisp] languages have nothing comparable to `read', nothing comparable to `eval', and nothing comparable to `print'. What gaping deficiencies!" Even after a few short years hacking on elisp and scheme, I tend to agree. (Of course, the above procedure comes with a _bunch_ of security concerns, but (a) it's just a hobby server, and (b) as long as I carefully vet the extension scripts I run and take steps to ensure nothing I haven't vetted is ever run, I think I'm pretty safe.) Okay, so once the document serving code was complete, I quickly hooked it up to the TLS front-end, fired up a local instance and verified that av98 could connect and retrieve documents. Too easy! I got excited and, in a bout of prescient madness, gave the server its name: RAGS, the Right-Awful Gemini Server. All that remained was to cross-compile a static binary for deployment on my VPS... $ csc_linux -static rags.scm /usr/bin/ld: /usr/local/lib/chicken/11/openssl.o: in function `stub1057': openssl.static.c:(.text+0xaf3b): undefined reference to `SSL_get_peer_certificate' /usr/bin/ld: openssl.static.c:(.text+0xaf52): undefined reference to `X509_get_issuer_name' /usr/bin/ld: openssl.static.c:(.text+0xaf5f): undefined reference to `X509_NAME_oneline' /usr/bin/ld: openssl.static.c:(.text+0xaf6a): undefined reference to `X509_free' ... Okay, you get the idea. Now, I've had trouble statically compiling my chicken projects before. A couple of times I've even had to ask for help on the #chicken freenode channel. There's always been a solution, and the Chicken community is one of the most notoriously helpful and friendly around, so it's always been a pleasant experience. However in this instance I suspected that the problem was more severe than normal due to the involvement of openssl, and the fact that my server code relied on an egg which provided bindings to this library but didn't include the library itself. I thus shelved the whole idea of producing a static binary. You might be asking - why am I bothering with static binaries? Well, running a Chicken binary with shared library dependencies requires, of course, the Chicken libraries to be installed. This on its own isn't a problem: my VPS runs Debian and there's, of course, a chicken-bin package containing the required files. However my VPS is (was) running Debian 8, for which the most recent Chicken version available was 4.something. There's are some pretty big difference between Chicken 4 and 5, and all of my code is written assuming Chicken 5. Then why not just compile Chicken 5 on the VPS? Well, I tried that when I was just getting set up a little over a year ago, but here's the thing - my VPS only has 128MB of RAM. Which is enough for an amazing number of things but, apparently, not enough to compile Chicken 5 with gcc. Thus, for the whole of the last year, I've been hacking together gopher, mail, corewar servers, I've been using Chicken's -static option to produce hulking 5MB ELFs that run wherever. (In combination with a Docker container allowing me to build such executables on the distinctly non-GNU/Linux machine I've been suffering with, for my sins.) So... what now? Just for kicks, I decided to do what I should have tried ages ago - I tried to see how far I could get by compiling a dynamic executable and then simply copying .so files over to the VPS. It turns out that there really weren't many that needed to be copied: just libchicken.so and those corresponding to the individual eggs. And it worked! I shouldn't have been surprised, but honestly you doubt everything when you get to my age. I quickly replaced all of my old static executables for dynamic versions and saved a bunch of memory across the board. And, once I installed the required openssl libs on the server, I was able to get RAGS running too. Fantastic! Now I just needed to check that av98 could connect to thelambdalab.xyz... .. but no. No it doesn't. It refuses to connect. The server reports that it and av98 failed to agree on a cypher to use for the connection. AAArrrrgh!!! Okay, deep breaths. The first thing I tried was commenting out the line of the av98 python code that restricts the choice of cyphers. It was then able to connect without trouble, meaning that it really was an issue with the required cyphers not being made available by my server. But why was this only a problem on my VPS? After all, I was using the same openssl libraries there as in my local environment, wasn't I? To try to understand this, I used a script to spam the server with TLS connections restricted to particular cyphers so that I could build up a list of which cyphers were being made available. Repeating this on the server when running on my localhost confirmed that the cyphers supported by the server running on the VPS formed a much smaller subset of the ones made available by the server running on my home machine. At this point I began to suspect the problem. Debian 8 ships with libssl 1.0.1, while on my local machine I'm using libssl 1.1.0. Happily, my VPS provider sent me an email sometime over the last few months indicating that I could now upgrade to Debian 10 with a click of a button. After confirming that Debian 10 would indeed allow me to easily install later versions of this library (but sadly not Chicken..) I figured that today was the day to press this button. After an amazingly short time (seriously, maybe 30 seconds? a minute?) I was able to ssh into the freshly re-imaged server and begin the painful process of putting pieces of my Rube Goldberg configuration back together. It took nearly half a day, in the end, and this is despite having anticipated this nightmare from the beginning and fastidiously backing up every one of the small number of configuration files that I've modified over the last year. Oh how I wish I were running Guix or Nix. Anyway, _finally_ I was able to test RAGS on the new configuration and confirm that it was able to meet the cypher requirements of av98. I quickly put together a Gemini landing page, together with a dynamically generated (using those wonderful scheme extension scripts!) list of my recent phlog posts and an atom feed generator. After checking that everything ran smoothly and rendered nicely in elpher, bombadillo and av98, I went to bed. Which, if you made it through all of that sorry tale, I'm sure you'll want to do too! 0.o If you're keen to see what the state of things is now, head over to gemini://thelambdalab.xyz! And if you want to glance over the RAGS source, comment on it or submit patches, visit the project's gopher page at gopher://thelambdalab.xyz/1/projects/rags. Until next time... --- [1]: gopher://thelambdalab.xyz/1/projects/scratchy/ [2]: https://stallman.org/stallman-computing.html .