Learning to love Lua
--------------------

As I have explained elsewhere[1], the Zaibatsu is a small place and I
am always looking at the memory consumption of things people are
running - possibly I make a bigger deal out of this than I need
to, as we still haven't run in to any actual problems yet.  Anyway,
through this habit I realised very early on that if I wanted to write
useful software for sundogs to use there I might need to learn some
new tools.  It doesn't matter how spartan a Python program is, its
memory footprint has an unavoidable lower-bound equal to the footprint
of the Python interpreter, which is not small.  This makes, e.g. VF-1,
which I had always thought of as "lean and mean", into probably the
heaviest user program which is regularly used there (the shame!).  Of
course the same is true for Perl, PHP (which our earliest BBS client
was written in), Ruby, etc.  Truly lightweight software needs to be
either compiled or written in an interpreted language with a very
small interpreter.

I can write C, although not terribly well.  I wouldn't at all mind the
opportunity to get better through practice (and when kvothe phlogged
about "Modern C"[2] my interest was piqued), but it didn't seem wise
to use a language so unforgiving with security issues in a pubnix
environment without really knowing what I was doing.  After a bit of
searching, I ended up with Go and Lua on my radar (yes, I've heard of
Rust, no, I don't want to talk about it).  I am still interested in
learning and using both, but for whatever reason I have ended up using
Lua first.  I wrote a replacement for our PHP BBS client in it, which
is now the standard client at both the Zaibatsu and Republic, and I am
currently writing a "social unix" tool in it - which some of you are
now testing and which I will write about at some length soonish.  I am
still very much a novice and still learning, but it's starting to be
vaguely comfortably and I'm enjoying using it and would like to keep
getting better at it.

Those of you who follow me on Mastodon may remember when I first
started looking into these leaner languages that I angrily dismissed
Lua very early on as soon as I discovered that it doesn't throw an
error when you try to access a variable which has never been used
before, instead just creating that variable with a value of `nil`.
This is probably my number one most hated language mis-feature and I'm
actually not sure what got me to push past it in this case.  In fact,
on paper, I *shouldn't* like Lua at all.  It turns typos into
transient empty variables ala PHP, it indexes arrays starting at 1 ala
MATLAB and it has overly verbose "My First Language" flow control
tokens like "then", "do" and "end" instead of God-fearing alternatives
like ":", "{" or, yes, significant whitespace.  Nuts to all that.  And
yet, I *do* find myself really liking it.

(I make this joke far too often, but I usually mean it: "Significant
Whitespace" would be a great name for a band)

I think what redeems Lua for me is its multifaceted minimalism.  There
is, of course, the thing which drew me to Lua in the first place, the
micropubnix-friendly tiny interpreter.  Lua is minimal in its
*implementation*.  The interpreter is less than 500 lines of very
vanilla C.  Allegedly, it builds just fine with Turbo C 1.0 from 1990.
The resulting binary is less than 200KB in size (cf over 4MB for
Python 3).  It can be made to run just about anywhere.  But Lua is
also minimal in its *design* (which, of course, is largely what
facilitates the minimal implementation).  It provides one non-trivial
but very flexible data structure, the table, out of which you are
expected to build whatever other structures you need.  There is a
healthy dose of the philosophy of building your own domain-specific
tools to solve the problem at hand.  In this respect it reminds me a
bit of Forth, with the notable exception that it is very practical and
very easy to actually write real-world software in Lua.  I think this
is maybe the real greatness of Lua - it seems to strike an excellent
balance between the ascetic minimalism that I find so conceptually
attractive and the practical usability and readability that I need to
actually, well, use something.

Of course, basic Lua comes very much "without batteries included", in
Python parlance, and some third party libraries are necessary to get
real work done.  But I've not yet had any trouble finding a
well-estalished library to do what I've needed yet.  The luaposix and
luafilesystem libraries do, well, what you'd expect.  Penlight is a
small-ish "Swiss army knife" of useful functions, influenced by the
Python Standard Library, and it has a smaller more spartan sibling
named microlight.  With these libraries it's quite possible to write
nice command-line tools which need far less memory than their Python
equivalents would, but with far less hassle than using C would
involve.  I might even try to port VF-1 someday, although it's not a
priority right now.

Encouraged by this success, I've tried to dive deeper into the Lua
world, but this has been a very different experience to what I
expected and, if anything, has been a bit discouraging.

I went to Stack Overflow, loaded all questions tagged "Lua" and sorted
them by "most votes".  I was hoping to find interesting questions with
detailed, thoughtful replies, or long explanations of accepted best
practices for Lua coding by respected members of the community.  This
didn't exactly pan out as I expected.  It seems that the Stack
Overflow Lua community is mostly used by beginners coming to grips
with the language.  Which is fine, there has to be a place like that,
but it's not the sort of place I was trying to find.  I was a bit
taken aback to find that the most upvoted Lua question of all time is
"How can I convert a string to an integer in Lua?".  The answer is to
use the built-in `tonumber` function.  This is explained in the
second chapter of "Programming in Lua"[3], a book written by one of
the creators of the language which is available online, for free.
Specifically, it's in the section on strings, which is entitled
"Strings".  I don't want to be too snarky, but...come on.

So, my search continued, and I was unnerved to find out how much of
the Luaverse online feels abandoned.  There is a suite of tools for
using Lua as a web development language (e.g a webserver, an API
similar to Python's WSGI, an MVC framework) called Kepler.  If you
head to www.keplerproject.org it might take you a little while to
notice that the site has, in fact, been hacked and turned into a a
spam blog.  It's been much more cleverly done than usual, as all the
original content is still present, at the top, so until you scroll
past it you don't see the spam at all.  It's been this way for over a
year.  I can only assume that the domain name was snapped up by
evildoers shortly after it expired and is no longer under the control
of anybody associated with the project, otherwise surely nobody would
leave things in this state even if development had ceased (which for
most components it seems to have).

There is a library repository / package manager called Luarocks[4],
which is analogous to cpan, pip, gem, cargo, etc.  Many of the most
popular packages, including those I mentioned above, have not seen
releases in years.  In principle, I actually welcome this.  I am a
firm believer that good software can, and *should*, be "done".  If you
tackled a well-defined and suitably-sized task, and designed your tool
well, a program should eventually reach feature completeness, bugs
should get shaken out, and eventually the thing should just do what it
is supposed to do without breaking.  It may never be perfect, but the
time between subsequent releases should slowly tend toward infinity.
If it doesn't, either the program is too poorly designed to reason
about and fix bugs, or it deals with a moving target.  Which is
sometimes unavoidable, e.g. with constantly updated network protocols,
but for a library with a clearly defined and unchanging task, should
not happen.  For some reason, this is a very rare opinion and people
seem to actively steer away from anything which is not actively
developed.  Frankly, I'd be happier if as much of the software I used
as possible was *not* actively developed.  Then I wouldn't have to
waste time worrying about updates and nothing would break when
something was changed for the sake of fashion.  So, in principle, I
should be happy that all the major third party Lua libraries are years
old.  Have I not, at last, found my people?  Well, maybe, but this
attitude is rare, and actual honest-to-God "finished" programs are
very rare in practice.  I have not yet encountered a vocal subset of
the Lua community spreading the gospel of finished software.  It
doesn't seem to be an acknolwedged part of the Lua mindset.  So, in
the absence of explicit declarations to the contrary, I do worry that
a lot of these libraries are not so much "finished" and in a mature
maintenance phase, as they are simply abandoned.

I get the impression that Lua may be a victim of its reputation for
embeddability.  It is well known as being a popular language to add
scriptability to games, because of its small footprint and high speed.
Because of this, information about how to use Lua as a standalone
language for non-game applications is relatively rare.  It seems
apparent that many years ago there was an effort to get it adopted
more widely, such as the now decrepit Kepler project, or widely used
but no-longer actively developed third-party libraries, but this seems
to have ultimately failed and now it is just "the gaming language", or
perhaps "the embedded language" (sometimes people ask Lua questions
and people reply asking "what's your host language?", even if the OP
never specified there was one.  It seems often to go without saying).
This reminds me a little bit of NetBSD; it has become so strongly
associated in the public mindset with extreme portability that it
seems like it never occurs to most people that it would ever be
sensible to install it on an x86_64 machine - despite the fact that it
is a perfectly good small, clean and reliable OS which makes a really
solid server and even a usable desktop if you have modest
expectations.  It seems that being really, really good at one niche
application is actually an impediment to being widely adopted as a
general-purpose tool, no matter how good a general-purpose something
might be.

So, I plan to just plod along at my own pace, reading manuals,
tackling larger projects, and learning as I go.  Not everything needs
a vibrant online community, anyway, as the gopher and pubnix scenes
illustrate.  In the unlikely event than any master Lua hackers read
this, feel free to email me if you want to share any particularly
valuable resources or insights.

[1] gopher://zaibatsu.circumlunar.space:70/0/~solderpunk/phlog/on-tiny-servers.txt
[2] gopher://zaibatsu.circumlunar.space:70/1/~kvothe/phlog/2018-09-30-modern-c/
[3] https://www.lua.org/pil/contents.html
[4] https://luarocks.org