* * * * * Why one might want to load dynamic objects from memory instead of from disk Years ago I started on a little project entitled the LEM Project [1]. It was a project to make distribution of applications written in Lua [2] easier (It's named after the Apollo Lunar Module [3]—aka (also known as) the Lunar Excursion Module; Lua is “moon” in Portuguese). I had a few goals: 1. allow distribution of multiple versions of a Lua module; 2. allow for different architectures for the same module written in C; 3. run the entire application from the distribution file (nothing to install). And I almost hit all the goals. The first two were easy. I read over the ZIP file specification [4] and saw that 1. multiple files of the same name were entirely possible—nothing in the specification disallows it; 2. additional metadata can be added to each file beyond what is defined in the specification. So armed, I created zip file with some addtional metadata and … -----[ shell ]----- [spc]lucy:~/projects/LEM>./zipf.lua sample.lem The Eagle Has Landed SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.base64 SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.crc SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.env SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.errno SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.hash SunOS sparcv9 LGPL3+ Lua 5.1 5.1 1.1 MODULES/org.conman.iconv SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.math SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.pollset SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.process SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.strcore SunOS sparcv9 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.sys SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.syslog Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.base64 Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.crc Linux x86 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.env Linux x86 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.errno Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys.magic Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.hash Linux x86 LGPL3+ Lua 5.1 5.1 1.1.1 MODULES/org.conman.iconv Linux x86 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.math Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net.ipacl Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.pollset Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.process Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.strcore Linux x86 LGPL3+ Lua 5.1 5.1 1.2.0 MODULES/org.conman.sys Linux x86 LGPL3+ Lua 5.1 5.1 1.0.2 MODULES/org.conman.syslog Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.tcc LGPL3+ Lua 5.1 5.1 MODULES/org.conman.cc LGPL3+ Lua 5.1 5.1 MODULES/org.conman.date LGPL3+ Lua 5.1 5.1 MODULES/org.conman.debug LGPL3+ Lua 5.1 5.1 MODULES/org.conman.dns.resolv LGPL3+ Lua 5.1 5.1 MODULES/org.conman.getopt LGPL3+ Lua 5.1 5.1 MODULES/org.conman.string LGPL3+ Lua 5.1 5.1 MODULES/org.conman.table LGPL3+ Lua 5.1 5.1 MODULES/org.conman.unix MIT Lua 5.1 5.1 0.10 MODULES/lpeg MIT Lua 5.1 5.1 0.10 MODULES/re Linux x86 MIT Lua 5.1 5.1 0.12 MODULES/lpeg Linux x86 MIT Lua 5.2 5.2 0.12 MODULES/lpeg MIT Lua 5.1 5.2 0.12 MODULES/re Linux x86 MIT/X11 Lua 5.1 5.1 0.4.work3 MODULES/zlib MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket Linux x86 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.core MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.ftp MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.http MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.smtp MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.tp MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.url MIT/X11 Lua 5.1 5.1 1.0.1 MODULES/ltn12 FILES/APPNOTE.TXT FILES/COPYING FILES/README FILES/Miscellaneous Things About Nothing [spc]lucy:~/projects/LEM> -----[ END OF LINE ]----- You can see there are several copies of each module. The modules listed without a system (like Linux or SunOS) are written in Lua; the other ones are in C and both the system and architecture are listed. The license is included, along with the Lua verions the module will work for (it's “minimum version” and “maximum version”) as well as the version of the module (if known). It's even a valid zip file: -----[ shell ]----- [spc]lucy:~/projects/LEM>unzip -l sample.lem Archive: sample.lem The Eagle Has Landed Length Date Time Name -------- ---- ---- ---- 25472 05-24-14 17:10 MODULES/org.conman.base64 13448 05-24-14 17:10 MODULES/org.conman.crc 12200 05-24-14 17:10 MODULES/org.conman.env 18688 05-24-14 17:10 MODULES/org.conman.errno 57032 05-24-14 17:10 MODULES/org.conman.fsys 24952 05-24-14 17:10 MODULES/org.conman.hash 17664 05-24-14 17:10 MODULES/org.conman.iconv 17648 05-24-14 17:10 MODULES/org.conman.math 77944 05-24-14 17:10 MODULES/org.conman.net 26296 05-24-14 17:10 MODULES/org.conman.pollset 88256 05-24-14 17:10 MODULES/org.conman.process 37848 05-24-14 17:10 MODULES/org.conman.strcore 15392 05-24-14 17:10 MODULES/org.conman.sys 24312 05-24-14 17:10 MODULES/org.conman.syslog 14175 06-12-14 22:04 MODULES/org.conman.base64 8214 06-12-14 22:04 MODULES/org.conman.crc 7193 06-12-14 22:04 MODULES/org.conman.env 10690 06-12-14 22:04 MODULES/org.conman.errno 31885 06-12-14 22:04 MODULES/org.conman.fsys 15567 06-12-14 22:04 MODULES/org.conman.fsys.magic 14067 06-12-14 22:04 MODULES/org.conman.hash 10197 06-12-14 22:04 MODULES/org.conman.iconv 10816 06-12-14 22:04 MODULES/org.conman.math 43651 06-12-14 22:04 MODULES/org.conman.net 25248 04-18-14 20:06 MODULES/org.conman.net.ipacl 15914 06-12-14 22:04 MODULES/org.conman.pollset 49607 06-12-14 22:04 MODULES/org.conman.process 15666 06-12-14 22:04 MODULES/org.conman.strcore 9763 06-12-14 22:04 MODULES/org.conman.sys 13303 06-12-14 22:04 MODULES/org.conman.syslog 21218 06-12-14 22:04 MODULES/org.conman.tcc 5617 06-12-14 22:04 MODULES/org.conman.cc 2500 06-12-14 22:04 MODULES/org.conman.date 3829 06-12-14 22:04 MODULES/org.conman.debug 2966 06-12-14 22:04 MODULES/org.conman.dns.resolv 3260 06-12-14 22:04 MODULES/org.conman.getopt 2464 06-12-14 22:04 MODULES/org.conman.string 5243 06-12-14 22:04 MODULES/org.conman.table 2732 06-12-14 22:04 MODULES/org.conman.unix 40081 05-25-14 15:17 MODULES/lpeg 6029 05-24-14 00:36 MODULES/re 40045 05-28-14 15:24 MODULES/lpeg 40045 05-28-14 15:24 MODULES/lpeg 6286 05-28-14 15:24 MODULES/re 19794 05-30-14 21:29 MODULES/zlib 4451 05-28-14 14:52 MODULES/socket 55449 05-28-14 14:52 MODULES/socket.core 9243 05-28-14 14:52 MODULES/socket.ftp 12330 05-28-14 14:52 MODULES/socket.http 8074 05-28-14 14:52 MODULES/socket.smtp 3612 05-28-14 14:52 MODULES/socket.tp 11036 05-28-14 14:52 MODULES/socket.url 8331 05-28-14 14:52 MODULES/ltn12 161412 05-09-14 01:24 FILES/APPNOTE.TXT The ZIP file format 7651 05-25-14 18:53 FILES/COPYING 9789 06-07-14 22:13 FILES/README Much Ado About Nothing 3946 05-10-99 23:00 FILES/Miscellaneous Things About Nothing If you this this has significance, think otherwise -------- ------- 1250541 57 files [spc]lucy:~/projects/LEM> -----[ END OF LINE ]----- (Although if you try to unzip this file, the unzip program will ask you what you want to do with the duplicate files.) The intent was that only those modules that match the Lua version, system (if a C-based Lua module) and architecture (again, C-base Lua module) match, load the module; otherwise, it would be ignored. And it worked. For the most part. I could load Lua modules written in Lua directly from the zip file with no problem. I even got LuaRocks [5] to run completely from the zip file (with some hacks so it could load the configuration file). But I could not do that for the C-based Lua modules. For those, you had to write them out to disk to load them up. And because of that, I lost interest in continuing the project. Until now. I just came across MojoELF [6] (via Project: 2ine [7] which was linked via Hacker News [8])—a project to load ELF (Executable and Linkable Format) binaries directly from memory (and those C-based Lua modules I have are ELF binaries), which is exactly what I wanted all those years ago. [1] https://github.com/spc476/LEM [2] https://www.lua.org/ [3] https://en.wikipedia.org/wiki/Apollo_Lunar_Module [4] https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT [5] https://luarocks.org/ [6] https://hg.icculus.org/icculus/mojoelf/file/tip/README.txt [7] https://www.patreon.com/posts/project-2ine-16513790 [8] https://news.ycombinator.com/item?id=16409584 Email author at sean@conman.org .