---------------------------------------- PicoGopher Part 2: sudo make a server October 29, 2022 ---------------------------------------- Written on my laptop ---------------------------------------- Welcome to Part 2 of my PicoGopher journey! If you want to see how everything started, you can find the previous parts in my phlog at gopher://gopher.club/1/users/mala/ (also mirrored at gopher://gopher.3564020356.org). You can find the project's code at https://github.com/aittalam/PicoGopher. In Part 1 we talked about the why of PicoGopher (TL/DR: it is a supercool project and you definitely want to walk around your city while serving your phlog to strangers ;-P), and we got inspiration from (ie. blatantly copied) the official documentation of Raspberry Pi Pico W to build a simple mockup Gopher server. That was just a proof-of-concept, serving a text file which already contained what a server is expected to send to a Gopher client. The purpose of this post is to document the following step, i.e. how we can transform our mockup into an actual server, which gets a request for a given path and returns the corresponding file, gophermap, or error message. ========= Project status ========= Below you can see the current status of the project. The steps marked as "x" have been completed, while those marked as "+" are described in this post. Namely we will focus on (1) translating gophermaps into the format that Gopher clients expect to receive and (2) parsing client requests and serving corresponding files. - [x] connect to the WiFi - [x] run a simple HTTP server - [x] run a mockup Gopher server - [x] load/save files - [+] make the Gopher server not a mockup anymore: - [+] translate gophermaps following gopher protocol - [+] load any file from disk -> at this point, you have a running Gopher server! - [ ] make the server a bit more resilient - [ ] enable async - [ ] better understand power saving - [ ] set up the pico as an access point for geolocalised access - [ ] powering PicoGopher ==== Interpreting gophermaps ==== I think this is the point where a serious Gopher developer would provide a link to the RFC [1], describing the protocol in detail. But, honestly, there's not a lot to interpret out of gophermaps and to have something up and running quickly I preferred to look at the simplest server implementation I could find, which is exactly what I am going to describe here. Searching for "tiny gopher server python" I stumbled upon gofor [2], an unsurprisingly tiny gopher server written in python. Be warned, this might not be the best server you can find around: all the three commits to the repo occurred on a single day three years ago, and the author describes it as a "stupidly tiny, nonconformant (but functional) Gopher server". But, hey, I liked the honesty and, above all, the 150 lines of super simple python code. Most of the code we need resides in the "data_received" function: first of all, it checks the request, making sure it matches a plausible Gopher selector (i.e. does not contain tabs which are specific to Gopher+ [3] and points to a valid path). Then, there are two possibilities: either a file was requested (so we need to directly serve it) or a directory, in which case we look for a gophermap and interpret it. The transfer of generic files is quite straightforward: data is read from the file in blocks of size "block_size" and then sent through the socket. There are a couple of caveats for PicoGopher though which is worth knowing: - the device does not have much memory, so choosing a rather small value for block_size is preferable (I got some OOM errors with 64KB so I reduced it to 4, feel free to play with larger values if you want to find the sweet spot) - when I first reimplemented this code in micropython I naively used cl.send() to send data to the socket, as in the HTTP example shown in the Pico W tutorial. Only after testing it I realised that cl.send() might perform "short writes", ie. send through the socket fewer bytes than the actual length of the data. The solution is to use another function such as write(), which tries to send all data to the socket. OF COURSE gofor's code was already correct, so I should have just blatantly copied it :facepalm: :-) The following step, i.e. gophermap interpretation, is quite simple as it does not really check for correctness of the input file. It just assumes every row returned to the client has to contain four columns as defined in the RFC: itemtype concatenated with some content, path, domain and port, all separated by tabs. So whenever a line is read from a gopherfile, it is split using the tab character as separator, and depending on how many columns we end up with one of five things occurs: - one column: it has to be text, so it is converted to "i" (inline text type) + the content itself, then empty path, domain, and port - two columns: they must be valid itemtype+content and path, so we just add domain and port - three columns: we must have just missed the port so we just add it - four columns: just keep the row as it is - more than four columns: we only keep the first four After this, gofor code takes care of fixing relative paths. For some reason there was a screwup with HTTP links: a "/" was prepended and Lagrange did not open them, so I decided to apply a small fix in my version of the code. ==== Accessing files on disk ==== Gofor's code strongly relies on pathlib to evaluate whether the received path matches an existing and valid directory or file. The research I did might have been quite superficial but I haven't found something equivalent in micropython, so I used the outputs of os.stat to implement my own tiny version which I called PicoPath. To do this I relied on the implementation of stat you can find here [4]. It is far from being complete, but it allows me to have code which is more readable and more similar to gofor's implementation. One more note regarding file access: I customised gofor's code so all requests are chrooted to a "gopher" directory, and kept all the extra checks as they were originally implemented. I am quite sure the system is not really secure so be careful with what you put on this thing! For now, I am more interested in making this work... ==== Running your server ==== ... Aaand it works! :-) If you want to try serving your own gopherhole from a Pico W: - clone the PicoGopher GitHub repo [5] - copy your whole gopherhole in the "gopher" dir - open the 01_gopher.py file in Thonny and customize it with your WiFi ssid and password - run the script The Pico will first try to connect to the WiFi, then you will see in Thonny's Shell pane which IP address it has been assigned. Open your gopher browser and connect to that IP! My post at https://fosstodon.org/@mala/109231106321211643 shows this very gopherhole being served by a Pico W. While I am quite excited about this, I feel like we are not "there" yet. We do have a nice proof of concept, but before this has any chance to become something useful we should at least make it: - self-contained: it has to run without being connected to a computer - portable: one should be able to put it into a backpack and bring it around, or leave it somewhere forever, without worrying whether it has enough power - ubiquitous: right now it relies on an existing wifi, but we should make it work everywhere by making it *provide* a wifi to other devices - available: we should be able to choose whether we want to make it available only to people who have a gopher client installed or to anyone who has a browser There are already a few more things I would like to implement but I will stop here :-) Go try this version of PicoGopher and let me know what you think about it! Next time we will tackle some more of the missing steps and decide how to continue from there. If you have questions or ideas before that, I am very happy to hear them. [1] https://datatracker.ietf.org/doc/html/rfc1436 [2] https://github.com/yam655/gofor [3] https://github.com/gopher-protocol/gopher-plus/blob/main/gopherplus.md [4] https://github.com/python/cpython/blob/main/Lib/stat.py [5] https://github.com/aittalam/PicoGopher