VF-1 updates and tips
---------------------

I'm very happy to have had both the time and motivation to get quite a
bit of good progress made on my gopher client VF-1 recently.  This
post will mostly be an update on some of the new functionality.  But
first, some usage tips!  I noticed recently that tfurrows has been
keeping a list of tips[1] on his circumlunar gopherhole (where he
explores the bookmarking functionality more deeply than I ever have!),
and thought I might contribute a few of my own.

First of all, an embarrassing revelation - when I recently wrote my
allegedly definitive guide[2] to viewing "long stuff", I forgot about
one option for handling long menus!  There are so many options even I
forget them.  That said, this one is not one of my favourites, it was
an early hack to help people who were struggling in earlier days
before "less" worked on menus.  When you use "ls" to list the current
menu selectors, you can give it the "-r" option (i.e. "ls -r") to view
the listing in the reverse of the usual order.  Thus, items at the top
which would normally go flying off the top of your screen appear at
the bottom where you can always see them.  The obvious downside, of
course, is that stuff is backward.  I don't really recommend this
approach, but thought I'd mention it for completeness.

Onto something a little more useful!  You are probably already aware
that VF-1 lets you set any external command you like as a handler for
different kinds of content.  By default, the handler for "text/plain"
is just good old "cat", which does nothing other than spit the text
onto your screen.  If it overflows, you can run the "less" command to
look at it in your favourite pager.  An alternative to this is to use
less as your text/plain handler, but feed it a few more options.  For
the past few days I have been using "less -FXR %s" as my default plain
text handler.  The -F option tells less to immediately quit if the
file is short enough that it fits entirely on one screen, and -X
option tells it not to clear the screen after exiting (as is the
default behaviour of more).  What this does is basically turn less
into an automatic "cat if short, less if long" viewer.  The -R is
just there so that ANSI colour codes don't get mangled (more on that
later).  This means stuff never flies off the top of your screen, and
you never have to manually run less to read the top of something.
This results in a pretty seamless experience and I think I'll stick
with it.

Okay, time for new features.

Starting with something very minor, the "text/plain" handler is now
used for both item types 0 and 1, whereas previously it only worked
for type 0.  This change was inspired by Tomasino who, when learning
about handlers, immediately set his to lolcat - something I'd never
heard of.  I encourage you to check it out, even if only briefly for,
well, the lols.  Basically you can pipe text through it and it uses
ANSI colour codes to render that text into a GLORIOUS RAINBOW.  We're
talking hundreds of colours, each character slightly different from
it's neighbours.  Tomasino was disappointed that this worked on
content but not on menus (which in his case means his entire phlog),
the handler is now applied to menus too so you can enjoy ubiquitous
rainbows in gopherspace.  Tomasino was *also* disappointed that the
colours disappeared when he used the "less" command, because until
now that command ignored the text/plain handler and just fed the
content straight to less.  Now, the "less" command runs your
text/plain handler and pipes the output of that to less (or rather,
less -R, to preserve colours), so you can get colours even when you
are lessing!

To more fundamental changes, Tomasino has once again spurred me to
make some improvements, in his recent championing of better support
for the "+" item type which is used to specify redundant severs, i.e.
gopher servers which host a mirror of the content at the current
server.  The RFC is pretty vague about exactly how this is supposed
to work.  Most modern clients take a very minimal approach to
supporting this, and just list the mirror items like they would any
other link but do something minor to indicated "hey, this is a
mirror".  I think the intent was probably for clients to do a bit
more with this.  The RFC has various comments in it which makes it
pretty clear (to me at least) that the target environment for gopher
was under-resourced university departments setting up servers on
whatever old and under-powered hardware they had lying around, and
spreading information over as many servers as possible to reduce
load.  Early gopher servers were probably expected to fail regularly.
So VF-1 tries to handle + items in such a way as to reduce the pain
of servers.  After seeing that content at server A is mirrored at
server B, if an attempt to fetch something from server A later
during the same VF-1 session results in any kind of network error,
VF-1 will automatically try to fetch the content from server B
instead.  The usefulness of this in 2019 is arguably limited - for
one thing, modern gopher servers are probably extremely powerful and
extremely under-loaded compared to early servers, and for another
there is no caching of redundant servers, so if the "main" server
you attempt to visit is down, you have no way to learn what the
backups are.  It's not perfect, but it's better than nothing, and
I'm proud that VF-1 actually makes an effort to *do* something with
this information.

Speaking of being proud, the other significant changes are related
to text decoding, and I suspect VF-1 might now be the best gopher
client in town for people who regularly visit content encoded in a
variety of non-UTF-8 forms.  Tomasino had nothing to do with this
change, which was instead prompted by the latest user at
circumlunar.space, tengu[3], who had some initial problems serving
Russian text from his gopherhole there, whether using UTF-8 or
older Cyrillic encodings like KOI8-R or CP1251.  With some digging,
it turned out that this was mostly the fault of Gophernicus, but
VF-1 could stand some improvement too.

In the earliest versions of VF-1, I assumed that all text coming
over the wire would be either ASCII or UTF-8 (which decode
identically) and left it at that.  This worked fine for about a
week until someone on BBOARD reported that VF-1 died when trying
to read some news article over at floodgap's feeds.  It turned out
that the article contained a name with an accented character in it,
which was encoded in ISO-8859-1.  So, I did a bit of research,
learned that the 3 most commonly used encodings on the web are, in
order, UTF-8, ISO-8559-1 and CP1251.  So, I updated VF-1 to try
these, in order, moving down the list each time one failed.

If you know anything about text encoding you'll recognise how
naive this was.  Any text which is valid CP1251 is also valid
ISO-8559-1, so an attempt to decode as ISO-8559-1 will never fail.
It may result in gibberish, but it won't throw an exception, and
so CP1251 text will never be decoded properly.  So, now VF-1
attempts to decode everything as UTF-8 first and, if that fails,
tries a single fallback encoding.  That fallback defaults to
ISO-8559-1, but it is under direct and easy user control using the
"set" command, so you can do "set encoding cp1251" to change the
fallback.  If you regularly deal with just one non-UTF-8 encoding,
you can of course stick this in your ~/.vf1rc file to make it
permanently.

But wait, there's more.  There is a very nice Python library
called chardet which attempts to automatically detect text
encodings making use of language statistics.  You can decode
CP1251 as if it were ISO-8559-1 no problem, but you'll end up with
gibberish text whose n-gram distribution won't match any natural
language.  Chardet uses this fact to guess encodings and with a
little practice it seems to work quite well.  Now, I am very proud
of the fact that VF-1 has no dependencies outside the Python
standard library and that all of the code is in one single file.
All of this makes it extremely easy to install, even in weird
environments where modern tools like pip are not available.  I
don't ever want to change this, so VF-1 does *not* depend on
chardet.  But, if you install it yourself, VF-1 will recognise that
it's there and adopt the alternative strategy of autodetecting
the encoding if UTF-8 fails, and will drop back to the
user-specified encoding only if chardet fails to identify an
encoding with confidence above 0.5.  With chardet installed, I
was able to use VF-1 to cruise around some Russian gopher sites
tengu linked me to, and whether I encountered UTF-8, KOI8-R or
CP1251 encoding, it all Just Worked, which was tremendously
satisfying.  VF-1+chardet seems bullet-proofly international,
which is fantastic.

As an aside, I was amused to note that the chardet FAQ[4] has the
following entry:

> Yippie! Screw the standards, I'll just auto-detect everything!

> Don't do that. Virtually every format and protocol contains
> a method for specifying character encoding.

The FAQ goes on to talk about HTTP, HTML, XML, etc.  Out here on
the plain text frontier, of course, there ain't any such thing
(well, maybe the /caps.txt hack does something about this,
hmm...), so I don't feel bad at all about auto-detecting
everything.  It's pretty much the only choice we have.

For the record, this is *not* something that I think it is
worth extending gopher to work around.  There is a much simpler
and nicer solution, which is simply to use UTF-8 for absolutely
all new content in gopherspace, so that there is no *need* to
explicitly specify the character encoding.

That's all that's new, aside from some tiny tidy ups and fixes.
There are a few other small things I'd like to tackle, but it's
starting to feel pretty complete for me.

[1] gopher://circumlunar.space:70/1/~tfurrows/tips/
[2] gopher://circumlunar.space:70/0/~solderpunk/phlog/looking-at-long-stuff-with-vf1.txt
[3] gopher://circumlunar.space:70/1/~tengu/
[4] https://chardet.readthedocs.io/en/latest/faq.html