Subj : Re: FTN  / Usenet to a new BBS
To   : Vk3jed
From : tenser
Date : Thu May 14 2020 02:23 pm

On 14 May 2020 at 11:55a, Vk3jed pondered and said...
 
 Vk> -=> On 05-14-20 05:08, tenser wrote to Vk3jed <=-
 Vk> 
 Vk>  te> That's a pretty heavy-weight approach, but is a
 Vk>  te> possibility.  I've already got code that will
 Vk> 
 Vk> True. :)
 Vk> 
 Vk>  te> transfer data via binkp, and I can read and parse
 Vk>  te> FTN messages; I just need to stream those to the
 Vk>  te> NNTP server and add some glue code, then write
 Vk>  te> a little more to accept streamed NNTP messages
 Vk>  te> from INN, translate those to FTN format and send
 Vk>  te> them to the hub and again some glue code.
 Vk> 
 Vk> Sounds like you're well on the way, but almost reinventing the wheel
 Vk> with your own NNTP - FTN gateway effectively! :)

Indeed, that's true.  Part of the reason for that
is that most of the existing code that does this is
either a) wired into some other BBS package, or b)
has atrophied over time from neglect and lack of
maintenance.  E.g., `fidogate`.

Also, `binkd` is not code I trust.  I ran into far
too many bugs far too quickly with it.  Just looking
at `hpt` and other similar things similarly.

Further, nothing seems to be particularly resilient
to handling crashes well: just opening a file and
writing into it isn't really sufficient, because the
OS may buffer the data, etc.  `ginko` is very careful
with the ordering of filesystem operations and
ensuring that the OS has actually written data to
the storage media before acknowledging receipt; a
crash at any point will leave the data store in a
way that's recoverable with no data loss after the
system comes back up.

Finally, I wanted to address issues around latency
that I saw as being problematic.  This is part of
understanding the larger design space: there's a
tension between having one program that's fast (because
all the steps to handling message delivery, for example
are centralized in one place) but also modular for
security and separation of concerns issues that then
implies slowness: splitting things up into multiple
binaries implies coordination between the two.  Binkd
may be able to invoke HPT directly, which can then
invoke fidogate, but what about on the return path?
Everything I've seen so far seems to be based on
polling at some fixed interval, possibly combined
with semaphore files, all of which is suboptimal.

But since I know I'm running on systems that provide
primitives for intra-machine IPC based on things like
Unix domain sockets, I can "ping" services directly;
basically, I have multiple daemons that coordinate
with each other via doorbells.  This is easy in a
language like Go, where I can spin up a dedicated
thread that spins on a Unix domain socket; another
process can connect to that socket and immediately
disconnect, effectively creating a doorbell that says,
"there's work for you; scan your input queue."  So
you synchronize on the filesystem, but the polling
and semaphore files are replaced by sockets and
blocking events in dedicated threads.  Since those
threads spend most of their lives blocked on an IPC
even from the OS, it's less resource-intensive (no
spinning) and lower-latency (no waiting for the
polling interval; the thread becomes unblocked as
soon as someone rings the doorbell).

So yeah, it's reinventing an FTN<->NNTP gateway, but
with some added niceties.

Finally, since this is all green-field code in a
language that encourages and facilitates testing,
we can up our game in the testing department.

--- Mystic BBS v1.12 A46 2020/04/20 (Windows/32)
 * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

.