______________________________________________

			  AN IDEA FOR BRINGING PHLOGGING TO THE MASSES

								  lro
			 ______________________________________________


							<2019-01-30 Wed>


Table of Contents
_________________

An idea
The Prototype
.. Imports
.. Global vars
.. Getting email
.. Mapping emails to users
.. From
.. Is that email address in the dictionary?
.. Phlog post
.. Write phlog post to file
.. Update gophermap
.. New Phlog entry
.. iterating
Conclusion





An idea
=======

  I had a what I think might be a good idea to help people get setup
  phlogging, and thats an email based phlogging service. Where a user
  would email say post@example.com and the subject is their chosen
  title, and the cotents of the emial is the phlog post.

  Each user would be identified by their email address, registrations
  would probably happen on a webpage, with a captcha to stop spammers
  etc.

  I would have each user have a user on the server, and use gophernicus
  to display the user gopher directory to host the phlog. An email bot
  would just watch the inbox of post@example.com and create the phlog
  post, and move it to the user gopher directory, and update the
  gophermap.

  Then we can have a master page at example.com:70 with the most
  recently posted phlogs.


The Prototype
=============

  I figured a good course of action was to setup a prototype using my
  sdf account. I will leave the web based registration for later as that
  seems the most trivial, I am more interested in getting what is
  essentially an email bot setup to watch my email and add phlog posts.

  For ease of writing and ease of development by others, i'm going to
  write it in python. Not a language I've used alot, but this should
  make things easier for others to jump in and modify and run the code
  themselves.

  What my initial implementation does is look for unread emails in a
  mailfile, and create a file named with todays date in that users
  directory as the phlog post. And updates the gophermap with the new
  phlog entry at the top with the subject of the email as a nice little
  description/title.


Imports
~~~~~~~

  I need to import a few libraries for various things, like reading the
  mailbox file, importing json objects from a file, and getting the
  date/time.

  ,----
  | #!/usr/bin/env python3
  | 
  | ###################################################################
  | # To the extent possible under law, the person who associated CC0 #
  | # with phlog_bot.py has waived all copyright and related or       #
  | # neighboring rights to phlog_bot.py.                             #
  | #                                                                 #
  | # https://creativecommons.org/publicdomain/zero/1.0/              #
  | ###################################################################
  | 
  | # NO WARRANTY!!!! 
  | # PLEASE DON'T RUN THIS SCRIPT UNLESS YOU HAVE BACKUPS OF YOUR
  | # MAILFILE OR ANYTHING ELSE THIS SCRIPT TOUCHES PLEASE.
  | 
  | import mailbox
  | import json
  | import time
  | import datetime
  | import os
  | 
  `----


Global vars
~~~~~~~~~~~

  Global variables.

  ,----
  | gopher_dir = "/sdf/arpa/gm/l/lro/test"
  | userdictionaryfile = "users.json"
  | mailboxf = "/mail/lro"
  | today = datetime.date.today()
  | today_fmt = today.strftime("%Y")+"-"+today.strftime("%m")+"-"+today.strftime("%d")
  | 
  `----


Getting email
~~~~~~~~~~~~~

  The first step is to open the email and get the contents, then collect
  the unread emails. For this prototype i'm just going to read my sdf
  mailspool file at `/mail/lro'.

  To get the unread emails I need to get all the mail from the file and
  return all the mail that doesn't have the read (R) flag set and mark
  all the new mail as read. Which as it turns out is a bit weird to set
  the read flag in mail, which is why I have to keep track of the index
  into /lromailbox/.

  See if you just use `add_flag' on /message/, it doesn't write that
  back to /lromailbox/, so to get around that I set the index into
  /lromailbox/ to the altered /message/, then when I write /lromailbox/
  back to disk the read flag is now set.

  And we return a list of all the unread mail.

  ,----
  | def get_unread_emails(mailboxfile):
  |     unread = []
  |     if not os.path.isfile(mailboxfile):
  |         print("Mailboxfile doesn't exist: %s" % (mailboxfile))
  |         return unread
  |     
  |     mailboxfp = mailbox.mbox(mailboxfile)
  |     ind = 0
  |     for message in mailboxfp:
  |         flags = message.get_flags()
  |         if flags.find('R') == -1:
  |             unread.append(message)
  |             message.add_flag('R')
  |             mailboxfp[ind] = message
  |         ind = ind + 1
  | 
  |     mailboxfp.lock()
  |     mailboxfp.flush()
  |     mailboxfp.unlock()
  |     mailboxfp.close()
  |     return unread
  | 
  `----


Mapping emails to users
~~~~~~~~~~~~~~~~~~~~~~~

  I need to store a dictionary of mappings of email addresses to
  usernames. At this point i've chosen to go with JSON so I can read it
  in easily from a file on disk.

  For the time being this is file is handled manually, but in the
  production version will be handled by the registration program.

  ,----
  | {"lrojr@example.com" : "lrojr", "lrosr@example.com" : "lrosr"}
  `----

  ,----
  | def load_dictionary(dictfile):
  |     if not os.path.isfile(dictfile):
  |         return []
  |     
  |     usersfile = open(dictfile)
  |     userdictionary = json.load(usersfile)
  |     usersfile.close()
  |     return userdictionary
  | 
  `----


From
~~~~

  What we need next is a function that gets the from address from each
  unread message and transforms it to lower case so that the output can
  be used to search through /userdictionary/.

  ,----
  | def get_from_lower(message):
  |     From = message.get_from()
  |     return From.split(" ")[0].lower()
  | 
  `----


Is that email address in the dictionary?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Checks if an email is one of our registered users by trying to get
  their username from the dictionary, returning the username if they
  are, otherwise returns False.

  ,----
  | def user_in_dict(email, dictionary):
  |     try:
  |         return dictionary[email]
  |     except KeyError:
  |         return False
  | 
  `----


Phlog post
~~~~~~~~~~

  Gets new phlog post text from email.

  ,----
  | def phlog_post_contents(email):
  |     return email.get_payload()
  | 
  `----


Write phlog post to file
~~~~~~~~~~~~~~~~~~~~~~~~

  Writes the phlog post file to disk. /filename/ includes full path to
  file.

  If file already exists then add a "-x" to the end.

  ,----
  | def write_phlog_post(filename, contents):
  |     ind = 0
  |     justfilenamenotpath = filename.split("/")[-1]
  |     while os.path.isfile(filename):
  |         if ind == 99:
  |             break
  |         filename = filename.split(".")[0].split("_")[0]+"_"+str(ind)+".txt"
  |         justfilenamenotpath = filename.split("/")[-1]
  | 
  |     with open(filename, "w") as phlogfile:
  |         phlogfile.write(contents)
  | 
  |     return justfilenamenotpath
  | 
  `----


Update gophermap
~~~~~~~~~~~~~~~~

  Updates the phlog gophermap with the new phlog post.

  /gophermap/ is the full path and filename for the gophermap.

  ,----
  | def update_gophermap(gophermap, filename, description):
  |     entry = "0%s	%s" % (description, filename)
  | 
  |     with open(gophermap, "r") as gfp:
  |         buf = gfp.readlines()
  | 
  |     with open(gophermap, "w") as gfp:
  |         for line in buf:
  |             if "==================================================" in line:
  |                 line = line+"\n"+entry+"\n"
  |             gfp.write(line)
  | 
  `----


New Phlog entry
~~~~~~~~~~~~~~~

  Creates a new phlog entry, and updates the gophermap.

  ,----
  | def new_phlog_entry(message):
  |     email = get_from_lower(message)
  |     username = user_in_dict(email, userdictionary)
  |     if not username:
  |         print("User: %s not in user dictionary" % (email))
  |         return
  |     phlog_title = message['subject']
  |     phlog_filename = today_fmt + ".txt"
  |     phlog_contents = phlog_post_contents(message)
  |     phlog_dir = gopher_dir+"/"+username
  |     phlog_gophermap = phlog_dir+"/gophermap"
  |     phlog_filename = write_phlog_post(phlog_dir+"/"+phlog_filename, phlog_contents)
  |     update_gophermap(phlog_gophermap, phlog_filename, phlog_title+" <"+today_fmt+">")
  |     update_gophermap(gopher_dir+"/gophermap", username+"/"+phlog_filename, username+" - "+phlog_title+" <"+today_fmt+">")
  |     print("Added phlog post for %s" % (username))
  | 
  `----


iterating
~~~~~~~~~

  The final step is to iterate over all the unread emails and make them
  phlog posts.

  ,----
  | userdictionary = load_dictionary(userdictionaryfile)
  | 
  | for message in get_unread_emails(mailboxf):
  |     new_phlog_entry(message)
  | 
  `----


Conclusion
==========

  So thats the initial prototype done, I don't know what I am going to
  do with it though, whether I should host a small gopher server and set
  up a new email or not. If someone else wants too, bloody go for it!

  This could also be used personally by anyone as well, just add a
  snippet to the /get_unread_emails/ that checks the subject for ones
  that start with [PHLOG_POST] or something and have it watch your mail
  account.

  If your worried about security you could always GPG sign the emails
  and have the dictionary verify each email, but that adds alot of
  complexity.

  Feature requests welcome!