________________

							 PHLOG-BOT V0.2

								  lro
							________________


							<2019-02-01 Fri>


Table of Contents
_________________

Some improvements
.. Imports
.. Global vars
.. Getting email
.. Mapping emails to users
.. From
.. Is that email address in the dictionary?
.. Phlog post
.. Write phlog post to file
.. Add phost to gophermap
.. Update gophermap
.. Create phlog archive directory
.. New Phlog entry
.. iterating
Conclusion





Some improvements
=================

  This is version 0.2 of phlog-bot.py, now with a rotation of
  gophermaps.

  They are rotated so that the main phlog gophermap has the 10 top
  latest phosts, and there is an archive sorted by each month linked at
  the bottom of the gophermap.


Imports
~~~~~~~

  I need to import a few libraries for various things, like reading the
  mailbox file, importing json objects from a file, and getting the
  date/time.

  ,----
  | #!/usr/bin/env python3
  | 
  | ###################################################################
  | # To the extent possible under law, the person who associated CC0 #
  | # with phlog-bot.py has waived all copyright and related or       #
  | # neighboring rights to phlog-bot.py.                             #
  | #                                                                 #
  | # https://creativecommons.org/publicdomain/zero/1.0/              #
  | ###################################################################
  | 
  | # NO WARRANTY!!!! 
  | # PLEASE DON'T RUN THIS SCRIPT UNLESS YOU HAVE BACKUPS OF YOUR
  | # MAILFILE OR ANYTHING ELSE THIS SCRIPT TOUCHES PLEASE.
  | 
  | import mailbox
  | import json
  | import time
  | import datetime
  | import os
  | 
  `----


Global vars
~~~~~~~~~~~

  Global variables.

  ,----
  | gopherdir = "/home/lro/gopher/test"
  | userdictionaryfile = "users.json"
  | mailboxf = "test-mbox"
  | today = datetime.date.today()
  | todayfmt = today.strftime("%d")
  | yearmonth = today.strftime("%Y")+"-"+today.strftime("%m")
  | 
  `----


Getting email
~~~~~~~~~~~~~

  The first step is to open the email and get the contents, then collect
  the unread emails. For this prototype i'm just going to read my sdf
  mailspool file at `/mail/lro'.

  To get the unread emails I need to get all the mail from the file and
  return all the mail that doesn't have the read (R) flag set and mark
  all the new mail as read. Which as it turns out is a bit weird to set
  the read flag in mail, which is why I have to keep track of the index
  into /lromailbox/.

  See if you just use `add_flag' on /message/, it doesn't write that
  back to /lromailbox/, so to get around that I set the index into
  /lromailbox/ to the altered /message/, then when I write /lromailbox/
  back to disk the read flag is now set.

  And we return a list of all the unread mail.

  ,----
  | def get_unread_emails(mailboxfile):
  |     unread = []
  |     if not os.path.isfile(mailboxfile):
  |         print("Mailboxfile doesn't exist: %s" % (mailboxfile))
  |         return unread
  |     
  |     mailboxfp = mailbox.mbox(mailboxfile)
  |     ind = 0
  |     for message in mailboxfp:
  |         flags = message.get_flags()
  |         if flags.find('R') == -1:
  |             unread.append(message)
  |             message.add_flag('R')
  |             mailboxfp[ind] = message
  |         ind = ind + 1
  | 
  |     mailboxfp.lock()
  |     mailboxfp.flush()
  |     mailboxfp.unlock()
  |     mailboxfp.close()
  |     return unread
  | 
  `----


Mapping emails to users
~~~~~~~~~~~~~~~~~~~~~~~

  I need to store a dictionary of mappings of email addresses to
  usernames. At this point i've chosen to go with JSON so I can read it
  in easily from a file on disk.

  For the time being this is file is handled manually, but in the
  production version will be handled by the registration program.

  ,----
  | {"lrojr@example.com" : "lrojr", "lrosr@example.com" : "lrosr"}
  `----

  ,----
  | def load_dictionary(dictfile):
  |     if not os.path.isfile(dictfile):
  |         return []
  |     
  |     usersfile = open(dictfile)
  |     userdictionary = json.load(usersfile)
  |     usersfile.close()
  |     return userdictionary
  | 
  `----


From
~~~~

  What we need next is a function that gets the from address from each
  unread message and transforms it to lower case so that the output can
  be used to search through /userdictionary/.

  ,----
  | def get_from_lower(message):
  |     From = message.get_from()
  |     return From.split(" ")[0].lower()
  | 
  `----


Is that email address in the dictionary?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Checks if an email is one of our registered users by trying to get
  their username from the dictionary, returning the username if they
  are, otherwise returns False.

  ,----
  | def user_in_dict(email, dictionary):
  |     try:
  |         return dictionary[email]
  |     except KeyError:
  |         return False
  | 
  `----


Phlog post
~~~~~~~~~~

  Gets new phlog post text from email.

  ,----
  | def phlog_post_contents(email):
  |     return email.get_payload()
  | 
  `----


Write phlog post to file
~~~~~~~~~~~~~~~~~~~~~~~~

  Writes the phlog post file to disk. /filename/ includes full path to
  file.

  If file already exists then add a "-x" to the end and return the new
  filename.

  ,----
  | def write_phlog_post(filename, contents):
  |     ind = 0
  |     justfilenamenotpath = filename.split("/")[-1]
  |     while os.path.isfile(filename):
  |         if ind == 99:
  |             break
  | 
  |         ind += 1
  |         filename = filename.split(".")[0].split("_")[0]+"_"+str(ind)+".txt"
  |         justfilenamenotpath = filename.split("/")[-1]
  | 
  |     with open(filename, "w") as phlogfile:
  |         phlogfile.write(contents)
  | 
  |     return justfilenamenotpath
  | 
  `----


Add phost to gophermap
~~~~~~~~~~~~~~~~~~~~~~

  adds another phost to the gophermap without deletion, used for the
  archive folders.

  ,----
  | def add_gophermap(gophermap, filename, description):
  |     entry = "0%s	%s" % (description, filename)
  | 
  |     with open(gophermap, "r") as gfp:
  |         buf = gfp.readlines()
  | 
  |     with open(gophermap, "w") as gfp:
  |         for line in buf:
  |             if "==================================================" in line:
  |                 line = line+"\n"+entry
  |             gfp.write(line)
  | 
  `----


Update gophermap
~~~~~~~~~~~~~~~~

  Updates the phlog gophermap with the new phlog post and deletes an
  entry off the end if more than 10 posts in the gophermap.

  /gophermap/ is the full path and filename for the gophermap.

  ,----
  | def update_gophermap(gophermap, filename, description):
  |     entry = "0%s	%s" % (description, filename)
  | 
  |     with open(gophermap, "r") as gfp:
  |         buf = gfp.readlines()
  | 
  |     posts = 0
  |     for line in buf:
  |         if line.startswith("0"): 
  |             posts += 1
  |             if posts > 10:
  |                 buf.remove(line)
  |                 posts -= 1
  | 
  |     with open(gophermap, "w") as gfp:
  |         for line in buf:
  |             if "==================================================" in line:
  |                 line = line+"\n"+entry
  |             gfp.write(line)
  | 
  `----


Create phlog archive directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  This functions checks if the archive dir for this moth exists, if it
  doesn't it creates it.

  ,----
  | def create_archive_dir(phlogdir, yearmonth):
  |     if not os.path.isdir(phlogdir+"/"+yearmonth):
  |         os.mkdir(phlogdir+"/"+yearmonth)
  |         with open(phlogdir+"/"+yearmonth+"/gophermap", "w+") as f:
  |             f.write("i==================================================\n")
  |         with open(phlogdir+"/gophermap", "a") as f:
  |             f.write("\n\n1Archive of phlosts from: "+yearmonth+"	"+yearmonth)
  `----


New Phlog entry
~~~~~~~~~~~~~~~

  Creates a new phlog entry, and updates the gophermap.

  ,----
  | def new_phlog_entry(message):
  |     email = get_from_lower(message)
  |     username = user_in_dict(email, userdictionary)
  |     if not username:
  |         print("User: %s not in user dictionary" % (email))
  |         return
  | 
  |     phlogtitle = message['subject']
  |     phlogfilename = todayfmt + ".txt"
  |     phlogcontents = phlog_post_contents(message)
  |     phlogdir = gopherdir+"/"+username
  | 
  |     create_archive_dir(phlogdir, yearmonth)
  | 
  |     phlogfilename = write_phlog_post(phlogdir+"/"+yearmonth+"/"+phlogfilename, phlogcontents)
  |     ymd = "<"+yearmonth+"-"+todayfmt+">"
  | 
  |     update_gophermap(phlogdir+"/gophermap", yearmonth+"/"+phlogfilename, phlogtitle+" "+ymd)
  |     add_gophermap(phlogdir+"/"+yearmonth+"/gophermap", phlogfilename, phlogtitle+" "+ymd)
  |     update_gophermap(gopherdir+"/gophermap", username+"/"+yearmonth+"/"+phlogfilename, username+" - "+phlogtitle+" "+ymd)
  |     print("Added phlog post for %s" % (username))
  | 
  `----


iterating
~~~~~~~~~

  The final step is to iterate over all the unread emails and make them
  phlog posts.

  ,----
  | userdictionary = load_dictionary(userdictionaryfile)
  | 
  | for message in get_unread_emails(mailboxf):
  |     new_phlog_entry(message)
  | 
  `----


Conclusion
==========

  I think I might have to setup a test instance of thus for some of you
  out there to use. I wil start getting some stuff together so that I
  can create my own pubnix instance, I do have a couple of raspberry
  pi's lying around that I could use initially...