Subj : Re: Dupe checking
To   : Dale Shipp
From : Maurice Kinal
Date : Sat Apr 10 2004 09:02 am

Hey Dale!

Apr 09 23:35 04, Dale Shipp wrote to Bo Simonsen:

 DS>   Not so.   Squish has the ability to use two different sorts of
 DS>   duplicate checking.   One is based on MSGID.

Which can be duped without the actual messsage being a dupe.

 DS> The other is based on
 DS>   the header info (TO, FROM, SUBJ and DATE).  If *either* of these 
 DS> two
 DS>   things is a match, then Squish calls the message a dupe.

The problem with this is that often some of these fields are altered such as 
the SUBJ field, which is quite common I've noticed.  Thus an actual dupe could 
slip by or nondupes end up a dupes.  The DATE, containing time, is based on the 
originating computer's time which can malfunction without the computer 
malfunctioning (ie create unique messages all with the exact same time and 
date).  The TO and FROM can be the same without the messages being dupes.

The only true way is to check the actual message but I wouldn't do that for 
every one coming through but instead if it fails any of the preliminary tests 
then only those would get a more rigorous test ... maybe.  :-)

The thing is that there is too much broken, unsupported stuff out there and far 
too many kludges being tacked on along the way by everyone and their dog.  
Personally I have gotten to the stage where I am just stripping everything out 
of messages, minus the TO, FROM, DATE (<- ignoring it though), and getting the 
originating nodenumber from the Origin.  Everything else is known without 
jumping through hoops once they are stripped.  Without compression I found that 
I average a 50% reduction in archive size when compared to normal Fido 
archiving methods.

Life is good,
Maurice

--- Msged/LNX 6.1.1
 * Origin: Little Mikey's Brain BBS - A work in progress (1:153/401.1)

.