From: "Gopher-Project" <gopher-project-bounces+rachael=telefisk.org@lists.alioth.debian.org>
       Date: Mon Dec 29 00:43:24 2014
       Subject: Re: [gopher] Spidering the gopherspace
       
       Mateusz,
       
       > I am spidering the gopherspace since a few days, collecting things for a 
       > new project I will publish soon.
       > 
       > If you see 178.170.108.36 hammering your gopher server, that would be me.
       > 
       > Do not hesitate complaining if you notice I'm too hard on you, I will 
       > try to spare your server then (I already see that sdf.org have 
       > blacklisted me).
       
       FWIW, I throttled several minutes between requests to the same IP (or would
       find another to visit in the meantime) and I always honour robots.txt if it
       can be fetched (and cache it).
       
       However, since V-2 only fetches menus and has a well-known reverse DNS, I
       imagine sites are a little friendlier to me.
       
       -- 
       ------------------------------------ personal: http://www.cameronkaiser.com/ --
         Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser@floodgap.com
       -- All things are possible, except skiing through a revolving door. -----------
       
       _______________________________________________
       Gopher-Project mailing list
       Gopher-Project@lists.alioth.debian.org
       http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project
       Thread start
 (DIR) [gopher] Spidering the gopherspace
 (DIR) Followup: Re: [gopher] Spidering the gopherspace
 (DIR) Followup: Re: [gopher] Spidering the gopherspace
 (DIR) Followup: Re: [gopher] Spidering the gopherspace
 (DIR) Followup: Re: [gopher] Spidering the gopherspace
 (DIR) Followup: Re: [gopher] Spidering the gopherspace