From: "Gopher-Project" <gopher-project-bounces+rachael=telefisk.org@lists.alioth.debian.org> Date: Mon Dec 29 00:43:24 2014 Subject: Re: [gopher] Spidering the gopherspace Mateusz, > I am spidering the gopherspace since a few days, collecting things for a > new project I will publish soon. > > If you see 178.170.108.36 hammering your gopher server, that would be me. > > Do not hesitate complaining if you notice I'm too hard on you, I will > try to spare your server then (I already see that sdf.org have > blacklisted me). FWIW, I throttled several minutes between requests to the same IP (or would find another to visit in the meantime) and I always honour robots.txt if it can be fetched (and cache it). However, since V-2 only fetches menus and has a well-known reverse DNS, I imagine sites are a little friendlier to me. -- ------------------------------------ personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser@floodgap.com -- All things are possible, except skiing through a revolving door. ----------- _______________________________________________ Gopher-Project mailing list Gopher-Project@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project Thread start (DIR) [gopher] Spidering the gopherspace (DIR) Followup: Re: [gopher] Spidering the gopherspace (DIR) Followup: Re: [gopher] Spidering the gopherspace (DIR) Followup: Re: [gopher] Spidering the gopherspace (DIR) Followup: Re: [gopher] Spidering the gopherspace (DIR) Followup: Re: [gopher] Spidering the gopherspace