From: gopher-project-bounces+rachael=telefisk.org@lists.alioth.debian.org Date: Thu Dec 8 15:50:38 2011 Subject: Re: [gopher] Gopher Proxy & Archive robots.txt On Dec 8, 2011, at 16:38 , Wolfgang F. wrote: > The gopher proxy & archive ( http://gopher-project:gpa-test-675@gpa.internetearth.com/ ) should now pay attention to robots.txt exclusion rules. I haven't tested this extensively, but it seems to work. Does anybody know of any gopher pages that have been blocked by robots.txt so I can test? I have a robots.txt in the root of gophernicus.org - it's (trying to) blocking my scratch coding directory with ever-changing number of source files in it. - Kim _______________________________________________ Gopher-Project mailing list Gopher-Project@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project Thread start (DIR) [gopher] Gopher Proxy & Archive robots.txt (DIR) Followup: Re: [gopher] Gopher Proxy & Archive robots.txt (DIR) Followup: Re: [gopher] Gopher Proxy & Archive robots.txt