From: gopher-project-bounces+rachael=telefisk.org@lists.alioth.debian.org Date: Wed May 23 22:23:59 2012 Subject: Re: [gopher] Robots.txt On 2012-05-23, Kim Holviala wrote: > On May 23, 2012, at 23:17 , Nick Matavka wrote: > >> Could I have the spec to, or perhaps somebody's explanation of, the >> robots.txt file? This will help me to explain it in my RFC. > > http://www.robotstxt.org/orig.html (AFAIK, the gopher crawlers out there which respect robots.txt, and the server administrators who have such files, are following the same format used in the web.) -- Nuno J. Silva (aka njsg) gopher://sdf-eu.org/1/users/njsg http://njsg.sdf-eu.org/ _______________________________________________ Gopher-Project mailing list Gopher-Project@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project Thread start (DIR) [gopher] Robots.txt (DIR) Followup: Re: [gopher] Robots.txt