----------------------------------------
       robots.txt
       May 24th, 2019
       ----------------------------------------
       
       I've run a little disallow robots.txt file on this gopher hole for
       a while. Now there's been more discussion about its inclusion on
       gopher for gopher search engines. This leads me to a little
       problem.
       
       My robots.txt file is not meant for gopher crawlers. I created it
       because I do not want my site crawled by web crawlers trawling
       through proxies to gopher. There's some question as to whether
       a robots.txt file is effective in that way (technically the proxy
       should have one, not the site itself), but I thought it better
       than nothing.
       
       Now I have a dilemma. I would very much like my site to be crawled
       by Veronica-2 and other gopher-centric search engines, but I don't
       want Google plopping my burrowy-thoughts on the web. What's
       a gopher to do?