Reddit Blocks Crawlers
======================
A decent search engine will look at the file "robots.txt" before indexing
pages. If the crawler is disallowed under the directory containing the
text file, it will not index. I read yesterday on Ars Technica[1] that 
Reddit disallows non-Google search engines to index recent results from
their site. Following is what the text file "https://reddit.com/robots.txt" 
looks like:
# Welcome to Reddit's robots.txt
# Reddit believes in an open internet, but not the misuse of public content.
# See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.
# See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.
# policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy

User-agent: *
Disallow: /

Wow! Where will I be without search results from Reddit? I guess I will get
the info I need from sites like the How To Geek[2] 

Someone had quoted someone's reply on the article and added the comment
"F*CK REDDIT"! I gave it an upvote. Found nothing bad about that comment.
And I don't like it when people say that they cannot find alternatives to
Reddit. That's a pain! How about being united? And do you know how many
accounts I have deleted or have just lost?
Feel free to use an internet archive to save contents you don't want to 
lose.
That's it! I don't care because the wen has become a PITA!

[1] https://pnqk.me/sxyjk4
[2] https://www.howtogeek.com/