Robots.txt File

I recently googled something that I knew was on my blog. Google didn’t find it. I could not believe it. I was going to write an article about how bad google is, and how it only returns commercial sites anymore, etc. etc. etc. After looking into it a little more, I found that my stupid robots.txt file on my webserver was set up to deny all spiders, and I would say that it has been this way for about a year now. Basically, google, and probably all other crawlers have not indexed this site in over a year. I think that the linux distro I use must set up a default robots.txt file which deny’s all webcrawlers, such as google’s Googlebot. I’m not sure why they would do that. If you want people to find you and your blog, you have to let google crawl it.