One of the last steps in configuring your BlogEngine site is to update your Robots file. Check out Wikipedia for a complete explanation: Here is a brief description:
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.
BlogEngine 1.3.0.0 comes with a default entry that looks like the following:
#Remove the ‘#’ character below and replace example.com with your own website address.
#sitemap: http://example.com/sitemap.axd
It looks like future versions of BlogEngine will have an updated Robots.txt file that looks like the following:
User-agent: *
Disallow: /login.aspx
Disallow: /search.aspx
Disallow: /error404.aspx#Remove the ‘#’ character below and replace example.com with your own website address.
#sitemap: http://example.com/sitemap.axd
So verify that your Robots.txt file has been updated to reflect your site. Once your done then I would suggest you head over to Google and validate your Robots.txt file is working correctly.
Wed, Jan 30, 2008
Technology