Posted January 30, 2008 and filed under Technology    tags:  ,

One of the last steps in configuring your BlogEngine site is to update your Robots file.  Check out Wikipedia for a complete explanation:  Here is a brief description:

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

BlogEngine 1.3.0.0 comes with a default entry that looks like the following:

#Remove the '#' character below and replace example.com with your own website address.
#sitemap: http://example.com/sitemap.axd

It looks like future versions of BlogEngine will have an updated Robots.txt file that looks like the following:

User-agent: *
Disallow: /login.aspx
Disallow: /search.aspx
Disallow: /error404.aspx

#Remove the '#' character below and replace example.com with your own website address.
#sitemap: http://example.com/sitemap.axd

So verify that your Robots.txt file has been updated to reflect your site.  Once your done then I would suggest you head over to Google and validate your Robots.txt file is working correctly.

If you liked this article why not share it with others?

Kick it up to DotNetKicks.com

Comments

Add comment


(Will show your Gravatar icon)

biuquote
Loading