Many of you who have followed my blog will have heard me <rant></rant> about web crawlers and spiders that bombard my site with unnecessary traffic. In the past I have created BlogEngine.NET extensions that blocked connections by UserAgent strings, by IP Addresses, and whatever else I could figure out to try and mitigate this nuisance. Today I release the fruits of my labor in this subject; GateKeeper Web Access Management.
GateKeeper is an ASP.NET 2.0 HttpModule based management solution that employs several filtering mechanisms to try an catch those sneaky crawlers and unwanted guests who like to ignore your robots.txt directions. Additionally, it is possible to maintain a list of approved override settings to prevent accidental blocking of legitimate requests. You can configure your site to trap requestors who have disregarded your robots.txt file and proceeded to follow every link on your site. Finally you can configure GateKeepr to block hotlinking to help reduce bandwidth stealers.
Why not head on over to the project page and read more details about this solution?
CSS-Tricks
Janko at Warp Speed
Kim Cameron's Identity Weblog
Pluralsight Blogs
Mon, Jan 12, 2009
Security, Technology