Chris Blankenship's Ramblings

Recently I was looking for a client based application that could monitor the availability of a URL address.  One of my customers asked for the functionality in parallel to their enterprise monitoring solutions in order to let him watch their URLs from his desktop.  I did some poking around the internet and came across an interesting solution called WebMonitor.  So before I made any recommendations I decided to give it a test run to see how well it worked.

After my initial testing went good I contacted my customer and recommended he take a look at the application.  Almost immediately after hanging up the phone I started wondering how WebMonitor was making the HTTP requests to check if a URL is valid.  So I fired up my handy WireShark and watched a URL check.  Here is the relevant result:

GET /default.aspx HTTP/1.1
User-Agent: WebMonitor UserAgent; v1.0.0.0
Host: www.dscoduc.com
Connection: Keep-Alive

I was disappointed to learn that a GET request is being used for what is an equivalent to an “Is-Alive” call.  Using a GET request it turned out that the bytes sent from my website to the WebMonitor was almost 34kb.

It turns out there is a better way to check if a URL is alive, the HEAD request.  The current RFC for HTTP outlines the use of the HEAD requests:

The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification.

A quick check on the internet and I found an article that outlines the use of a HEAD request.  So armed with that information I decided to open up the source code of WebMonitor and see if there would be a significant difference between using a HEAD vs. GET request…

First I found the section in the source code that actually sends the HTTP request to the targeted URL in the Monitor.cs file, line 104.

request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = this.userAgent;

Next I added a single line to change the default behavior of a GET to a HEAD:

request.Method = "HEAD";

Now it seems that using a HEAD request from my website to the WebMonitor was only 193bytes.  A significant difference, meaning it’s clear that using the HEAD request would be more efficient for testing the availability of a URL.

It’s important to point out that WebMonitor has an option to search for the inclusion/exclusion of text in the result of the URL check.  For example, suppose you needed to ensure a response included some text to be sure that everything was working correctly.  In this case a GET request would be appropriate over a HEAD request.  So to accommodate this configuration it is necessary to wrap the update in an IF statement that looks something like the following:

if (!this.rules.RunCollectHtml)
    request.Method = "HEAD";

And that’s what it takes to make WebMonitor perform more effecient “Is-Alive” checks.