Regarding Your Performance - Remote Windows Performance Monitoring

Posted by Tom on 2010-12-24 11:36

So what with my work on Oh Teh Noes I was wondering if I could co-opt it to do performance monitoring. As it happens, I couldn't figure out how to do it without doing some serious square-pegging, but it did get me thinking about how I would like to attack the problem. And as is so often that case that eventually turned into me actually attacking the problem. That happens a lot. I nerd snipe myself daily.

And thus Regarding Your Performance was born. It consists of a Windows service which sits on the host machine and farms basic performance data, and a HTML page which collects that data from a list of hosts via AJAX and drops it into some graphs. It became an excuse for me to learn some new things, so I thought I'd share the experience. This is a fairly high-speed tour so you might want to saunter over to Github and grab the source

InstallUtil Can Die in a Fire

I wanted the agent (the side of the system which resides on the target host and harvests delicious, delicious performance information) to run as a Windows service. But there's one particular element of Windows services that I find especially uncomfortable . . .

InstallUtil.exe can piss right off.

I've been packaging that dowdy little app with all of my Windows Services for years now and enough is enough. It's time to find a more elegant solution. Fire up Reflector and it quickly becomes apparent that InstallUtil doesn't actually do a lot. In essence it simply creates a ManagedInstaller, which itself is a thin wrapper around a TransactedInstaller. By taking that code and then adding on the contents of the usual Service designer files you can roll your own pretty easily. I ended up using command line arguments to install and uninstall the service, resulting in a Main method like this:

static void Main(string[] args)
{
    if (args.Length > 0)
    {
        if (args[0] == "-i" || args[0] == "-u")
        {
            ServiceInstaller serviceInstaller = new ServiceInstaller();
            serviceInstaller.ServiceName = "Performance.Agent.Service";
            serviceInstaller.StartType = ServiceStartMode.Automatic;
            serviceInstaller.DisplayName = "Colourblind Performance Agent";
            serviceInstaller.Description = "Agent for the Colourblind performance monitor";
 
            ServiceProcessInstaller processInstaller = new ServiceProcessInstaller();
            processInstaller.Account = ServiceAccount.LocalSystem;
            processInstaller.Username = null;
            processInstaller.Password = null;
 
            TransactedInstaller installer = new TransactedInstaller();
            installer.Installers.Add(processInstaller);
            installer.Installers.Add(serviceInstaller);
            installer.Context = new InstallContext("install.log", null);
            installer.Context.Parameters.Add("assemblypath", Assembly.GetCallingAssembly().Location);
 
            if (args[0] == "-i")
                installer.Install(new Hashtable());
            else if (args[0] == "-u")
                installer.Uninstall(null);
        }
    }
    else
    {
        ServiceBase[] ServicesToRun;
        ServicesToRun = new ServiceBase[] 
        { 
            new Service() 
        };
        ServiceBase.Run(ServicesToRun);
    }
}

Next up, how to poll our host system for performance information.

Real-time System Information Voyeurism

PerformanceCounter is the .NETified way of querying the Windows for performance metrics.

PerformanceCounter cpu = new PerformanceCounter("Processor", "% Processor Time", "_Total");
PerformanceCounter memory = new PerformanceCounter("Memory", "Available MBytes");
PerformanceCounter requestsPerSecond = new PerformanceCounter("ASP.NET Applications", "Requests/Sec", "__Total__");

The strings you use to instantiate PerformanceCounters come firmly in the realm of magic. They're not terribly well documented and in all honesty you're better off just opening up Windows' Performance Monitor (WinKey-R 'perfmon') and seeing which counters you can add through that. It'll even give you a convenient description of quite what some of the more esoteric ones mean. Failing that you can enumerate them yourself

It would make a lot of sense to create our performance counters in the constructor, but it turns out they can spend a lot of time sitting around doing nothing which then causes net start to report a timeout. This is not the case if we move the object instantiation to an OnStart event of the service though, so that's what I ended up doing. The last thing on the startup todo list is to get the total amount of physical memory for the host, and for this we take a brief forray into the nightmare realm of WMI.

ManagementObjectSearcher wmi = new ManagementObjectSearcher("select * from Win32_ComputerSystem");
foreach (ManagementObject o in wmi.Get())
    TotalPhysicalMemory += Convert.ToSingle(o["TotalPhysicalMemory"]) / (1024 * 1024);

Now we simply poll the counters every second and store the results in a dictionary.

Status["CpuUsage"] = cpu.NextValue().ToString();
Status["MemoryUsage"] = ((TotalPhysicalMemory - memory.NextValue()) * 100 / TotalPhysicalMemory).ToString();
Status["RequestsPerSecond"] = requestsPerSecond.NextValue().ToString();

All well and good, but now our service has shut-in syndrome, and that frankly will not fly.

Bare Bones HTTP

At this point I had only vague ideas about the form that my client would take. With that in mind I wanted something flexible enough that my later decisions wouldn't be affected by my earlier design choices. With that in mind I decided to go for HTTP as my application protocol, and JSON as my information format.

.NET's HttpListener provides an itty-bitty HTTP server, capable of asynchronous responses and not a lot else. That said, it's perfect for small HTTP-enabled apps like this.

string prefix = String.Format("http://+:{0}/", ConfigurationManager.AppSettings["ServerPort"]);
HttpListener httpListener = new HttpListener();
httpListener.Prefixes.Add(prefix);
httpListener.Start();
 
// 10 worker threads
for (int i = 0; i < 10; i ++)
    httpListener.BeginGetContext(new AsyncCallback(RequestCallback), null);

Besides that, you're on your own. You handle each request as you see fit in your request handler, and once you've done what you need to you spool up another listener to replace the one you've just 'used', then wait for more calls to come rolling in.

private void RequestCallback(IAsyncResult result)
{
    HttpListenerContext context = null;
    try
    {
        context = HttpListener.EndGetContext(result);
 
        string output = "{";
        bool foo = false;
        foreach (string key in Status.Keys)
        {
            output += String.Format("{2}\n\t{0} : '{1}'", key, Status[key], foo ? "," : "");
            foo = true;
        }
        output += "\n}";
 
        Encoding encoding = new UTF8Encoding(false); // Whenever I have UTF8 problems it's BOM's fault
        byte[] outputBytes = encoding.GetBytes(output);
 
        context.Response.OutputStream.Write(outputBytes, 0, outputBytes.Length);
    }
    finally
    {
        if (context != null && context.Response != null)
            context.Response.Close();
        HttpListener.BeginGetContext(new AsyncCallback(RequestCallback), null);
    }
}

Now we can consume the one-trick web service via AJAX requests spawned from an HTML page!

That's Not a Client - It's an HTML Page

Damn right it's an HTML page. Why bother with apps and servers and all that cruft when all you need is a few lines of Javascript? The 'client' is a simple HTML page containing a list of active agents which are polled every second and the results are used as the data for a set of sparklines graphs care of a jQuery plugin. The only gotcha here is the scale on the requests/sec graph. We can treat CPU and memory usage as percentages, but requests/sec is an absolute value so you may need to tweak the y scale accordingly.

Hmmm. Hang on a minute . . .

Oh crap, I forgot about the Same Origin Policy

Every time I think of something cool to do with a web client the Same Origin Policy comes and bites me on the arse. SOP states that an XmlHttpRequest can only be sent to the same domain as the Javascript file which initiates it, and therefore prevents the kind of cross-domain AJAX shenanigans we're attempting here.

JSONP is a cheeky little hack designed to circumvent this very restriction. Rather than being just Javascript object literal notation, as is the case with JSON, JSONP actually returns a javascript source file which is fetched by the host page using a script block, dodging the SOP. When making a call to a JSONP-aware service you provide the name of the function that is ready and waiting to process the response, and the service wraps it's JSON response in a call to that function so that when the file is fetched by the client the response handler is executed. Conceptually it feels weird as you're essentially pulling a pushing, or maybe pushing a pull, but getting down to the brass tacks this means we have to add two lines to our request handler:

string callback = context.Request.QueryString["callback"];
output = String.Format("{0}({1})", callback, output);

On the client side there is a handy jQuery JSONP plugin which makes for a similarly smooth transition.

~fin

Those are the bits that interested me, at least. Feel free to grab the code and have an explore. It's a testament to the .NET framework that it packs so much into so few lines of working code and made for a fun train-journey-home-from-work project over a few days.