Black Box to avoid software crashes

Imagine if you will the following situation. A customer of yours phones to report that his system has failed in a mysterious way. You are given a good error report with details of the transactions. So you try to replicate the problem. Nothing happens. You even go to the extent of making a site visit and you chat to an intelligent operator who tells you which keys were pressed, the mouse actions and even the recent history; all to no avail. I have often heard engineers say, “it is just one of those things.” Perhaps they mean that the problem does not really exist. However, there is a simple dictum and it should be emblazoned in ten foot letters by every help desk. “If a customer reports a problem then a problem exists but it is probably not the reported problem.”

The trouble is that operators just cannot remember what they have done. This is nothing against operators. Any programmer knows the embarrassment of chasing down a bug in initial testing only to discover that it was totally down to finger trouble. Let’s get back to our scenario. We know there is a problem but we do not know how it was caused and we only have a rough idea of the circumstances that caused it. This is where Identify can help.

Identify’s AppSight Black Box does what it says. It works just like a black box flight recorder in an aircraft. The information that can be stored is configurable but it can include external events such as keyboard presses and mouse movements, as well as the synchronised source code statements that are being executed. You can configure internal events within Black Box that can direct the recording

The recording uses the endless loop technique. You set the time you want recorded, say 4 hours, and earlier records are discarded. Consequently, although you have a large log file it remains manageable. When you have an error, you will have the last 4 hours recorded in exquisite detail. If that is not enough then you can always increase the time loop but generally, the last 10 minutes contain enough information to solve the problem—even if the real fault occurred yesterday.

I was lucky enough to see Black Box in action. Admittedly, it was an artificial situation because the operator knew what the problem was and what sequence of keys was needed to cause the fault. I, however, did not. On examining the log, the immediate cause of the error was instantly apparent. It then took me about two minutes to deduce the real cause of the problem. In real life, on discovering the cause I would have to start laying out hypotheses as to what the operator did to cause the error. With Black Box I think it would have taken about an hour to complete the process and test them.

Without Black Box, I think I would have been in trouble. Because I now know the problem, I can work out how my real life solution would have gone. It would have been about a day to hit on the immediate error and another day to test hypotheses to find the underlying cause. Sixteen hours hit and miss reduced to one hour straightforward procedure. I like that ratio.

Identify only claim a 60% reduction in solution time. So in the above example my one hour with Black Box should have taken 3 hours without. I have been reviewing recent software problems that have engaged me. Black Box would not have helped me with some problems, but they were the easy ones. When I consider the horrors, then we have a completely different scenario. I think that Black Box would have cut down my solve time by nearer 10 to 1.

AppSight can be used for .NET as well as J2EE applications. The logging module can be added to deployed applications in the field without having to add software instrumentation to them. I cannot think of a more useful tool for solving software errors.