Combining the Use Cases and the Test Data Sets results in a process known as a Test Plan. A sample Test Plan will take a single Use Case and run it through multiple data scenarios, fairly representing the types of situations the user will encounter or produce.
Regression testing is the term used to describe the need for retesting something in order to make sure it still works after having made modifications. Note that these modifcations aren’t necessarily in the module being tested – most systems are sufficinetly intertwined that pressure in one spot can have an effect in another.
Having a clean set of data, use cases and testing scenarios enables you to retest and ensure that the same outputs are being produced.
Who tests what?
I’ve often said that the worst person to test an application is the developer’s mother, since she has an interest in never finding anything wrong with you. The second worst person to test is the developer himself or herself. The reason is that the develop, also, wants to prove that the system works, and will, subconsciously, attempt to prove only that the system performs the intended functions, not that it doesn’t break.
These issues bode well for having another person do the testing.
There are two potential sources of test personnel – in your company, or in the company of the user. And there are advantages and disadvantages to both.
If your firm does the testing, the following advantages accrue:
The following disadvantages are associated with your firm doing the testing:
The following advantages accrue when the customer firm does the testing:
The following disadvantages accrue when the customer firm does the testing:
Once someone other than the original developer sees the application, that person is going to find bugs. More accurately, that person is going to provide feedback. This feedback may be a bug, but it may also be a question, or an enhancement request. And it is possible that the user making the report can’t necessarily make the distinction.
Ordinarily, a developer fixes the bugs, answers the questions, and offers to make the changes requested. And that’s that. We decided that we need to track this feedback - and came face to face with four issues. First is - when do you start tracking application feedback? Second, how do you categorize it? Third, what mechanism to you use? And fourth - what information do you track about each feedback incident?
But first, let’s talk about that initial assumption - that you want to formally track this feedback at all. Isn’t it a lot easier to just deal with each communication from a user and be done with it? We’ve found two distinct benefits to formally tracking each application feedback incident.
First, we can provide better customer service. Feedback that the customer provides is tracked and thus, these reports are less likely to fall through the cracks. Also, one of the steps in our tracking process is to provide resolution to the customer.
Second, we categorize the bugs. One of the attributes we track for each bug is the type of bug and where it came from. From this, we can determine where our weak spots are, and, thus, determine where we need to improve our development processes. The bug reports are a tool from which the developer can learn.
Third, „feedback“ might not be a bug. It may simply be a question, or it may be a request for a modification in disguise.
We start tracking our bugs as soon as they go into internal testing. The reasons for this are twofold: The first is that we want to fix it, right? If we don’t document each bug that our QA department finds, it’s going to be really hard to keep track of them, and if we don’t keep track of them, some will get lost - and they won’t get fixed.
The second is that we want to track this defect and find out why it occurred. We want to find out who is putting bugs in the code, and why. Once we find this out, we can not only fix the bug, but also take corrective steps to fix the source of the problem. We can provide better training, better coding techniques to make a specific mechanism less fragile, or even learn to avoid certain types of interfaces, process or techniques that prove to be more fraught with peril than other types.
We’re really critical about what is a bug and what isn’t. Most people think that there are two kinds of bugs: those that the customer finds and those that they don’t. We think about bugs a bit differently. We categorize bugs when we enter them into our bug tracking system. Here is how we categorize them:
Analysis.
This means that we made a mistake in the analysis phase of the project. Examples
of an analysis bug would be the situations where we misunderstood the way a
customer process worked, or how they recorded data.
I’ve mentioned earlier about the application where we understood the term „part“ to mean a component of the product that the customer was shipping, while they used the term to refer to a component of a machine that was used to make those products. We spent several weeks of iteration before realizing this - if we hadn’t realized it at all, and had implemented a system based on our poor understanding of the term, we would have recorded it as a bug in analysis. True, it’s pretty hard to document something like this, but we need to have a mechanism in place. It may not be that important to track when I’m the only one doing the analysis, because I never make mistakes, but once we have 38 analysts at the shop, we’ll want to track these bugs just like any other.
Design.
Two obvious examples of design bugs would be mistakes in data structure design
and screw-ups in designing forms. A data structure design bug would be the incorrect
normalization of a set of tables, the use of an overloaded table when it was
inappropriate, or the use of data attributes as keys when the instances of that
data weren’t going to be unique. (Of course, if we didn’t realize that the data
wasn’t going to be unique, that might be an analysis bug.)
An example of a bug in form design would be the creation of a form that required the user to hit four keys to save a record, and then three more keys to start the Add process for a new record - when the form in question is specifically a heads-down data entry tool. Again, if we heard this in analysis and didn’t design the form to be able to do so, then it’s a design bug. If we didn’t catch on to the customer’s requirements, then it would have been an analysis bug. If a customer calls and asks you the same question four times, it could be a design bug. If they have to ask you incessantly, then there was probably a better way to have designed the interface or process that the user is asking about. But it could also be an end-user training bug - that we need to do a better job explaining something to a user.
Coding.
The third type of bug is a coding bug. We’re all familiar with these. Syntax
errors, mistakes in algorithms, incorrect usage of commands; a coding bug is
defined as „We knew what to do (analysis and design were correct) but didn’t
do it right.“
Environmental.
We use this category to flag a situation when the application runs fine on our
system, and runs fine on their system, but when they add another machine to
the network the app comes crashing down. Turns out they didn’t configure memory
properly, or the network card is conflicting with something, or they installed
an old set of DLLs, or whatever. In any case, the bug is resolved either by
changes to the environment, or by adding features to the code that make checks
for the environment problem at hand. The key is that we didn’t have to change
our code.
Installation.
Installation issues are the fifth type of bug we track. These obviously only
happen at one point during the cycle, but it’s important to note them. This
bug is flagged whenever we have an application that runs fine at our location
but we fail to get it running at the customer’s site. This could range from
technical issues like bad drivers, lack of disk space, bad floppies to human
problems like forgetting the diskettes or overwriting files by mistake.
Training.
This next type of bug is a bit of a gray area, but we’re paying attention to
it nonetheless. If a customer has to ask a question about the application that
they should have known after we were finished installing the system, then we
mark it down as a training bug. For example, a question like „Why don’t I see
all of the transactions in the list box?“ indicates that we didn’t do a good
enough job documenting or explaining how the list box is populated. The reason
that it’s a gray area is that we might have documented it and felt that we did
a good enough job, but sometimes users need a bit more clarification than we
need.
The important issue here is to make sure that we’re taking care of the customer’s learning curve - the best application in the world isn’t going to do any good if they don’t know how to use it.
Data.
The next category is our most favorite and least favorite at the same time.
We lump these issues under the heading „Data.“ Suppose the user imports a file
and suddenly every list box in the system has garbage in it. They turn the system
on and are getting intermittently screwy results and a lot of processing errors.
The records in one table are missing most of their children. In each case, the
error is not fixed by changing code, because there are problems associated with
the actual data in the tables.
What happened? The import file was not in the correct format. The user went into the file manually and deleted all the fields with surrogate keys. A helpful administrator from another department restored a backup and overwrote the lookup table with an old lookup table without the most current values.
On the one hand, „it’s not our fault“ but on the other, these are issues that we should consider when enhancing the system and making it ‘idiot proof.’
Irreproducible.
Finally, the most famous category of them all: irreproducible. It’s just one
of those things. This could be traced to a flaky network card, an errant video
driver, or just plain magic. I know of one installation where the notebook of
a certain executive would flake out and just die every once in a while. It turned
out that his office backed onto the freight elevator, and every time he had
his modem on and the elevator went by, some sort of interference crashed the
system.
When someone encounters a bug, we require the user to fill out an Application Feedback Incident form.
The reason for the long name is that oftentimes we get a form from a customer (or from an employee who is using one of our internal systems) and they are not sure what they are reporting. They think it’s a bug when in actuality it’s simply a question about a capability they are unsure of. Perhaps it’s really a request for a new feature or a twist on some existing functionality. They will phrase the request in terms of a bug („The Framboozle report doesn’t break out the prior year and current year values“) but when we review the „bug“ report with them, we can show them that the Framboozle report was never designed to break out the yearly values like they want. Of course, we’d be happy to write up a Change Order and modify the report so that it does so!
We track each reported AFI in a table. Our table, however, is not just growing longer, it’s also growing wider. In other words, as we produce more bugs, we learn more about the types of information we should track, and keep adding fields to the bug table.
Our bug table started out simply - the name of the customer and the system, the date the bug was found, a description of the bug, and when it was fixed. When I was one man shop, this worked out pretty well. But then I added more people, our systems got bigger, and we were more rigorous about tracking every feedback incident received.
We now also track who reported the bug (and this could be either an internal person or a customer), the specific module and task (screen, report or process) that the bug occurred in (if we can pinpoint it to that extent), and a number of dates and the initials of the person responsible for each of those events. For example, we track who investigated the bug report and when they did it, when we tried to fix the bug and who did the fixing, when we tested the fix and who did the testing, and when we agreed that the bug was taken care of. We also track when we got back to the customer about the bug - specifically informing them that the bug has been taken care of.
One note about attaching a bug to a specific screen or process - it’s not always possible. The bug may be system wide. It might not be just a single combo box that’s not working, but rather, when a toolbar button is pressed when opening a form that causes everything to come crashing down. Where is the bug? Which module or task do we assign the bug to? We don’t know yet, so we just assign it to the project.
We also use our bug database to track enhancement requests and questions. Those will probably not be attached to a single task, they would just be attached to the project as well.
Upon delivery of a module, we request the user to fill out an Acceptance Form that documents that we delivered what we said we did, and that the user agrees that we did so. Just as when you have something delivered, the delivery person has you fill out a form accepting shipment, this Acceptance Form performs the same function.
However, while it’s fairly easy to verify that the guy with the lift truck did indeed deliver 12 boxes of paper, it’s not as straightforward to determine if a set of delivered modules is acceptable. For this reason, we go through several steps. First, we have given the user a chance to beta test the module so that they can verify that the module performs as generally expected.
Second, we walk through the module with them upon delivery, again confirming specific functionality. And, finally, we allow them to spend a fixed amount of time after delivery to ensure that the system does perform as expected.
One caveat, however, is that we require them to accept without examination after a given period of time, so that they can’t drag acceptance out for an indeterminate period of time. We don’t get paid until acceptance, and so we don’t want to get into a situation where we’ve delivered something and the customer doesn’t accept it for six months.