[Home] [Essays] [Book_Reviews] [Tools] [Links] [News] [Blog] [About]

Page 3

2     Graphs in Practise. 5

2.1      In the beginning there were design techniques. 5

2.2      Graphs as a test case design mechanism.. 6

2.3      Test Script Development from a Graph. 6

2.4      The Script Meta Model (An Aside)8

2.5      Communication Graphs leading to test development9

2.6      Client based exploratory testing. 10

2.7      An Exploratory Testing aid, and Beta Testing deliverable. 11

2         Graphs in Practise

2.1      In the beginning there were design techniques

I started in this business as a software developer and a software designer and I learned many visual modelling techniques: FlowCharts, Jackson Structured Programming (JSP), ERD, Petri nets, State Transition diagrams, Use Cases, and the list goes on...

The structural models; Flowcharts, JSP were interesting as they abstracted the underlying implementation source code, and, given a certain level of detail in the model, the code could be automatically generated from the model itself. In order to generate code, the models have to use the concepts of sequence, selection and iteration.

Flowcharts seemed to be a particularly poor way of representing source code because, as a developer, it was faster and less error prone for me to write the code, than to write the code out as a diagram. JSP was more appealing. JSP added a layer of abstraction about the code, and I had to build models of the data as well as structural models of processes.

Most people are familiar with flowcharts so I won’t explain those, but many people skipped JSP and went straight into OO. So I’ll give a quick overview of JSP here.

JSP has 3 notational concepts:

JSP used the notion of analysing the input data and diagramming it into the above form, analysing the output in a separate diagram, also using the above notation, and then merging the diagrams into a final program. That last sentence actually generalises quite wildly, but JSP was really quite interesting for me and opened up the possibility of programming via diagrams rather than code. And for a while, in an academic environment, I generated all my program code, but in the real world on development projects I quickly stopped generating in favour of a design model, with a separate code model.

When I didn’t have to have a one to one mapping between the design model and the code model, I had a vast array of choices for design models, the UML is full of them, and I used state transition diagrams, Entity Relational diagrams, Entity Life History diagrams. The lot.

When I started testing other people’s developments, clearly I couldn’t use the code, but I was allowed to use the design models. All testers learn techniques for testing design models, none of which I will cover here, but most relate to test condition derivation, test data derivation or test case design. This was fine and I could generate dozens of test cases, but my problem was the number of test scripts I had to write, all of them similar to one another.

I envied the programmers who had the concepts of iteration and selection because it didn’t seem allowable to add these constructs into test scripts. If I gave testers a choice in the script then they might choose the wrong path when re-running the test. The scripts were all written in English and became confusing when I used too many procedural flow constructs. Instead, I embraced the workload of writing hundreds of scripts using just the sequence construct.

2.2      Graphs as a test case design mechanism

I remember when I first started using graphs on test projects. I was still a young tester and I had just started to construct my models of the testing process and the relationships between the various entities in the testing life cycle.

I was aware of the concept of requirement coverage and had skimmed, but hadn’t fully understood, Beizer’s coverage of graphs in Software Testing Techniques[Beizer90]. Something from Beizer obviously stuck in my mind though, as I started to use graphs as a test design technique.

By diagramming a process, or system, or set of requirements as a hand drawn graph, I could use a path coverage technique to construct test cases. By path coverage I mean that I would trace paths through the graph to derive tests, and I would do this until I had covered all the edges and covered all the paths that I thought were important.

I did this in secret though as this wasn’t something that the other testers in the team were doing. I was dabbling in the black arts, the occult realms of software testing and the graphs became undisclosed glyphs and magic talismans in my notebook. I justified each of my graph designed tests by cross referencing them back to the test conditions which we had generated from the requirements and specifications.

An Early Example of a Graph Based Testing Grimoire

But I was selective where I applied the graphs. I didn’t want to model everything in the system as a graph, some of the tests were obvious and many were one offs, but sometimes I needed a batch of tests for an area and in those instances, graphs were my tool of choice.

The technique was a secret, and it might have been magic. However the results were tangible and visible. By using graphs, I understood the requirements better, could create test cases faster and, where my path didn’t match a test condition, I was able to evidence a higher degree of insight into the requirements and spot a gap in our analysis.

Thinking back to my design days, I knew that these graphs evidenced selection and iteration constructs and if I could only figure out how, then I could start generating scripts from these models too.

2.3      Test Script Development from a Graph

As a tester, the majority of the projects I have worked on have been manual structured testing projects where most of the test cases and test scripts are produced far in advance of test execution. The most frustrating aspect of constructing testware in structured testing projects, for me, is the construction of so many test scripts.

Test scripts are programs which testers execute. But unlike the programs that programmers write, we don’t write one program with loops and conditional statements, we write hundreds of programs using just the sequence construct. We write a thousand scripts to run a thousand paths.

The test script model, encourages this situation

There is, it has to be said, a certain degree of redundant representation in the test script model. Too many of the scripts are too similar, as are too many of the test case descriptions. And yet this is the model which is supported by most of the test management tools in use today.

I distinctly remember the moment when the test script model and the tool support for it started to hamper my testing abilities. I had a rather small system to test, but it had a high degree of complexity and there were a variety of developer produced flowcharts which had been produced to help explain the system. The flowcharts were out of date and I had to remodel them. I have already mentioned my distaste for flowcharts so I turned instead to my new found ally, the graph.

The graphs I began producing for this system were very detailed. It was the perfect system to model using graph notation, but given the tools in use on the job and the culture I was working in, a different level of abstraction or a different representation in the test scripts was not acceptable. However, I was running out of time, and I had no choice, I had to reveal my secret method of test construction and this time I had to use it to construct scripts.

I created a fairly detailed set of graphs. For each of the nodes and links in the graph I created a textual description which was a process description (script) of what to do at each node or edge when encountered during path traversal.

Test data was analysed and described and partitioned into test data sets. This was an easy way of partitioning the test data and combining it with a path so the text description on a node could read “Enter the customer ID”, and I could look at the test data set and see what the customer ID was for that test. The test scripts were then documented in terms of the Path and the Data.

In order not to make the graph too complicated, I still handcrafted some of the scripts that were too tricky to model or would add undue complexity to the graphs.

The graphs were drawn in Visio and took quite a long time to layout and get right.

I never seemed to find the time to extend this approach further and model it all in an access database, as this would have allowed me to automatically generate the script that I was running. Instead, when executing a script, I had to read the various descriptions and data set information from MS Word documents. This took a little longer when executing the script as I had to pull together various bits of information, but took far less time than actually writing the scripts.

I did experiment with automatically generating the paths from a graph and wrote a small Perl script to help do this. The Perl script can be found on the web [wwwPP] and there are various pages describing its limitations. Automatic path generation is a topic of research on its own and will not be covered in this paper, suffice it to say that:

-          the Perl script is rather naive,

-          generates too many paths,

-          and because I was new to graphs I made the mistake of representing paths as sequences of nodes instead of sequences of links – in my defence, my graphs were only drawn with a single edge between each node (it seemed right at the time), but a lesson learned. Paths are sequences of edges, not sequences of nodes.

2.4      The Script Meta Model (An Aside)

Throughout my various roles on client sites, I have been making notes towards a model of the testing process. A sub part of a larger testing entity model is the Test Script Model which has already been covered.

A model which I use to provide a different view of the testing process is the Script Meta Model.

The Script Meta Model was constructed out of a desire to avoid writing the many deliverables that testers write when using the Test Script Model; reams of test conditions, numerous test cases, hundreds of test scripts. One of the reasons for using a graph model is that a graph can be converted into a number of scripts, therefore instead of maintaining a set of scripts why not just maintain the graph that they can be derived from?

I started to view the testware that was being produced as the result of 3 different models:

Test Cases seemed to be the result of the intersections between these models and Test Scripts were intersections between these models where the structural model was involved. I started to term this the Script Meta Model:

 The Script Meta Model Test Case Definition

Different intersection points map to different levels of Test abstraction. High level test cases map on to a requirement model and data model intersection. Completely instantiated test scripts map on to an intersection between the structural model and the data model.

The utopian ideal behind this model is that if the tester’s structural models were complete and accurate enough then all test scripts could be generated automatically. Personally I don’t have time to build models like that, nor do I have the tools, so I use the Script Meta Model to remind me that I should use my models to reduce the amount of testware that I have to construct.

2.5      Communication Graphs leading to test development

The project was being conducted in a structured testing and development environment, but it was not going well. The development specifications were ambiguous, being produced late, delivered late to the test team, and they were wrong.

Writing test scripts was taking too long and the constant change resulted in too much rework.

As a mitigating strategy I produced a number of graphs which I could use to communicate my understanding to the development team. Normally I didn’t use my graphs in this way, but it seemed fairly sensible that if I was using them to understand the system then I could use the graphs to communicate my understanding of the system.

The graphs went through quite a few iterations until they were understood by the development team and everyone was satisfied that they represented what the testers could do to the system. The development team also made a few changes to the system as a result of reviewing the graphs as they identified issues with the system or specification. And where the graphs were not complete, we supplemented them with some high level test case descriptions.

An unexpected benefit from all this communication was that the development team reported it was the first time they actually had proper visibility into the test process. They had always been too busy to read the test documentation before or plow through dozens of test scripts to see what the testers would be doing.

As the project was running to tight timescales, and we still had to plan the testing, the scripts were documented at a high level of abstraction; with an overall aim, the path through the graph to be taken, the test data to be used, and the various test conditions that would be covered.

A high level communication graph suitable for test case derivation and scripting

Because I used graphs early in the process, the development team were used to seeing them and I could use some of the higher level graphs showing system interaction to communicate the progress of testing and what areas were still to be tested. This is something that I will provide more information on in the later section Experiments in visualisation

The approach used was fast, documented, structured, reviewed, had buy in, meant less rework to the testware, and the testing was still repeatable.

2.6      Client based exploratory testing

My previous examples have all been related to structured testing, and I’ll be summarising the lessons learned later on, but graphs are not just applicable to structured testing.

My first experiences with exploratory testing were before I knew what the term was. I was asked to test a system on a client site with no notice, as I was filling in for a tester on the team who was on holiday, I didn’t even know what the system did. Fortunately the client did give me a single page release note before seating me in front of the web browser. I told them I  was only going to be able to do adhoc testing, they said that was fine as they had already tested the software.

Fortunately I had been mulling over in my mind the possibility of constructing graphs at the same time as I would be learning the software, to try and minimise the rework that is involved when a graph built at design time doesn’t quite match the delivered system.

Rework is always an issue when you construct your test designs before the system is available, but I found the impact of rework when using graphs was far less than the amount of rework that had to be done when I was using the test script model to document my tests.

To aid me, I had found a graph drawing tool on the internet which I was going to explore. The tool was interesting in that it was free; great, our test budget didn’t extend to cover my experiments and it was a tool designed to draw graphs, not diagrams.

A generic diagramming tool, like Visio, can be used to draw graphs but its very generic nature makes it slower to use for a specific purpose. A tool which only draws graphs is faster to use, and has use features like automatic graph layout, and a text based representation of the graph which can be parsed later if required.

The tool I had identified was VGJ which is covered in more detail in Appendix 4: Tools List (VGJ). VGJ is a simple little browser based java applet. But because I was testing a web based app, when the app brought down my browser, it also brought down my graph. Fortunately as I was merrily testing away, I was saving my diagram frequently as I generally do with any tool.

VGJ in action

This was an interesting exercise to do, but I found that the user interface of the tool didn’t support the speed of diagram creation that I required for exploratory testing. It was still faster than Visio, but VGJ didn’t quite have the speed I required, and I was still making written notes as I went along. I found it easier to draw the graph by hand and to represent the graph using the GML structured notation in a text editor and paste it into VGJ for visualisation.

Inevitably the development team were only interested in the defects I found and fortunately the holidaying tester came back so I didn’t have to repeat the process or document my notes any more thoroughly.

2.7      An Exploratory Testing aid, and Beta Testing deliverable

My initial experiences with graph based exploratory testing did not put me off, and I did eventually learn a little more about exploratory testing and I have been practising, by Beta Testing other people’s software. Beta Testing is a great way to improve your testing skills as you get to experiment with whatever test technique or approach takes your fancy at the time.

One of the important aspects of exploratory testing, I have since learned, is knowing what you have done and being able to report on it. I have tried a number of ways of doing this; an open text editor file, mind maps, diagrammers, outliners and graphs.

I have settled on text editor notes and hand drawn graphs on an A3 sheet of paper. This is not far different from my initial attempts at graph based testing but my final documentation process is now different.

My Beta Test Mapping Process is simple:

1. open the app
2. scan the available options, they become links on the graph with no terminating node
3. follow the links, drawing the graph, testing and making notes as I go
4. repeat steps 2 & 3 until the urge to test is sated

When I need to start again on the next session, I already have a map so I start filling in the areas that I haven’t yet visited.

At the end of a particular test session I document the graph really quickly, either using a front end to an automatic graph visualiser, or representing the graph directly in the graph visualisation tools graph markup language. By using a graph visualiser I don’t have to worry about the layout of the graph, as I would do if I was using Visio or any other diagramming package.

The graph visualiser that I use is called Dot and is part of the AT&T GraphViz package. I use a variety of front-ends to Dot: sometimes I use the supplied Dotty editor, sometimes Tintfu, occasionally JGraph but now I normally I use Compendium-TA which is a program that I wrote to help me do graph based testing. Details of all these tools can be found in Appendix 4: Tools List.

The Dot markup language is a structured text language file which is very simple to construct and amend, details can be found inAppendix 5: Graph Languages.

I then send the visualised graph, with the supporting text file notes and issue logs to the product supplier. I have found that this level of detail is more useful to the product supplier than just the issue logs as they get a feel for what tests have been run; can ask me to test certain areas in more depth and can replicate the issues better as they see the context in which the issues arose. The developer can also point out paths on the graph that I missed or that they would like tested in more detail.

Page 3