iValidator is an open source test automation framework, designed especially for integration tests. Guenter, in this article, motivates why there is a need for a new framework and how it is used.

iValidator is an open source test automation framework, designed especially for integration tests. The article motivates why there is a need for a new framework and how it is used.

We started to build the iValidator framework for integration tests when we were asked to automate the test of a complex application (running a fully automated container terminal) since it tooks the test team several days to test the system manually for each release.

The system was composed of two main components that were developed and released independently, and several other systems that were connected to it. Each component included hundreds of classes and communicated via different interfaces (e. g., RMI, JMS). The customer had about 230 test scenarios in mind, with about 15 steps each. Many steps occurred several times in different scenarios. Some of the systems connected had to be simulated (since they moved containers physically, for example) and these simulators had to be controlled by the automatic tests since they receive and trigger events. Even to build a valid starting point for a scenario was a difficult task that included the initialization of several systems. In the end the test scenarios and test data were defined in several XML files (consisting of 30000 lines). They produced a report in an XML file on each run that took about 10 hours.

Test process and test scenarios: An example

The test process starts in the analysis phase of the project with the use cases of the system under development - becoming the system under test (SUT) soon. From these use cases the test scenarios for component integration tests are derived. Each test scenario shows that a business requirement is fulfilled.

Let's consider a simple order management system. It includes these components:

• Customer Management

• Order Management

Each component consists of several classes that store the data and enable the services of the component, e. g., creating a new customer, finding a customer, changing the attributes of a customer, selecting customers that match a search expression.

The components are developed by different teams. Each class of the different components is thoroughly tested by JUnit tests. Even the components are tested by JUnit. But who is responsible for the integration test? A test team that should test manually and cannot cope with the flow of major releases, minor releases, bug-fix releases and quick-fixes delivered by the development team. So they test only the parts changed. Let´s see what happens.

Some poor guy John sits in the call center using our system, finding that the main use case he requests is not fulfilled:

• A new customer calls, John creates the customer using the customer management. Great, it worked! Some minor problem: The address of the customer could not be checked since the post code checking service is down (the SOAP services not working reliable). But no problem: The status of the customer address simply remains ´unchecked´ until the service is up again and will be checked then automatically (smart system, uh!).

• John enters the orders of the customer. No problem either. But then John has to tell where to deliver the goods to - hoops, ´unchecked´ addresses cannot be selected (this did work until some guy in controlling found out that 5% of the deliveries were returned due to wrong addresses and requested the change of the customer component - which was tested, sure). We are stuck! Saving the order is not possible. The customer is lost.

Each component works for itself fulfilling its requirements - but the process that uses both components does not work any more since one component was changed and nobody checked the overall process - or he checked the process and it worked - but he did not consider the failure of the post code checking service since this failure is difficult to simulate.

Figure 1. Context

We will find many variations of this scenario that do differ slightly. e. g., the good one - all works fine; the address is not correct; we select a customer instead of creating him; the customer did order yesterday so we may combine the deliveries; the good ordered is not in stock...

For each release of the system each of these scenarios has to be checked - by some poor guy or by automatic tests. Lets choose the automatic tests and let the poor guy derive the test scenarios from the use cases and automate them.

Requirements for an integration test framework

What are our requirements for an automatic integration test framework?

• Automation: The tests should run automatically (obviously) - we will check the results the next morning.

• Fail-safe: A failing test scenario should not abort the whole process - we want to see how many scenarios still work.

• Error-tolerant: Each error has to be reported, but it may be feasible not to abort the whole scenario, e. g., solely because the age of the customer is not correctly calculated from his birthday - since other test steps may not refer to this data.

• Flexible: A new test scenario that it is only a variation of existing scenarios or uses another set of test data may be added without programming - our test team does not like to wake up the developers for each new scenario they are inventing.

• Configurable: A test scenario should be runnable easily on itself - so that we may show our developers the errors with full logging and debugging switched on if they do not believe that their system fails.

• Adaptable: Technical changes should not require our whole testing system to be re-implemented - so when our lead architect decides that the components do not implement CORBA interfaces any more but deliver SOAP web services we will run our test scenarios against them the next day without complaining.

• Extensible: New components or simulators may be integrated easily - with our evolving test scenarios trying to cover each hole left in the use cases.

• Integration: The framework should integrate into different environments, i. e., the test scenarios and test data may be defined in different ways, e. g., in XML files or in a database.

Building blocks and interfaces of the test framework

Simply spoken, iValidator implements a test runner like JUnit. But since integration testing is a little bit more complex than class testing, it defines some more building blocks and interfaces.

The building blocks of test scenarios are:

• Setups and Teardowns: In contrast to the simple setUp and tearDown methods in JUnit we have classes that implement setups and teardowns since we want to reuse in different test scenarios, maybe even parameterizing them.

• Test steps: Each test step realizes a reusable business task, e. g., creating a customer. Since this step may be part of different test scenarios it has to be parameterized.

• Checkups: A checkup checks the results of a test or several test steps for the expected outcome. If it finds an error it may decide how severe the failure is - and in the test scenario description it has to be decided if the scenario has to be stopped or may continue on this error.

• Flows: A flow combines several Setups, test steps, checkups and/or tear downs to create a reusable higher abstraction. Flows may be nested. A test scenario in therefore a flow - maybe consisting of several sub-flows.

• Adapters: An adapter abstracts from the technical and business interface of the SUT or a simulator. It is used to call the SUT or a simulator from the setups, test steps, checkups, and teardowns. When the technical realization of the interface changes, e. g., from CORBA to SOAP, only the adapter has to be changed - the test scenarios using the adapters are left unchanged. If the business interface changes, e. g., taking one more parameter, the adapter may provide a default parameter while another adapter exposes the new parameter for usage in new scenarios.

• Test parameters and test data: Test parameters and test data are fed into the scenario and flow through the scenario, being changed or enhanced (consider generated technical ids, for example) by the setups and test steps. The teardowns may use them to clean up.

The interfaces of iValidator are used to integrate the framework into different environments:

• Repository: The repository contains the definition of test scenarios, test parameters and test data. The repository may be, e. g., a set of XML files, some database tables. iValidator simply requires the environment to supply these resources. iValidator contains an XML reader as standard adapter for the adapter.

• Reporting: The result of the test run is reported via the reporting interface. The standard implementation creates XML files but other reporting sinks as, e. g., a database, are supported.

• Logging: iValidator may use different logging systems, e. g., log4j or Java 1.4 logging, to log its execution. This logging is separate from the SUT logging.

Developing tests with iValidator

The major design principle behind iValidator was to enable reuse as much as possible and such reduce costs for testing (btw cost is the main reason for not testing). The main reusable components are:

• units

• test data

• adapters

Units -divide test cases in small, reusable parts. The developer has to write small parts that could be used in different scenarios. Therefore the unit's parameters are separated from the unit´s implementation. The parameters are provided to the unit on test execution. The assembling of units to test cases is done declaratively using XML meta data. (This was true in version 1, now with iValidator version 2 XML is the default repository but it can be replaced by, e. g., a database.)

Most integration testing needs a lot of test data as input for the SUT and for control purposes. Data can be defined and grouped within the test description once and used in different test scenarios. For instance, you can reference a whole bunch of data and pull it into the scenario at run time.

Figure 2. Plugin

sum:

0 €

» Library of the Month
» Workshop
» Software Engineering
» Programming Tools
» Interview
» High quality and
secure programming
» Reverse Engineering
» Portable GUI
» Artificial intelligence