It could also be less sinister, maybe unit testing was performed, but the fix had other unintended impacts, or the unit test was not creative enough to produce errors, or worse still, the defect goes unnoticed until it is released into the wild.
Unfortunately it happens, but can it be prevented?
Imagine your Siebel developers could run a suite of automated test cases, to verify the sanity of the core application just before check in. This would provide your project with a good measure of quality assurance that the application isn't broken. How good the measure is, depends on your test coverage.
Testing can be exhaustive, and take hours to regression test the Application. To make things more manageable, we can split it into two categories.
1. Core application
2. Business functionality
2, can be covered by more comprehensive regression test methods, and are done post check in. Sometimes this is too late. The concept of ‘The Test’ is to focus on 1. Testing the core application, to verify that what’s being checked in, doesn't cause widespread damage to the application in the next migration.
In the past, I've covered this topic in the following articles
SiebUnit - an xUnit implementation
http://www.impossiblesiebel.com/2012/02/siebunit-xunit-implementation.html
SiebUnit - Build Inventory
http://www.impossiblesiebel.com/2012/02/siebunit-build-inventory.html
I've had a lot contact from various Siebel customers over the above articles, confirming that the tool meets a real world need.
“The Test” is not as ambitious, its purpose is to be simple, effective and a lot easier to implement, but it’s not declarative. Writing code to test code, and maintain that extra code, isn't a deal breaker, what’s more important is that your developers have the confidence to make core changes, and your application is protected from serious defects.
Like SiebUnit, the main requirement of ‘The Test’ is to have testable interfaces. Testable interfaces are API’s that can accept an input, produce an output, without any reliance on Siebel’s application context, and are predictable. Ideally it should also have as few dependencies as possible.
In Siebel, these API’s take the following forms.
Business Service
Workflow
Script Library
eScript
Browser/OUI Script
To test the above interfaces, I propose we create a Test library using eScript (for testing server side APIs), it could sit inside your existing Script library if you have one.
There should be an external method called “The Test”, this method could instantiate a new Test class, with a main entry point called “run”, that would run groups of test cases, each of these groups would contain 1 or more test cases. To facilitate the writing of clean test cases, I suggest we enforce a formal structure.
The following code snippet shows an implementation of the above concept in Siebel.
1. Calls a script library function called parseINI that will be tested. The util namespace is used, and invokes parseINI, to return object representation of a configuration file
2. assert is defined as a class function.
3. The purpose of ‘assert ‘, is to determine if a test case has passed or not. The first argument passes the evaluated the condition to assert.
4. The Output PropertySet is passed to assert for logging the result
5. A description is provided to identify the test case.
6. Test case helpers to minimize coding.
Examples of helpers
* Create a common complex hierarchy for PS manipulation
* Invoke Siebel APIs without using setting up PropertySets
* Invoke WF without the normal setup
* Create a data randomizer to facilitate more creative test cases.
The good news is that with a bit of organization, we can reduce the amount code that a developer has to write to test your APIs. I claim that these test cases are readable, are easily maintained, and require no effort to pick up, other than the inertia to change ones habits.
Now that we have a proposed test case construct, we need to have an ability to report on the test results. Step 4 states this from a high level, but the report can be as simple as name value pairs on an Output PropertySet, or as complex as you need. The report should at the very least, display the number test cases that have passed and failed. This will allow the developer identify where the problem is, fix the issue, and re run The Test again to ensure that all expected test cases pass.
Establishing your Test library a great first step, but acquiring it alone is not is actually not enough, the project has to foster the culture of writing test cases into developers, and factor the effort into the build estimates. The project also has to identify what is testable, and augment what is not testable. A script library is a good example of a testable interface, members of the audience who have the ABS framework on their projects, should be familiar with "The Test".
It takes effort to write the test cases, as well as to maintain them going forward. To be conservative, your current unit testing estimates should be doubled when implementing automated test cases. This may sound a little daunting, so lets take a look a little example that demonstrates the ROI.
Testing an email address validator
You have a written a library function to test the validity of emails. The manual method of test this might be to launch the UI, and navigate to a view, where you repeated enter a series of bad, and good emails, to test the string. Lets say that takes you 5 minutes to perform, and writing automated test cases for the same scenario takes 10 minutes. Now imagine a defect was raised against your code, you would need to fix the defect, and spend another five minutes to re-validate the function to ensure you haven't broken something else. On the other hand, your automated test cases can be re run in an instant.
The initial effort to write the test case, will be paid back in spades. Your application will require less re-development, and suffer less downtime as a result, your developers will have the confidence to make core changes, automated unit tests can be your unit test documentation, your application will be more robust, and my personal favorite, is that developers can write the unit test once, and run it multiple times.
Once you have the right framework in place, and given enough experience (1-2 weeks), writing test cases actually saves more time than not writing them.
The Test is simple to build, and can be taken as a quick win towards a larger goal, for ensuring a better quality product. However be warned that once you start writing automated test cases, you won't want to look back to manual testing.
0 comments:
Post a Comment
Comments are open to all, please make it constructive.