Showing posts with label Testing. Show all posts
Showing posts with label Testing. Show all posts

The Test

Every once in a while, a developer decides to make a fix without unit testing and breaks the application. There are a variety of reasons why it occurs, it could be that the developer got too confident, maybe it hard to setup the test case, it could be due to inexperience, or it could be due to pressure to turn around the defect, and check it in.

It could also be less sinister, maybe unit testing was performed, but the fix had other unintended impacts, or the unit test was not creative enough to produce errors, or worse still, the defect goes unnoticed until it is released into the wild.

Unfortunately it happens, but can it be prevented?

Imagine your Siebel developers could run a suite of automated test cases, to verify the sanity of the core application just before check in. This would provide your project with a good measure of quality assurance that the application isn't broken. How good the measure is, depends on your test coverage.

Testing can be exhaustive, and take hours to regression test the Application. To make things more manageable, we can split it into two categories.

1. Core application
2. Business functionality

2, can be covered by more comprehensive regression test methods, and are done post check in. Sometimes this is too late. The concept of ‘The Test’ is to focus on 1. Testing the core application, to verify that what’s being checked in, doesn't cause widespread damage to the application in the next migration.

In the past, I've covered this topic in the following articles

SiebUnit - an xUnit implementation

SiebUnit - Build Inventory

I've had a lot contact from various Siebel customers over the above articles, confirming that the tool meets a real world need.

“The Test” is not as ambitious, its purpose is to be simple, effective and a lot easier to implement, but it’s not declarative. Writing code to test code, and maintain that extra code, isn't a deal breaker, what’s more important is that your developers have the confidence to make core changes, and your application is protected from serious defects.

Like SiebUnit, the main requirement of ‘The Test’ is to have testable interfaces. Testable interfaces are API’s that can accept an input, produce an output, without any reliance on Siebel’s application context, and are predictable. Ideally it should also have as few dependencies as possible.

In Siebel, these API’s take the following forms.

Business Service
Script Library
Browser/OUI Script

To test the above interfaces, I propose we create a Test library using eScript (for testing server side APIs), it could sit inside your existing Script library if you have one.

There should be an external method called “The Test”, this method could instantiate a new Test class, with a main entry point called “run”, that would run groups of test cases, each of these groups would contain 1 or more test cases. To facilitate the writing of clean test cases, I suggest we enforce a formal structure.

The following code snippet shows an implementation of the above concept in Siebel.

1. Calls a script library function called parseINI that will be tested. The util namespace is used, and invokes parseINI, to return object representation of a configuration file
2. assert is defined as a class function.
3. The purpose of ‘assert ‘, is to determine if a test case has passed or not. The first argument passes the evaluated the condition to assert. 
4. The Output PropertySet is passed to assert for logging the result
5. A description is provided to identify the test case.
6. Test case helpers to minimize coding.

Examples of helpers

* Create a common complex hierarchy for PS manipulation
* Invoke Siebel APIs without using setting up PropertySets
* Invoke WF without the normal setup
* Create a data randomizer to facilitate more creative test cases.

The good news is that with a bit of organization, we can reduce the amount code that a developer has to write to test your APIs. I claim that these test cases are readable, are easily maintained, and require no effort to pick up, other than the inertia to change ones habits.

Now that we have a proposed test case construct, we need to have an ability to report on the test results. Step 4 states this from a high level, but the report can be as simple as name value pairs on an Output PropertySet, or as complex as you need. The report should at the very least, display the number test cases that have passed and failed. This will allow the developer identify where the problem is, fix the issue, and re run The Test again to ensure that all expected test cases pass.

Establishing your Test library a great first step, but acquiring it alone is not is actually not enough, the project has to foster the culture of writing test cases into developers, and factor the effort into the build estimates. The project also has to identify what is testable, and augment what is not testable. A script library is a good example of a testable interface, members of the audience who have the ABS framework on their projects, should be familiar with "The Test".

It takes effort to write the test cases, as well as to maintain them going forward. To be conservative, your current unit testing estimates should be doubled when implementing automated test cases. This may sound a little daunting, so lets take a look a little example that demonstrates the ROI.

Testing an email address validator

You have a written a library function to test the validity of emails. The manual method of test this might be to launch the UI, and navigate to a view, where you repeated enter a series of bad, and good emails, to test the string. Lets say that takes you 5 minutes to perform, and writing automated test cases for the same scenario takes 10 minutes. Now imagine a defect was raised against your code, you would need to fix the defect, and spend another five minutes to re-validate the function to ensure you haven't broken something else. On the other hand, your automated test cases can be re run in an instant.

The initial effort to write the test case, will be paid back in spades. Your application will require less re-development, and suffer less downtime as a result, your developers will have the confidence to make core changes, automated unit tests can be your unit test documentation, your application will be more robust, and my personal favorite, is that developers can write the unit test once, and run it multiple times.

Once you have the right framework in place, and given enough experience (1-2 weeks), writing test cases actually saves more time than not writing them.

The Test is simple to build, and can be taken as a quick win towards a larger goal, for ensuring a better quality product. However be warned that once you start writing automated test cases, you won't want to look back to manual testing.

SiebUnit - Build Inventory

This article is a continuation on SiebUnit - an xUnit implementation, and provides an inventory of the parts needed to construct a SiebUnit framework.

Like the first article, this is aimed at the Technical Architect, who understands the concept, and knows how to put the necessary pieces together to bring this idea to life.


The architecture diagram allows us to visualize important concepts at a high level, and can be used to break up big jobs into more manageable pieces.

Rules Engine

You can replace Rules Engine with any Siebel interface that supports receiving Inputs and returning Output PropertySets. SiebUnits will also work against a standard invoked business service, or workflow.

The only caveat is that you cannot have context, that is, you must re-design your BS/WFs to decouple your logic from any specific Application context. This naturally suits the testing of a rules engine, external API such as ESB, internal APIs like custom business object libraries, and scripting utilities.

SiebUnit Framework

A SiebUnit framework can be composed of the following sub components

* Test Case Runner
* Test Case start up routines
* Test Case tear down routines (optional)
* Test case assertion
* Random data generator
* Expression language
* Expression language parser
* Expression Executor
* Expression Builder
* Test history and performance timing
* Script Library
* Business Object (BO) Library
* XML Parser
* SQL/IO Layer

Test Case Runner

This component reads the configuration data from the data store, loads all the test cases in memory, and runs each test case one by one. It records start, stop times, and logs the output, but does not make any determination on the test case result. Errors are captured, and any exceptions are suppressed, allowing the Test Case Runner to continue on until it completes all test cases.

Test Case Startup/TearDown

In order to test certain scenarios, prerequisite test data may need to be setup before the rules can be tested. This component constructs the initial test data before the test suite runs. Optionally, there can be a Tear Down component, which cleans up after all test cases.

Test Case Assertion

Assert is a computer science term used to ensure a condition is true/false to enforces application integrity. The assertion component performs the same role, it checks the outputs of each test case, and validates it against the expected result for that test case, logging the result to a history data store.

XML Parser

This component converts XML into a structured object and back. This can be used to send dynamic outbound messages, and validate inbound responses, for integration testing.

SQL/IO Layer

This component retrieves the related data from an entity for validation purposes. It is used by the Test Case Assertion component.

Random Data Generator

Even within predefined test boundaries, we can allow the generation of random data to simulate dynamic conditions.

Eg. Generate a bunch of expense activities of random type, and amount, and test it against a business rule for assignment or approval logic.

The random data generator can be used by the Setup component, or the individual test case to randomize inputs into a rule.

Expression Language

To make SiebUnits declarative, we need a way of allowing the developer to construct data, supply dynamic inputs, and validate the results of the test cases, without writing a line of code. This can be done via an Expression Language, similar to calculated field expressions in Siebel.

Expression Language Parser

This component is responsible for translating the above expressions into a form which is understandable by the Siebel eScript compiler or external engine, that will in-turn run the instructions.

The expression parser needs to differentiate between a literal and expression. This is done via 4 ways, as illustrated throughout Siebel.

1. Field Pre/Post Default

Expressions need to be prefixed with "Expr: " or a key word such as "System", "Field" etc. Anything else is considered a literal.

If you wanted to pre default another field via the square bracket notation "[Id]" this will actually result in a literal value

2. Calculated Field

A calculation is always expected to be an expression, unless it is surrounded by quotes. So the square bracket notation "[Id]" is treated as an expression in this context.

3. Field Type

This example can be seen in WFs in the Output argument of some steps. There is another field that indicates the type. Eg: Literal vs Expression.

4. Dot Notation

WFs also have another interesting expression, with the dot notation to reference child elements within property sets. This dosnt really fit into either of the above 3 categories. As it can easily be interpreted as a literal, and does not have any prefixes to indicate that it is an expression.

You have to make a call on which method you will use, and your expression parser has to recognise the above patterns if you mix the scenarios.

Eg 1. For an eScript implementation, the expression


Should translate this to an eScript call like this


The parser has to recognise that this is an expression with a literal argument.

Eg 2. The next expression provides a dynamic input.


This follows Siebels WF standard, of referencing WF properties via the ampersand prefix inside square bracket notation, requires a bit more work because you have to deal with the nesting of expressions.

This is where you can draw the line, and define styles for your expressions and have another expression to deal with that scenario


Now the literal references a property that holds the value to this method. You will have to plan out your expression language in advance, and know how your parser will deal with each scenario.

Expression Executor

After you've translated the expression into the native language, it needs to be executed, and the result returned. The method that you execute the expression, will impact on the style of your expressions, and how complex your parser needs to be.

Consider implementing a short circuiting mechanism so that, if part of a condition expression is met, you can abort executing the rest of the expression.

Expression Builder

This component allows the developer to choose from a defined set of expressions, this aids in learning, reduces mistakes, and as well as documenting all the supported functions of SiebUnit. Ideally It has to integrate with the Siebel UI session, and apply the expression to the list/control.

Test History and Performance timing

This data is produced by the Assert component which validates the test case runners run history with the expected outputs, it also extracts performance timings for analysis.

Inputs and Outputs of the run are also included in the run history to help debugging. Having the results in the UI is consistent of the expectations of a declarative tool. Take note BRP engineers!

Test history and performance data can be transient, and the user can export the results, if a base line needs to be established.

Script Library and BO Library

It is important to differentiate between the above two components. A script library provides utilities, and prototypes on native objects, while a BO library provides business functions.

A SiebUnit library is tightly integrated with the script library for performing internal operations, and provides proxy functions for invoking the BO library to setup and perform test cases. When creating framework functions you will have to make conscious decisions in which library a function belongs to.

Unit Testing and Regression Testing

SiebUnit does not replace commercial regression testing suites. An important differentiation is that SiebUnits is designed to be used by developers for unit testing code/WFs before it is checked in, and to ensure application integrity is maintained before it reaches the testers.

SiebUnits are also more thorough, as only the developer could understand all the exception paths of a program. SiebUnits test technical functionality, while regression test tools test functionality.

Although SiebUnit can be used to regression test business rules, it does not test UI components. The developer is still expected to ensure that UI rules are unit tested. A badly configured link, can still bring the system down and while writing SiebUnits to create data, and check the data returned from the link could be a valid test case. It was not intended to micro manage every piece of individual configuration.

SiebUnits can offer a great sense of security for the developer, but poorly designed test cases, can also provide a false sense of security. This is normally due to poor test case coverage. Average developers will usually test the positive scenario, good developers test both positive and negative scenarios. SiebUnits require the developer to understand all the possible logic paths and design test cases to thoroughly cover off the scenarios that can result the program to fail.


There are numerous benefits to having SiebUnits on a project. The immediate return on investment, is that your testing period is reduced, and your application will be more robust, but the real payoff of really comes down the track, when you build highly complex systems, where one small change can break another part of the system. SiebUnits can provide your developers with confidence to make that change, without costly regression testing. SiebUnits will keep your Application, and also your sanity in check.

xUnits is a proven concept, and is a staple of any modern development platform. SiebUnit provides an evolution path, so that your developers no longer need to manually unit test. The world has JUnit , jsUnit, CUnit, OCUnit, PHPUnit, PyUnit, SUnit and now we can add SiebUnit to the list.

SiebUnit - an xUnit implementation


Over a year ago, I presented a concept of xUnits in Siebel and how they could be used. Because it was only a concept, it was missing implementation details, so there was some doubts to whether Test driven development would work in Siebel, where code is meant to be restricted to fringe cases.

At that time, I ported some code from a back end Java project that I was working on, to automate Siebel interface verification. The idea was to pickup XML templates from a file system, insert randomize data, send it to the ESB, and validate the results.

It was used for several releases, reducing a days worth of effort to verify 50 interface interactions, to about 5 minutes. With the JUnits in place, they could be used to verify an integrated environment instantly on demand, instead of employing highly paid professionals to perform repetitive regression testing.

The tool was written in Java, didnt have a UI, it had access to the eApps SSO hashes, which meant that an evil developer could use it to log onto a server environment, and impersonate anyone they wanted. It was more of a regression testing tool than a unit test tool, and being written in Java, it was inaccessible to most of the Siebel Team.

In the spirit of a true Siebel developer, I thought a real implementation of xUnits in Siebel had to be scriptless.

So "SiebUnit" was designed to be just that, a fully declarative unit testing framework for Siebel developers.

This framework also happens to plug into a rules engine like BRP, OPA, or it could be run against a collection of business services/workflows, as long as you can pass in an Input and get an Output property set, the "SiebUnit" framework handles the rest.

This idea was sold to a Client, who immediately picked it up as a way to get developers to unit test, improve build quality, and reduce dependence on testers, and provide a structured development process.

This article and its followup, takes a step further, providing architectural and implementation concepts. It is aimed at the Technical Architect, who can grasp high level concepts, fill in the low level implementation, possibly innovate on this idea, and introduce SiebUnits into your Siebel project.

What is SiebUnit

If you had to sell the concept of xUnits in Siebel, this slide should sum it up.

This screenshot shows a SiebUnit implementation that was built for a Client.


Notification System for test results

Uses a custom notification system, written in a client side library, to add persistent messages to the current view. The messages can be dismissed, by clicking the X button.

Design and Run Test Cases from the UI

Test Cases are configured, executed and debugged all within a standard Siebel UI. Debugging the rules itself, depend on the capability of the rules engine. Eg. BRP can only be debugged via log files, WFs can be debugged using the instance monitor.

Export/Import test Cases

Allow easy export and import of hierarchical test configuration data between different environments.

Trace Toggling

Toggles developer tracing, for performance testing of rules

Dynamic test data

Uses XML templates, and proxy functions to the BO Library to generate unique test data. Random data can be generated and provided on inputs, or overridden in the case of XML templates.

Expression Builder

A helper utility designed for the developer to construct the test cases declaratively. This is built using HTML, which integrates with the Siebel session.


SiebUnits comes with pre built internal test cases to validate your eScript engine, and profiles ST objects vs non ST objects.

SiebUnits can be configured to run recursively to load test an environment

Case Study - Custom Rules Engine

SiebUnits was used on a project to ensure the integrity of a Rules Engine.

The design of the custom rules engine started off in small components, followed by unit testing, further build and some more unit testing. The components were then integrated, followed by another round of unit testing.

As the design got exponentially more complex, the integration of all these components, made it impractical to regression test all the exception paths, after every change.

SiebUnits was originally built with integration to BRP. So for this project, it was modified to work with this new engine.

A collection of SiebUnits was built to test the following aspects.

Declarative IF conditions
Declarative loops
Declarative procedures
Expression Translation
Expression Results
Business Service Invocations
Property Assignment
Cancel Propagation
Error Messages
Math Operations
Rule inheritance
PropertySet Interactions

Some further internal test cases were also designed to validate the behaviour of the eScript engine. This ensures that if Siebel changes the eScript engine, these internal SiebUnits will pick up on the expected outputs of dependent core functions, and allow the root cause to be addressed.

Once the SiebUnits were designed, and configured, they were used to repeatedly validate the Rules Engine after every change, to ensure previously built functionality still functioned. Changes often broke other unexpected parts of the engine, due to the tight integration and dependency between components, but with the SiebUnits in place, the report of failed test cases helped to quickly pinpoint the cause.

Towards the end of the build, a critical defect was discovered with the eScript engine, that corrupts a core object of the rules engine. This happens during random runs of the test suite, but it was easily reproducible using the load runs.

The problem required a re-design of the core object, which unfortunately touches every function in the engine. Since the new object had a different interface to the old object, it required a re-design of most of the interactions around this object as well.

The affected parts of the program was identified, and a fix was implemented in two hours. The SiebUnits test suite was run afterwards to ensure that the rules engine passed all the test cases. This provided a QA stamp on the product, and was similiar to having a team of regression testers on hand to validate your work at will.

The collection of SiebUnits also allowed load testing of the engine in a real environment. During these load runs, server resources can be monitored to ensure that they run within acceptable standards and allow capacity planning based on expected usage.

It would have been much more difficult if not impossible, to complete such a complex build without SiebUnits in place, and the regression testing and maintenance effort would have been an ongoing burden.


Do you have BRP, OPA or a collection of Business Services that you would like to keep in check? The next article will look at the architecture of SiebUnits, and all the components that need to come together to form the framework of SiebUnit.

Note: The term Rules Engine is used broadly in this article. Even though BRP stands for Business Rule Processor, it is actually a declarative eScript replacement engine. OPA is more suited to the term for a Rules Processor, as it takes a bunch of inputs, makes a determination and returns the result. However the Rules Engine interface of Inputs/Outputs is what makes it suited to SiebUnits.

View the next article here