Showing posts with label Integration. Show all posts
Showing posts with label Integration. Show all posts

Asynchronous Workflow With Real User Login


Background


A common requirement amongst many Siebel customers, is to have the ability to run an asynchronous workflow under the real user's login.

This issue is highlighted in the following Oracle support web document, and is representative of the problems faced by many customers.

NOTE:494815.1 Setting CREATED_BY to the user that invoked the workflow process asynchronously

The customer stated that they have an asynchronous workflow, and wanted to know if it was possible to set the creator login of a server request. The solution offered by the support personnel was to run the work flow in synchronous mode.

Asynchronous workflows are used in Siebel to offload work from the user session to perform background processing, however that leaves an undesired effect of stamping the record with SADMIN. This behavior obscures information about the creator, updater, and prevents audit trail from being utilized.

Workflows can be run synchronously as suggested in the Oracle support web document above, but it has the rather unwanted effect of blocking the UI.

Fortunately, for customers who can't accept running background tasks synchronously, there is a viable solution, but it requires a little bit of effort.

Requirement


To re-affirm the requirement, we are going to walk through a solution that allows any workflow or business service in Siebel to be run asynchronously, under the actual users credentials, and without actually supplying the password!

If we step back, and look at the problem from an architectural perspective, we know that when the user logs into the Application and executes any type of code such as eScript, or workflow, it will run synchronously under the users application object manager, and it will operate under the users own credentials.

The only supported method to run code and avoid blocking the UI, is to run code under a background process, however any code that is run in the background will operate under SADMIN by default.

The first challenge is to find a supported method to execute code in a background session under the current users credentials. Experienced Siebel Integrators in the audience can probably guess, that we can achieve this with any Siebel API that supports SSO, or utilize the inbuilt impersonation capability of certain Siebel components.

The two most suitable choices are:
1. Siebel WS
2. Siebel JDB

I've highlighted these particular interfaces, because both provide APIs that can be invoked from a background session, and has the potential to allow arbitrary code to be dispatched, and run under a different set of credentials. As an added benefit, both of these APIs can be built to support load balancing for high availability.

With a suitable SSO API identified, we have the necessary key piece required to solve the above problem.

Solution Overview


The diagram below shows an overview of this proposed solution.

1. The user triggers an asynchronous task from the current user session, and is allowed to continue with their work in the application

2. In the background, A WFProcMgr component on the server executes the process, and calls a bespoke service called "JLE Session Transport Service" that acts as a wrapper for the transport.

3. A synchronous call is invoked over the transport, using the Siebel API identified above

4. The Siebel API instantiates a new session on behalf of the real user, executes the destination business service, and returns the results back to the background process.

Siebel API


An important implementation aspect worth considering upfront, is designing a transport agnostic API for the Siebel developer. The developer should be able to specify the desired subsystem, and the service should abstract the low level transport requirements, and perform the necessary acrobatics to make the call.

The following diagram shows how a facade is used for invoking the actual SSO interface.


A standard Siebel business service is defined, so it can be substituted in any existing workflow, or hooked into any existing Siebel trigger. The designer now has to implement the adapter code, and build the interface to communicate with the relevant SSO interface.

The implementation of interfaces to these Siebel APIs are out of scope of this article, as it requires different spectrum's of design, involve advanced integration, and require environment dependencies to exist, but the key considerations are provided below for customers who are interested in building this capability.

Depending on which SSO API is chosen, you may require an Integration specialist to implement a WS/BS dispatch wrapper, or a Siebel EAI/Java specialist to implement the Siebel-JDB bridge.

Please consult with your Siebel Integration Architect for more localized implementation advice.

Input Arguments


To support a plug and play design, I propose that all input arguments and child properties passed into this service would be dispatched to the remote business, in the same way as any other business service in Siebel including workflows.

This design allows us to go into tools, and retrofit this capability to any business service, without redevelopment effort. It is also deliberately designed to avoid hierarchy changes required for the correct invocation of the destination business service.

To control the dispatch, lets define a custom child hierarchy, which contains 3 key elements.

1. The user login to impersonate
2. The destination business service to dispatch to
3. The destination business method name

The diagram below illustrates the property set structure, with the required information to dispatch the call



This special PropertySet can sit at any index, however it should be removed before it is dispatched to the remote business service, in-case there is logic that is sensitive to property sets with specific indexes.

PropertySet Serialization


PropertySets are a Siebel proprietary representation of objects that only exist in memory. In order to send a PropertySet out of Siebel, and receive it back in, it has to be serialized, de-serialized, and potentially encoded to match transport constraints.

XML is usually used to transfer data between different systems, and Siebel provides methods to convert the PropertySet to XML, and back, however a more efficient method is to utilize Siebel's own PropertySet to text encoding format, or a utilise a custom JSON parsing engine in eScript. This minimizes the size of the message, results in less IO, and ensures that request is sent as efficiently as possible.

Error Handling


This is the most critical aspect of the design, as every component in the design has to be proactive in handling, and bubbling errors back up the chain.

The caller is effectively making a synchronous request/response call between two sessions. Errors at the end of the chain in the impersonated session, has to be propagated through the transport, to the calling session, which has the responsibility to handle and log any errors.

It is imperative that each component have guards in place to handle environmental, transport, and general unhanded exception scenarios, which could cause the call to fail.

This can happen for a variety of reasons including

Component is offline
Component Maxed Task
Connection timeouts
Failed authentication
Out of resource

The Siebel JDB interface has OOTB capability with extra options for configuring, settings such as retry, and timeouts, to counter some of the above issues. Customers choosing the Siebel WS API will have to consider implementing the appropriate mechanisms to guard against transient errors.

Load Testing


Both the Siebel WS, Siebel JDB options can be configured for load balancing through virtual server definitions. This capability allows this solution to scale easily. In practice this only works natively for the WS option, the JDB interface seems to only get load balanced if sessions are spawned from the thread. An as alternative, the load balancing responsibility for the JDB interface can be offloaded to the SRProc component.

Its important to understand the current expected load, and forecast the expected number of tasks that are needed. Every invocation can potentially spawn a new session, which could effectively double the amount of threads that the system has to spawn, along with the increased resource utilization, so it would prudent to involve your performance test team, and environment specialists to ensure that the application is tuned to handle the extra load.

Maintaining Impersonation


One last consideration in this design, is that once the user dispatches the work flow to be picked up in the background, no part of the invocation chain is allowed dispatch to another server process.

The impact is that the developer needs to ensure that all work flows are run inside the impersonated object manager. If a server request is initiated, that would then dispatch to a server thread, breaking the impersonation for that particular thread.

Conclusion


The above capability was designed and delivered for a customer who needed to meet regulatory audit requirements, to capture the identity of the users when they performed bulk CRUD operations across the application.

In the past customers who needed to meet this requirement would have chosen to take a hit in performance, and kill the user experience, while expensive background operations are forced to run in the foreground, or resort to creating extra columns across entities in the application to separately capture the identity of the user.

With a little bit of effort, Siebel customers now have the option of creating a custom impersonation component, that can run asynchronous tasks under a real user.

Advanced Siebel Integration - A SOA Review



A recent Oracle SOA review into one of my customers Siebel Integration capability, rated it as highly advanced. The innovative systems, processes, and tools that were put in place, evolved immensely over the years, and enabled the customer to build, deploy, test and maintain their interfaces quickly and with little effort.

In this organization, the Siebel team provided the front line EAI development, and supported a large portfolio of services across numerous touch points and multiple transport types including MQ, WS, File, HTTP. While adhering to architectural principles such as SOA, P2P, Pub/Sub, this Siebel system plays the role as a consumer, subscriber, publisher, and router of services.

The environment mix, architecture, and scale may differ in your organization, but there are some fundamental challenges that all Siebel Projects face.

High Level Challenges

- Simplify and standardize Integration
- Reduce Integration expenditure
- Agility and Time to market

Simplify and standardize Integration

From a high level, there are a variety of messaging requirements that we may encounter, including Request/Response, Notification, Async callbacks, Pub/sub, then we have various transports to handle, which may include HTTP, Message queues, file formats. We then have to choose between the various transformation options, and figure out how to generate our message with the correct headers, deal with message dispatch and error handling.

There are a variety of implementation options, and each developer will have their own approach to solving the above problems, which can lead to non standardized solutions, that overtime becomes difficult to maintain. Reducing the complexity, and standardising the way we implement integration requires a system that helps us to deliver a consistent solution to dealing with the above scenarios. 

A key component of this system is establishing a Siebel Integration competency centre (ICC), which acts to standardise Integration solutions. An ICC can be defined as team that is responsible for the Siebel EAI needs across different streams and projects. 

Another vital component is the integration framework. An integration framework encapsulates all common integration logic, implements standard design patterns, minimises customisation, coding, and minimize deviation from the standard design entry/exit points. A good integration framework provides the ability to integrate a lot of different technologies, and is instrumental in enabling projects to simply their Integration.

Reduce Integration expenditure

Some organisations approach integration on a case by case basis. The work is put together in an adhoc manner to achieve a certain result, and the grunt work is then repeated on the next project. Issues such as maintenance, error handing, and robustness, unfortunately is an after thought. This causes a lot duplication, waste, increases cost, maintainability, and risk.

An alternative approach is to look at introducing an integration factory. The concept of an integration factory, is not dis-similar to how car factories mass produce vehicles using standard core components, where robots perform the heavy lifting and mundane tasks on a production line, and engineers move in to perform the specialist tasks that require finesse and skill. This results in a consistent and quality deliverable, that is cost effective.

In the same way a Siebel integration factory, can delegate or automate certain integration chores, to free up the Siebel Integrator to perform more value added tasks. This factory consists of a set of tools and processes which allows projects to churn out interfaces using a cookie cutter approach to reduce integration build and test.

Under this approach, less experienced integrators can be used to safely deliver interface work, because the factory enforces a consistent approach to the build. Contrast this with an adhoc build approach, where a less experienced developer is provided with 101 ways to implement something simple like validation, and countless ways to trigger, and implement automation. We can imagine the outcome will be inconsistent, to say the least, and is likely to result in increase costs for the next project who has to maintain this work.
 
An Integration factory provides tools to work with messaging requirements, scaffolding for quick deployment of interfaces, and a controlled process to produce interfaces. Integration artefacts such as VBCs, Inbound, Outbound, Request/Response interfaces all have a certain flavour, and are produced with the same common components, and are orchestrated to meet a requirement. 

By applying standardisation, establishing reusable components, and using proven design patterns to solve known problems, we can reduce the cost and risk of implementing integration.

Agility and Time to market

Many Siebel projects need to support rapidly changing business requirements, and deliver on future initiatives at the same time. Integration is traditionally seen as risky, costly, and usually delegated to major releases, which limits the ability to be agile and deliver solutions quickly.

One of the keys to agility in Siebel is to keep logic out of the SRF, and fortunately Siebel provides a lot of options to configure EAI capability on the fly.

Client side business services

Client side business services allow Siebel customers to configure and deploy technical logic to a run time database. This capability allows project to change logic on the fly, and maintain a 24/7 application. There are customers that purely use Client side business service because of the agility advantages over eScript.

Siebel Workflows

Siebel workflows are the work horse of automation in Siebel, it provides a nice interface with a visual representation of customer logic. Workflows can be configured and deployed to a run time database, allowing projects to switch workflow definitions on the fly, and apply logic changes without downtime.

Business Rule Processor (BRP)

BRP was designed as declarative replacement for eScript, and allows projects to build application logic using a browser. These rules are then executed by a highly optimised engine that works beneath the Siebel BO layer to maximise logic execution. BRP rules can configured and deployed using the Siebel thin client.

Data Mapper

Datamapper is Siebel’s declarative engine that performs an object to object transformation. Data map transformations can be also configured using the Siebel thin client.

Runtime Integration Objects

Integration objects are representations of a model in Siebel, which is usually built using Siebel tools, and compiled into a SRF. However, with an appropriately connected thick client, Integration object definitions can be deployed to a run time database. When runtime Integration Objects are combined with the right mix of EAI technologies, Siebel customers can build and deploy interfaces without downtime.

XSLT

Siebel provides XSLT 1.0 as standard out of the box, but customers can easily implement an XSLT 2.0 engine to get access to an expanded set of transformation options, and speed benefits. XSLT provides a powerful declarative transformation alternative to eScript or Data mapper. 

The Siebel product has a solid range of construction tools out of the box, that allow for changes to be deployed without downtime. Siebel customers that make best use of these technologies, have the opportunity to be very agile and reduce the time to market for their solutions.

Summary

In this article we’ve introduced the high level challenges of Siebel Integration that organisations  face. This prepares us for the next set of articles, where we’ll dig deeper, go through real world scenarios, consider solutions that are aligned to the principles introduced above, walk through the features of a Siebel integration framework, and address the following low level challenges.

- Reducing technical burden
- Dependency on external interfaces
- Working in disconnected environments
- Working within constraints of Siebel product
- Reduce scripting and hard coded references
- Improving code quality 

A Better XML Logger

This article is Part 3 of Siebel Error Handling Approach

Introduction

Last time we looked at an XML logger implementation using the vanilla "EAI XML Queuing Service", and ran through its shortfalls. This time we go through the design of a better XML logger, and introduce a new XML tagging feature.

The keys to building a better XML Logger comes back to understanding how the EAI XML Queueing Service works. The Queueing service takes a SiebelMessage, converts it from a PropertySet to an XML document, and saves it to a file attachment BC, all of which are standard Siebel operations. So we can use this design as the basis of our own EAI XML Queueing service implementation.

High Level Design

1. Create an IO around the EAI Queue/Item BC
2. EAI Queue Item will store References to our Error table, and XML Tagging information
3. Implement an XML query capability to extract information from the message
4. Dynamically Generate a Siebel Message around the EAI Queue structure, with our XML message as an attachment
5. Use EAI Siebel Adaptor to insert the XML message
6. Put a view on top of the above components to allow association to the error table, and to support querying through the UI

Build Steps

1. Create a new WF: "A Better XML Queuing Process"

This workflow has to support logging of either a SiebelMessage, XML document, or any other type of interface message Eg. If a SiebelMessage is provided, this has to be converted to XML Doc.

2. Generating the EAI Queue SiebelMessage

This WF step generates a SiebelMessage that contains the above interface message as an attachment.

There are 3 mains method for creating an instance of this IO

1. XSLT
2. PRM ANI Utility Service
3. Scripting

PRM ANI Utility will easily generate a SiebelMessage structure which is 90% of your work, but it will not generate the File Attachment IC, so it has to be complemented with some scripting, or go for the declarative option using XSLT, which will allow us to generate a message complete with the Attachment IC without resorting to scripting.

3. Populating the SiebelMessage

This WF step populates the SiebelMEssage with actual data and will require some sort of transformation, so you have the following options

1. Data Mapper
2. XSLT
3. Scripting

Once you have your SiebelMessage, using Data Mapper with inputs should be straight forward, however, using the XSLT method requires a little more work, because you will need a source document that contains all your source values, plus the XML payload. This can be achieved by using an empty SiebelMessage, and using dot notation to inject all your variables, including the XML payload, into the SiebelMessage header as attributes. These attributes would then be available in your source document to complete the transformation.

The following, is a list of the attributes required to produce an IO with attachment

XML Payload
Comments
Queued Timestamp
Reference Id
Reference Val2
Reference Val3
Sequence Number
Status
Queue Name
AttachmentIsTextData
Extension
MsgFileAutoUpdFlg
MsgFileDate
MsgFileDeferFlg
MsgFileDockReqFlg
MsgFileDockStatFlg
MsgFileExt
MsgFileSrcType
MsgFileName

The vanilla EAI XML Queueing Service produces attachments with a rigid format similar to this

s_eai_queue_itm_1-62O6DB.txt

The file type is hard coded to txt, and the filename cannot be controlled

The "MsgFileName" is of particular interest to us, as it allows us to produce a more meaningful filename.

4. Insert SiebelMessage

At this point, the populated SiebelMessage is ready to be Inserted via EAI Siebel Adaptor.

5. Build a nice UI to retrieve these XML logs, in association to its Error log.

Our XML files are saved to the Siebel file system, allowing users to retrieve the XMLs without going through file system permissions.

Message Tagging

Message Tagging provides the ability for a project to peek inside the XML message while in transport, and log certain elements/attributes for searching. This was one of the downfalls of EAI Queueing Service, as key details of the message were hidden, until you opened each message individually. This capability is important because Siebel zips the files stored in its attachment folder, and the contents become unsearchable, unless we tag it.

The following XML is a sample of an image notification message. The key pieces of information to be tagged from this message are the Id, PartyType, SourceSystem, and NewImageFlag.
<?xml version="1.0" encoding="UTF-8"?>
<PartyUpdate>
   <PartyIdList>
       <PartyIdDetails>
           <Id>1-XXXX</Id>
           <Type>Siebel</Type>
       </PartyIdDetails>
   </PartyIdList>
   <PartyType>PERSON</PartyType>
   <EventSource>ESB</EventSource>
   <SourceSystem>Imagr</SourceSystem>
   <NewImageFlag>true</NewImageFlag>
</PartyUpdate>
The schema would obviously allow for other types of notifications, and the Message Tagging capability would need to be flexible enough to cater for these other schema options. Message Tagging can be implemented with different kinds of tools, but at the heart of it, we need to query data from the XML.

A few Siebel favourites which can be used for deep hierarchy extraction are

1. FINS Industry XML Query Service
- Allows XPathish notation to be used

2. PRM ANI Utility Service.GetProperty
- Supply the IC name, and property to be extracted

3. WF Alias
- Similiar to Dot notation, but without the 75 character limitation

To be honest, none of the above options are particularly flexible, especially for dealing with namespaces, repeating groups, filtering, getting the nth child, dynamic querying etc. There isn't any vanilla querying capability, that meets this particular requirement.

So for your more ambitious Error handling needs, I would go for a Java based solution, that implements the full XPath standard, allowing all of the previous problems to be addressed. This would be implemented in a JBS, which accepts two arguments the XML, the XPath expression, and returns the XPath result.

To follow our theme of flexibility, the system shouldn't be hard coded to look for certain elements, the design should allow the customer to configure XPath statements, and associate them with an Interface, and maybe more specifically an Interface direction. Going down this path, takes us down a level of abstraction that is not available out of the box, which is a good thing, except for the poor designer who has to materialise this concept!

Conclusion

In this series on Error Handling in Siebel, we've taken a look into the key features of an Error Handling framework. We also delved into the complex error handling requirements of an EAI professional, dissected EAI Queueing Service, and provided the map for framework designers to build a better XML Logger. For an alternative approach, fellow Siebel expert Mik, also provides a different design that readers should also consider.

The capability to capture an XML message, along with its associated error detail, should be a core part of every projects toolset and there should be no doubt, on the importance of this feature. Error Handling Frameworks are not new to Siebel, but they are only available behind closed doors, so the comments are open to readers.

Do you have classy Error Handling ideas that you would like to share with the rest of the world?

An XML Logger

This article is Part 2 of Siebel Error Handling Approach

Introduction

When we think of the top Integration tools used in a Siebel project, typical answers such as SOAP UI, Oxygen, XML Spy, Wireshark, Fiddler, MQ Harness, and even Hex Editors may come to mind.

But there is an even more important tool which is missing. You've probably guessed it, what list would be complete without an XML logger?.

Not to be confused with an eScript logger that prints a property set or a WF step that simply writes the XML to a text file. An XML logger is a component, that is part of a more larger Error Handling Framework, which should provide the capability to associate the XML message with the related error information.

Importance of an XML Logger

Once upon a time, I took over the reigns of an Error Handling framework that had an Achilles heel: it chopped off every single EAI message at the 2000 character mark, and all that was captured was a partial message header full of name spaces. Not only was the payload discarded, only part of the message wrapper was captured.

For the benefit of non EAI professionals reading this article, imagine that as a kid, someone gave you a lollipop, except that there was no candy, not only that, half the plastic wrapper was torn off, so you don't know what flavour it was, or even the name of the bastard who gave it to you. This situation could bring any kid or EAI professional to tears.

For an EAI professional, living without XMLs is like working blindfolded, it's almost impossible to pinpoint where failures are occurring.

If the system didnt log the outbound message, the developer would have to spend time recreating the test case, and simulate what the message "could" have looked like at a point in time, assuming the problem is reproducible. Without access to the full inbound message, your hope would rest on the external system providing the message, that your system should have rightfully logged.

Having to live with poor error handling, is a hard lesson to learn, but it does illustrate a common need for projects to have access to all Inbound and Outbound messages across all Interfaces.

EAI XML Queuing Service

Siebel provides a solution in the form of the "EAI XML Queuing Service"

http://docs.oracle.com/cd/B31104_02/books/ConnSAP/ConnSAPeAIQueue12.html

Professionals in the Siebel Community have already discovered this business service has the capacity to act as an XML logging tool.

http://siebelexplored.blogspot.com.au/2011/01/exception-handling-using-eai-queue.html

http://siebeltips.wordpress.com/2009/02/19/eai-queue-for-error-handling/

http://geeksbloggingat.com/2009/05/03/siebel-eai-queue-usage/

To summarise, this out of the box business service can be used to log a SiebelMessage to the Siebel file system for storage and retrieval.

The EAI Queuing service has the following benefits

★ Can store more than 2000 characters! Hooray
★ Message can be logged with Error Information
★ Can be used to replay a transaction
★ OOTB

However, if you have a diverse portfolio of services to interact with, where the SiebelMessages have to conform to Enterprise schemas, or even third party proprietary formats, then you will need the capability to log the raw interface message. This could be in the form of XML, CSV, flat file, or it could be SiebelMessage with a different wrapper.

The problem with the "EAI XML Queuing Service" is that it only supports logging of a valid SiebelMessage, which might seem to dismiss this business service as a credible solution, but there is an easy work around. A simple XML based IO can be created, with one IC, and one Field called "Payload". The XML can simply be injected into this Payload property to store your XML. This XML based IO simply acts as a wrapper to allow the XML to be accepted by the EAI XML Queuing Service.

This works, but it does have a few major drawbacks.

★ The wrapper has to be stripped to reveal the Raw Message
★ The raw message, wont actually be raw anymore. XML cant be nested in other XML without being interpreted as part of the schema, so Siebel will automatically HTML encode your raw XML, so it can be accepted as text.
★ The file type is TXT, and the file name cannot be controlled
★ There is a 1-1 relationship between the error and the message
★ The available fields to store error information is limited

There are major limitations with using this OOTB business service as a standalone solution, because in order to exchange messages with external teams, the EAI Queued message has to be stripped of the wrapper, and decoded before it can be used. EAI Developers can perform this task without complaint, but it is hardly accessible to other teams.

Eg. Testers who have access to the raw messages, can raise defects, and exchange the raw XMLs with external teams directly, but wont necessarily have the tools or knowledge to decode XML, so providing access to the raw XML provides convenience and saves time.

Using the EAI Queue Item as a means to store error details, also means you are limited to storing one message per error record, unless a method is devised to flatten the SiebelMessages, and store them as properties in a wrapper IO, which comes back to the decoding problem described above.

The other issues described in the points above are fairly obvious, and don't need any further narrative, so lets move on, and look at the ideal solution.

XML Log Details

In the last article, I provided a list of error details that should be captured. Here is a revised list of the error details that should be captured when an Integration error occurs. I've expanded on the original list to reveal the specific details an Error Framework and XML logger should handle for Integration (highlighted in blue).


★ User Name
★ Error Time
★ Host Name
★ Primary Id
★ Internal Error Code
★ Error Message
★ Error Category
★ Transaction Reference
★ Transaction Time
★ Calling Function/WF
★ Stack Trace
★ External Error Type
★ External Error Code
★ Transport Time
★ Transaction Id
★ Transport and Transport Parameters
★ XML Logs
   Outbound XML
   Outbound SiebelMessage
   Inbound XML
   Inbound SiebelMessage


It should be clear at this stage, that we'll need a custom table, and a parent child relation between the error message, and the interface messages. It should also be clear that the EAI Queuing Service works OOTB, but dosnt provide the flexibility for complex EAI error handling requirements.

There is a lot to take in, so in the next article, we go through the solution to overcome these limitations, and introduce a new capability, that will allow you to have full visibility of the interfaces firing around you.

Funky Town: Non blocking UI

This post is part of the Funky Town series of articles, where we take ridiculous User Centric Design (UCD) requirements and put them into practice.

Imagine that the user

1. Clicks on a button that kicks off a long running report on the server and gets back immediate control of the UI

2. Navigates to another view that fires off several interfaces to get data back

3. As soon as each interface call completes, the screen refreshes with the new data

4. Minutes later the report finishes and the user is presented with a real time browser notification

If the advantages dont jump at you, this list will make it obvious

1. Non blocking UI
2. No nasty polling to maintain
3. Reduced network traffic
4. Less waiting time
5. Real time server to browser notification
6. Parallel processing

This sounds like some modern website, but with a bit of engineering, we can transform Siebel into a more feature rich application.

Basics

Siebel executes code in two modes
1. Synchronous
2. Asynchronous

Synchronous

Server processes invoked by a users session eg. eScript/WFs run synchronously. In other words, you cannot have two pieces of code run in parallel. Synchronous execution freezes the users screen (including any browser side animation) and forces the user to wait until one process has completed before it runs another process.

Asynchronous

Processes can be made to run asynchronously, by dispatching the execution to a background process. This mode provides control back immediately, but a callback is not provided when that process finishes.

However with a little fancy footwork, we can get the best of both worlds. Dispatching code asynchronously with a callback.

Implementation

We will build a button that fires a piece of code that runs on the server for a 10 seconds. The user will be free to do what they choose, and when the server process is complete, a sexy message slides into view, displaying to the user the number of seconds the code took to run.

Ingredients

1. Ajax

Allows us to execute code in the background and get a callback. We will use javascript to launch our background process and listen for the response.

2. Interface

Since the code is triggered client side, we need a browser accessible Siebel interface that allows us to execute code and return some values.

Pick one of the following or come up with your own

A. SWE Querystring
B. XML Command Block
C. SOAP

3. Read Results

I'm going to pick option A. Options B and C will also work, but require dealing with XML payloads and using HTTP POST in javascript, which I'll leave for EAI developers to play with.

The solution entails using Ajax, which dispatches a call to a background process, essentially running synchronous code in a separate session. The user is free to move on and perform other tasks, when the background process finishes, it returns to the browser which registered a callback, the callback function can then notify the user of the results of the process.

4. Proxy

The proxy BS will act as a wrapper to call the intended process, and add the HTTP response headers. These headers are required because invoking a business service using the query string kicks off a server process, which dosnt return a property set to the browser. The proxy business service will convert the outputs into headers that the browser can understand.

5. Session

If we run Ajax calls against our active session, it will still block our UI, so to get around this problem we need a separate Application to offload and invoke our interface. Your second Application can be SI or HI, as long as its different from your active UI session.

The interaction sequence looks like this



There are two key pieces of code that is required for this to work.

1. Ajax Call

The following code is initiated from a button on the browser, and makes a HTTP GET request, using the SWE Querystring API. This querystring calls a Proxy BS, passing in the destination Business Service, Method, and arguments.

It also registers a callback routine to handle the result from the background process. This callback function can take summary data from the server business service and display the results to the users session or it can detect the users view, and refresh data that has been loaded in the background.

var reqUrl = "/Application2/";
reqUrl+="start.swe?SWECmd=ExecuteLogin&SWESetMarkup=XML&SWEDataOnly=1"+
"&ParamBS="+sBS+
"&ParamCallMETH="+sMeth+
"&SWEAC=SWECmd=InvokeMethod&SWEMethod=test&SWEService=MyProxyBS";

var xmlHttp = new XMLHttpRequest();
xmlHttp.open('HEAD', reqUrl, true);
xmlHttp.onreadystatechange = function() {
if (xmlHttp.readyState == 4) {
if (xmlHttp.status == 200 || xmlHttp.status == 204) {

var sResp = ""+
",Status: "+xmlHttp.status+""+
",Result: "+xmlHttp.getResponseHeader("time elapsed");
alert(sResp);
}
}
}


2. Proxy BS

This Proxy BS written in eScript, handles the invokation of the destination Business Service, and creation of the custom HTTP headers. Once this piece of eScript finishes, it returns to the onreadystatechange handler in the code above.

var oBSWeb = TheApplication().GetService("Web Engine HTTP TXN");
oBSWeb.InvokeMethod("GetAllRequestParameters", Inputs, Outputs);

//Get parameters from querystring
var sBSToCall = Outputs.GetProperty("ParamBS");
var sMethToInv = Outputs.GetProperty("ParamCallMETH");          

//Call your BS
//processing time here....
var sElaspedTime="1000";

//set response headers for the browser to read
Inputs.SetProperty("time elapsed",sElaspedTime);
oBSWeb.InvokeMethod("SetResponseHeaders",Inputs,Outputs);


The example above just uses an alert, but an imaginative developer can implement a sexier dialog, or invoke a refresh on the current view

Summary

I've demonstrated usage of the SWE API, but Siebel provides a similar more supportable architecture in the form of an EAI Object manager, making the SOAP interface a more suitable fit.

This is no simple piece of configuration, and it is not recommended that you attempt it without the consultation of your local Siebel Architect. The intention of this article, is to illustrate how Siebel and browser technologies can come together to achieve a solution that was written off as impossible.

This concept was POC'd for client who needed real time notifications for bulk processing, it allowed the user to continue on, and get a fancy notification once the process was completed.

To satisfy our Funky town series, we have to ask ourselves

1. Will it perform well?
Yes, code can be distributed in parallel, and run in the background

2. Is it maintainable?
Yes, if the code is put into a framework, and maintained by an experienced professional

3. Can it be upgraded?
Yes, the solution can be built using a supported architecture, and abstracted through a business services. and

4. Is it sexy?
Yes, its what Siebel Open UI promises to be, except Open UI dosnt do server side callbacks! (Confirmed with Siebel in 4th Qtr 2011)

EAI Phone Format Challenge feat. JL Engine

BRP

Business Rule Processor (BRP) is a diamond in the configurators toolkit, and it should be at the forefront of Oracle's marketing efforts to license more copies of Siebel. If you've never heard of BRP, then you've never lived Siebel, just imagine an Application that is fast, which can be complex and flexible, and at the same time being virtually script free.

Amazing as it is, it has a few flaws, rules cannot be validated, there are UI bugs that make development annoying, rules dont synchronise across multiple OMs, procedures don't support parameters, but the most obvious flaw is the debugging capabilities, every thing is just dumped to the OM logs, no workflow instance monitor, or TBUI like debugger around here, just old fashion log files.

However the design approach of BRP is apparent, the first release of BRP was meant to deliver a fast and stable tool, anything else was second priority, but there is a question that hangs around the future of BRP, with OPA being crowned the rules engine of choice within Oracle. There seems to be a gap in the strategy, OPA is a policy determination engine, while BRP is an eScript replacement, one is business focused, while the other is technically focused, two different things in my opinion.

So what if you dont have BRP on your project? I was engaged by a customer to design solutions to a common problem, getting rid of a sea of code and improving maintainability. All the standard Siebel declarative and non declarative options were considered and weighed up against another. One of the key components of this redesign was a rules engine, and BRP was the primary choice, but due to various factors, including whether BRP could be sourced. I was engaged to build a BRP replacement: a custom rules engine that overcomes some of BRP's limitations that is declarative and fully customisable.

JL Engine

For the sake of identity, the rules engine is conveniently referred to as JL Engine.

Imagine that you could extend the native Siebel Query Language Expressions, and create your own functions, that can be accessed declaratively. This is a powerful feature that allows developers to bypass many of the complexities around implementing rules using the traditional Siebel expressions, and potentially replace a large amount of scripting.

What kind of complexities do you ask? To fire your imagination, how about we solve the EAI Scriptless Phone challenge using this new engine? Previously I demonstrated the power of BRP, and showed how BRP can be used to solve this challenge. It required 15 expressions to complete the task

If you had control over your expressions, you could design convienent utitlities to more effectively solve this challenge, and also re-use these utilities across the application, or you could cheat and design a fancy expression that solves the EAI phone challenge in 1 step, but in consideration of good application design, we are going to design components which are re usable, hence this new rules engine solves the challenge in 8 steps.

Here are some of the new exotic expressions that will aid us in this Challenge

ApplyMask(Mask,MaskCharacter,Value)
//Usage: ApplyMask("$XX.XX","X","1234") = $12.34

AscFromChar(AsciiCode)
//Usage: AscFromChar("65") = A

GetArrayPart(ArrayString,Delimiter,ArrayIndex)
//Usage: GetArrayPart("A,B,C",",","1") = B

CountChar(Value,CharToCount)
//Usage: CountChar("$XX.XX","X") = 4

Solving the EAI Challenge with JL Engine

1. Getting the LF character

This was the original stumbling block to solve this challenge when it was first announced. I presented a solution using XSLT. Impossible Siebel reader Dos, then chimed in with other 2 solutions.

BS: SSSE Address Parser
Method: GetCRLF

This business service returns a CRLF, which is actually two characters chr(10) + chr(13), so the return string has to be post processed to return only chr(10)

SetProfileAttr("LF","\n")

Using SetProfileAttr to set \n, and retrieving it using GetProfileAttr is elegant, but requires two steps. Using our JL Engine expression below allows us to complete this in one step

AscFromChar("10")


2. Separating the Phone and Format from the EAI Phone String

Once we have the LF character, string expressions can be used get our string. In BRP this is done using two expressions Instr is used to find the position of the LF character, then Left/Right is used to chop the string.

"+44123456789LF(00) 00 000 000"

Using our JL Engine expression below allows us to complete this in one step. GetArrayPart splits the string, and returns the desired index.
GetArrayPart("+44123456789LF(00) 00 000 000" ,"LF","0")

3. Applying the Format Mask to the Phone Value

To make things harder, the phone value has the international prefix, which is additional to the number and should not have the format applied to it. BRP and the original XSLT solution solved this by using two cursors, one to track the position of a Mask character, and the other to track the position of phone digit that corresponds to this mask characters, and involved looping and recursion. Using our JL Engine expression below allows us to complete this in one step (three nested expressions)
CountChar("(00) 00 000 000","0")
Counts the number of format characters, which we then use to perform a Left/Right on the phone string to split the prefix and phone Once we have the phone without the international prefix, we use the following Expression to get the formatted number.

ApplyMask("(00) 00 000 000","0","123456789")
The joined expression looks like this

ApplyMask("(00) 00 000 000","0",Right("+44123456789",CountChar("(00) 00 000 000","0")) )
The final step is to concat the Prefix with the result from the above step to get our result

JL Engine solution step by step

The screen shots below provide the step by step detail, of how the solution is physically implemented in the JL Engine.

 1. Create a rule and define all the properties



 2. Create a Main procedure, and create the 8 steps as shown, thats all.



 3. Test the Rule in Business service simulator, or if you are a lucky customer that has SiebUnits on your project, you could write a declarative test case, to test your declarative rules engine.



The test results applet shows the correct output value for our challenge, with the test case passing in 32ms.

Conclusion

This new custom rules engine passes the EAI Phone challenge with flying colors, and manages to do it more nimbly, with less steps of declaration, thanks to its custom inbuilt functions, but is this enough to make it a superior solution than BRP? Both engines are built for different purposes, but share a lot of similiarities.

BRP is a speed demon, and was built for handling complex calculations, which just happened to come with a declarative engine that allows a project to banish casual scripting, but sadly it is not a universal module, and looks likely to be sidelined from future updates.

JL Engine was also designed to remove scripting, comes built with connectors to a custom application framework, it achieves speed by taking shortcuts and also provides a declarative interface for developers, but best of all it is openly extensible, allowing developers to build upon the gaps of BRP with custom requirements.

Want a real time instance monitor that shows step by step execution, condition determination and variable assignment. You got it, JL Engine comes with a rule audit history view to do just this. Want a visulizer that shows the rule composition in a graphical UI? No problem, JL Engine comes with this as well. Want a expression builder that allows developers to build rules faster? Yes Sir, we have that covered.

So how about some performance figures? In the next episode of the EAI Scriptless phone challenge. I will compare all the declarative solutions to this challenge, and provide performance metrics to determine which is the fastest and the best, so be prepared to be blown away.

1. XSLT using Xalan Engine
2. XSLT using Saxon Engine
3. BRP Engine
4. JL Engine

Note 1: While other solutions exist that simply strip away the phone format mask, the above script less solutions apply the format to the phone and have the flexibility to deal with the international prefix. 

Note 2: The JL Engine is not available commercially. The product has a limited audience, and is only available to selected customers.

SiebUnit - Build Inventory

This article is a continuation on SiebUnit - an xUnit implementation, and provides an inventory of the parts needed to construct a SiebUnit framework.

Like the first article, this is aimed at the Technical Architect, who understands the concept, and knows how to put the necessary pieces together to bring this idea to life.


Architecture



The architecture diagram allows us to visualize important concepts at a high level, and can be used to break up big jobs into more manageable pieces.


Rules Engine

You can replace Rules Engine with any Siebel interface that supports receiving Inputs and returning Output PropertySets. SiebUnits will also work against a standard invoked business service, or workflow.

The only caveat is that you cannot have context, that is, you must re-design your BS/WFs to decouple your logic from any specific Application context. This naturally suits the testing of a rules engine, external API such as ESB, internal APIs like custom business object libraries, and scripting utilities.


SiebUnit Framework

A SiebUnit framework can be composed of the following sub components

* Test Case Runner
* Test Case start up routines
* Test Case tear down routines (optional)
* Test case assertion
* Random data generator
* Expression language
* Expression language parser
* Expression Executor
* Expression Builder
* Test history and performance timing
* Script Library
* Business Object (BO) Library
* XML Parser
* SQL/IO Layer


Test Case Runner

This component reads the configuration data from the data store, loads all the test cases in memory, and runs each test case one by one. It records start, stop times, and logs the output, but does not make any determination on the test case result. Errors are captured, and any exceptions are suppressed, allowing the Test Case Runner to continue on until it completes all test cases.


Test Case Startup/TearDown

In order to test certain scenarios, prerequisite test data may need to be setup before the rules can be tested. This component constructs the initial test data before the test suite runs. Optionally, there can be a Tear Down component, which cleans up after all test cases.


Test Case Assertion

Assert is a computer science term used to ensure a condition is true/false to enforces application integrity. The assertion component performs the same role, it checks the outputs of each test case, and validates it against the expected result for that test case, logging the result to a history data store.


XML Parser

This component converts XML into a structured object and back. This can be used to send dynamic outbound messages, and validate inbound responses, for integration testing.

SQL/IO Layer

This component retrieves the related data from an entity for validation purposes. It is used by the Test Case Assertion component.


Random Data Generator

Even within predefined test boundaries, we can allow the generation of random data to simulate dynamic conditions.

Eg. Generate a bunch of expense activities of random type, and amount, and test it against a business rule for assignment or approval logic.

The random data generator can be used by the Setup component, or the individual test case to randomize inputs into a rule.


Expression Language

To make SiebUnits declarative, we need a way of allowing the developer to construct data, supply dynamic inputs, and validate the results of the test cases, without writing a line of code. This can be done via an Expression Language, similar to calculated field expressions in Siebel.


Expression Language Parser

This component is responsible for translating the above expressions into a form which is understandable by the Siebel eScript compiler or external engine, that will in-turn run the instructions.

The expression parser needs to differentiate between a literal and expression. This is done via 4 ways, as illustrated throughout Siebel.

1. Field Pre/Post Default

Expressions need to be prefixed with "Expr: " or a key word such as "System", "Field" etc. Anything else is considered a literal.

If you wanted to pre default another field via the square bracket notation "[Id]" this will actually result in a literal value

2. Calculated Field

A calculation is always expected to be an expression, unless it is surrounded by quotes. So the square bracket notation "[Id]" is treated as an expression in this context.

3. Field Type

This example can be seen in WFs in the Output argument of some steps. There is another field that indicates the type. Eg: Literal vs Expression.

4. Dot Notation

WFs also have another interesting expression, with the dot notation to reference child elements within property sets. This dosnt really fit into either of the above 3 categories. As it can easily be interpreted as a literal, and does not have any prefixes to indicate that it is an expression.

You have to make a call on which method you will use, and your expression parser has to recognise the above patterns if you mix the scenarios.

Eg 1. For an eScript implementation, the expression

"GetProfileAttr('Me.Id')"

Should translate this to an eScript call like this

TheApplication().GetProfileAttr("Me.Id")

The parser has to recognise that this is an expression with a literal argument.


Eg 2. The next expression provides a dynamic input.

"GetProfileAttr([&Property])"

This follows Siebels WF standard, of referencing WF properties via the ampersand prefix inside square bracket notation, requires a bit more work because you have to deal with the nesting of expressions.

This is where you can draw the line, and define styles for your expressions and have another expression to deal with that scenario

"GetProfileAttrByProperty('Property')"

Now the literal references a property that holds the value to this method. You will have to plan out your expression language in advance, and know how your parser will deal with each scenario.


Expression Executor

After you've translated the expression into the native language, it needs to be executed, and the result returned. The method that you execute the expression, will impact on the style of your expressions, and how complex your parser needs to be.

Consider implementing a short circuiting mechanism so that, if part of a condition expression is met, you can abort executing the rest of the expression.


Expression Builder

This component allows the developer to choose from a defined set of expressions, this aids in learning, reduces mistakes, and as well as documenting all the supported functions of SiebUnit. Ideally It has to integrate with the Siebel UI session, and apply the expression to the list/control.


Test History and Performance timing

This data is produced by the Assert component which validates the test case runners run history with the expected outputs, it also extracts performance timings for analysis.

Inputs and Outputs of the run are also included in the run history to help debugging. Having the results in the UI is consistent of the expectations of a declarative tool. Take note BRP engineers!

Test history and performance data can be transient, and the user can export the results, if a base line needs to be established.

Script Library and BO Library

It is important to differentiate between the above two components. A script library provides utilities, and prototypes on native objects, while a BO library provides business functions.

A SiebUnit library is tightly integrated with the script library for performing internal operations, and provides proxy functions for invoking the BO library to setup and perform test cases. When creating framework functions you will have to make conscious decisions in which library a function belongs to.


Unit Testing and Regression Testing

SiebUnit does not replace commercial regression testing suites. An important differentiation is that SiebUnits is designed to be used by developers for unit testing code/WFs before it is checked in, and to ensure application integrity is maintained before it reaches the testers.

SiebUnits are also more thorough, as only the developer could understand all the exception paths of a program. SiebUnits test technical functionality, while regression test tools test functionality.

Although SiebUnit can be used to regression test business rules, it does not test UI components. The developer is still expected to ensure that UI rules are unit tested. A badly configured link, can still bring the system down and while writing SiebUnits to create data, and check the data returned from the link could be a valid test case. It was not intended to micro manage every piece of individual configuration.

SiebUnits can offer a great sense of security for the developer, but poorly designed test cases, can also provide a false sense of security. This is normally due to poor test case coverage. Average developers will usually test the positive scenario, good developers test both positive and negative scenarios. SiebUnits require the developer to understand all the possible logic paths and design test cases to thoroughly cover off the scenarios that can result the program to fail.

Conclusion

There are numerous benefits to having SiebUnits on a project. The immediate return on investment, is that your testing period is reduced, and your application will be more robust, but the real payoff of really comes down the track, when you build highly complex systems, where one small change can break another part of the system. SiebUnits can provide your developers with confidence to make that change, without costly regression testing. SiebUnits will keep your Application, and also your sanity in check.

xUnits is a proven concept, and is a staple of any modern development platform. SiebUnit provides an evolution path, so that your developers no longer need to manually unit test. The world has JUnit , jsUnit, CUnit, OCUnit, PHPUnit, PyUnit, SUnit and now we can add SiebUnit to the list.

SiebUnit - an xUnit implementation

Introduction

Over a year ago, I presented a concept of xUnits in Siebel and how they could be used. Because it was only a concept, it was missing implementation details, so there was some doubts to whether Test driven development would work in Siebel, where code is meant to be restricted to fringe cases.

At that time, I ported some code from a back end Java project that I was working on, to automate Siebel interface verification. The idea was to pickup XML templates from a file system, insert randomize data, send it to the ESB, and validate the results.

It was used for several releases, reducing a days worth of effort to verify 50 interface interactions, to about 5 minutes. With the JUnits in place, they could be used to verify an integrated environment instantly on demand, instead of employing highly paid professionals to perform repetitive regression testing.

The tool was written in Java, didnt have a UI, it had access to the eApps SSO hashes, which meant that an evil developer could use it to log onto a server environment, and impersonate anyone they wanted. It was more of a regression testing tool than a unit test tool, and being written in Java, it was inaccessible to most of the Siebel Team.

In the spirit of a true Siebel developer, I thought a real implementation of xUnits in Siebel had to be scriptless.

So "SiebUnit" was designed to be just that, a fully declarative unit testing framework for Siebel developers.

This framework also happens to plug into a rules engine like BRP, OPA, or it could be run against a collection of business services/workflows, as long as you can pass in an Input and get an Output property set, the "SiebUnit" framework handles the rest.

This idea was sold to a Client, who immediately picked it up as a way to get developers to unit test, improve build quality, and reduce dependence on testers, and provide a structured development process.

This article and its followup, takes a step further, providing architectural and implementation concepts. It is aimed at the Technical Architect, who can grasp high level concepts, fill in the low level implementation, possibly innovate on this idea, and introduce SiebUnits into your Siebel project.


What is SiebUnit

If you had to sell the concept of xUnits in Siebel, this slide should sum it up.



This screenshot shows a SiebUnit implementation that was built for a Client.



Features

Notification System for test results

Uses a custom notification system, written in a client side library, to add persistent messages to the current view. The messages can be dismissed, by clicking the X button.


Design and Run Test Cases from the UI

Test Cases are configured, executed and debugged all within a standard Siebel UI. Debugging the rules itself, depend on the capability of the rules engine. Eg. BRP can only be debugged via log files, WFs can be debugged using the instance monitor.


Export/Import test Cases

Allow easy export and import of hierarchical test configuration data between different environments.


Trace Toggling

Toggles developer tracing, for performance testing of rules


Dynamic test data

Uses XML templates, and proxy functions to the BO Library to generate unique test data. Random data can be generated and provided on inputs, or overridden in the case of XML templates.


Expression Builder

A helper utility designed for the developer to construct the test cases declaratively. This is built using HTML, which integrates with the Siebel session.


Other

SiebUnits comes with pre built internal test cases to validate your eScript engine, and profiles ST objects vs non ST objects.

SiebUnits can be configured to run recursively to load test an environment


Case Study - Custom Rules Engine

SiebUnits was used on a project to ensure the integrity of a Rules Engine.

The design of the custom rules engine started off in small components, followed by unit testing, further build and some more unit testing. The components were then integrated, followed by another round of unit testing.

As the design got exponentially more complex, the integration of all these components, made it impractical to regression test all the exception paths, after every change.

SiebUnits was originally built with integration to BRP. So for this project, it was modified to work with this new engine.

A collection of SiebUnits was built to test the following aspects.

Declarative IF conditions
Declarative loops
Declarative procedures
Expression Translation
Expression Results
Business Service Invocations
Property Assignment
Cancel Propagation
Error Messages
Math Operations
Rule inheritance
PropertySet Interactions


Some further internal test cases were also designed to validate the behaviour of the eScript engine. This ensures that if Siebel changes the eScript engine, these internal SiebUnits will pick up on the expected outputs of dependent core functions, and allow the root cause to be addressed.

Once the SiebUnits were designed, and configured, they were used to repeatedly validate the Rules Engine after every change, to ensure previously built functionality still functioned. Changes often broke other unexpected parts of the engine, due to the tight integration and dependency between components, but with the SiebUnits in place, the report of failed test cases helped to quickly pinpoint the cause.

Towards the end of the build, a critical defect was discovered with the eScript engine, that corrupts a core object of the rules engine. This happens during random runs of the test suite, but it was easily reproducible using the load runs.

The problem required a re-design of the core object, which unfortunately touches every function in the engine. Since the new object had a different interface to the old object, it required a re-design of most of the interactions around this object as well.

The affected parts of the program was identified, and a fix was implemented in two hours. The SiebUnits test suite was run afterwards to ensure that the rules engine passed all the test cases. This provided a QA stamp on the product, and was similiar to having a team of regression testers on hand to validate your work at will.

The collection of SiebUnits also allowed load testing of the engine in a real environment. During these load runs, server resources can be monitored to ensure that they run within acceptable standards and allow capacity planning based on expected usage.

It would have been much more difficult if not impossible, to complete such a complex build without SiebUnits in place, and the regression testing and maintenance effort would have been an ongoing burden.


Intermission

Do you have BRP, OPA or a collection of Business Services that you would like to keep in check? The next article will look at the architecture of SiebUnits, and all the components that need to come together to form the framework of SiebUnit.

Note: The term Rules Engine is used broadly in this article. Even though BRP stands for Business Rule Processor, it is actually a declarative eScript replacement engine. OPA is more suited to the term for a Rules Processor, as it takes a bunch of inputs, makes a determination and returns the result. However the Rules Engine interface of Inputs/Outputs is what makes it suited to SiebUnits.

View the next article here

EAI Phone Format Challenge feat. BRP

Over two years ago, I issued a challenge to the Siebel community, through the EAI Scriptless phone number challenge.

The original challenge was started by a reader from Siebel Essentials's scriptless challenge, which was to find a scriptless solution to strip off the formatting from a DTYPE_PHONE, and leave the raw phone number.

The challenge that I subsequently issued to the community was to find go further and find a solution which applies the format mask to the phone number without any scripting.

At the conclusion of the challenge, I presented a 35 line recursive XSLT template that demonstrated that while it is possible to go to the end of the earth to find a scriptless solution, a 10 liner business service would have between more maintainable, given that most Siebel professionals don't spend their time swimming in stylesheets.

I know that our conservative Siebel readership wont sleep well with the scripted solution, so I've been working hard on an alternate solution that will please all our declarative readers out there, and also impress the eScript Jedi's and hopefully convert them back to the light side.

Here is the challenge outlined again.

"+610212345678[LF](00) 0000 0000"

The system must take the above DTYPE_PHONE value and convert it to (02) 1234 5678 without any scripting.

Note that the DTYPE_PHONE consists of the raw number + a line feed character + a format mask with the zeros representing the position of the number, any non zero characters represent a formatting character that must also be applied to the formatted string.

[Business Rules Processor]

This requirement can be achieved by using Business Rules Processsor (BRP).

BRP was introduced to Siebel as a FINS module around v7.8. It provides a fully declarative means of processing complex business logic.

This is an except from bookshelf about the capabilities of the BRP engine.

"The Business Rule Processor features include:

* Business logic administration through the application user interface.

* Appropriate for complex business logic; supports procedure, loops, if-then-else and switch-case constructs.

* Appropriate for business logic that changes frequently. You can modify business logic without deploying a new SRF.

* Query, read, and write access to business components.

* Error handling.

* Support logging at multiple levels for easy testing and debugging.

* Can potentially replace large amounts of custom scripts."

That list is pretty impressive, but is it up to our challenge?

Here is the psuedo code for rule that we are going to implement.

1. Separate the DTYPE_PHONE into separate PHONE and FORMAT components

2. Start at the last character of the FORMAT component

3. If the character is a Zero, then take the corresponding character in the PHONE component and store it in RESULT

4. If the character is non-Zero then take the current FORMAT character and store it in RESULT

5. Move to the next preceding character and continue processing until no FORMAT characters are left

[Solution]

1. Goto the Business Rule Processor Screen

a. Create a new process called "EAI Phone Format"
b. Create the variables for our rule as shown in the screenshot below


2. Goto the Procedure view

a. Create a procedure called "DeterminePhoneString"
b. Create a procedure called "Main" and set this as the Entry procedure into this rule.
c. Create 4 steps as shown


3. 1st Step - Get CRLF character

Here we call the SSE Address Parser BS to get our CRLF character and assign it to the sCRLFChar property.


4. 2nd Step - Init variables

We initialise and setup all the variables before we start our routine.


5. 3rd Step - Loop through pattern

We define a loop that iterates through the pattern backwards, until our cursor reaches the beginning. On each iteration, it will call "DeterminePhoneString".


6. 4th Step - Set remainder of the string to the final output (Optional)



7. Define the "DeterminePhoneString" procedure

Create a switch statement with the following logic


8. Create a default branch for our switch statement



9. Activate your rule, and test it in BS simulator using "FINS CAP Processor Service"



[Conclusion]

I've only scratched the surface of BRPs features here with this example, but amongst some of the more interesting features, BRP introduces a new property type called 'Vectors" which provides quick access to a hash table like reference, which is populated directly from a BC.

Is BRP the holy grail of declarative configuration?, In my opinion no. When used badly, the problems associated with scripting will only be shifted to a declarative interface. As with scripting, BRP can be the source of duplication and spaghetti procedures.

BRP is also limited by its expressions, and does not support custom functions, so you will still need script to provide a wholistic solution. Eg. Regex is not supported, so performing email validation still requires you to call a business service.

Scripting can be used for good, but too often it falls in the hands of in-experienced developers, which is why I really like BRP. It can be picked up really quickly by beginners, as it utilises the Siebel Query Language, and provides the flexibility of eScript through declarative loops, if/else/switch conditional statements and procedures. It also forces developers to break down their routines to manageble statements that improve readability of the design.

Where BRP really shines, is the capability it provides to developers to implement the majority of requirements with programming constructs but in a declarative interface, and more importantly since the logic is outside the SRF, changes to business logic can be made outside of releases.

Siebel MTOM Attachments

Audience

This article highlights the importance of MTOM in the enterprise, and provides a high level overview of the challenges that I faced bringing the MTOM standard into Siebel. Due to the proprietary nature of the environment, the source code for the implementation cannot be provided, but it is hoped that this article will provide a strategy for system designers/architects out there wishing to achieve the same goal and bring their project forward into the 21st century.

Background

As we slowly move towards the paperless society, handling attachments becomes a crucial part of the Enterprise Messaging landscape. With the popularity of Web service SOA architectures, SOA professionals found themselves with a tricky challenge. How to send binary data in XML?

XML is a textual data format, it has to be well formed and contain legal characters. If you take a binary string and put it between a set of tags, certain characters of the binary string may prematurely escape the tag, or make the XML invalid. In the early days SOA professionals got around this problem by encoding the binary string into Base64 format, which allows binary data to be represented in ACSII format. Although this does solve the problem of transmitting binary using XML, it is not very efficient. A Base64 string is 33% larger than the raw binary, which means more resources are need to handle the larger message, as well as taking longer to transmit.

The next evolution came when SOA professionals realized they can use MIME to send attachments, and thus SOAP with Attachments (SwA) was invented. In short its just an XML message with the binary data tacked at the end of the payload. The attachment is distinctly separate from the payload, which means the message processor has to implement logic to get the URI and retrieve the attachment, another downside is that SwA does not support Web Service Security (WS-S).

MTOM is the next advancement in attachment messaging, which combines the best of SwA and XML with inline attachments. MTOM is based on a slightly modified version SwA, which utilizes an XML Optimized Packaging (XOP) command to logically include the attachment as part of the message, instead of just referencing to the location of the attachment, it also supports WS-S.

That was 2004, today is 2011.

MTOM and Siebel

Siebel does not support MTOM, or even MIME through WS, the only supported method of sending attachments is through Base64 encoding. Base64 works well for small attachments, and is universally accepted, but what happens when Siebel is required to handle large attachments? There is little choice, it is Base64, MIME (non WS), or hire a technical architect that will design a custom MTOM framework for you.

Creating MTOM Messages

There are several approaches to creating MTOM messages in Siebel, here is a high level run down of the options.

Option A

1. Query IO and get a SiebelMessage with Base64 encoded attachments
2. Pass SiebelMessage to JBS, to construct a MTOM message and create the XML payload
3. JBS converts Base64 to Binary stream and appends to the above MTOM message

Adv: Easy to implement, No file management
Con: The Attachment will be converted from Binary to Base64, and back to Binary each way


Option B

1. Query IO and Get a SiebelMessage with Base64 encoded attachments
2. Convert SiebelMessage to MIME Hierarchy in Siebel
3. Parse MIME binary chunk to JBS
4. JBS reconstructs the MIME to MTOM structure, and inserts the proper XML payload

Adv: No file management, good fun for Java programmer
Con: Difficult to implement for a Siebel developer,
The Attachment will be converted from Binary to Base64, and back to Binary each way


Option C

1. Extract file attachments to filesystem
2. Pass file locations to the JBS
3. JBS reads file into memory, creates MTOM message and XML Payload
4. Delete extracted files

Adv: Easy to implement, Attachment is never converted to Base64.
Con: File management. Increased disk IO.


MTOM messages can be created using standard MIME classes, since MTOM is packaged as a multi part MIME message. After the MTOM message has been created, it can be passed back to Siebel or sent to the transport directly from the JBS.

The main theme in these options, is the usage of Java Business Services (JBS). eScript does not have any methods to retrieve binary from a PropertySet, hence it needs to be passed down to a JBS where it can be retrieved, and manipulated.


Importance of MTOM

Dealing with attachments is Business As Usual for many industries. For example, record keeping is a core part of every Government departments DNA, it serves as a means for accountability, and preserves memory for historical purposes. With the proliferation of broadband, hospitals can send X-rays and other medical documents to roaming doctors. Insurance companies can process claims through PDF and automatically store them in the document management system. Integrating with document management systems becomes one of the key programmes of work in the Enterprise, and handling attachments efficiently becomes a key non functional requirement.

The world has moved to MTOM, and for Siebel to remain relevant it must keep apace with new technologies, otherwise it risks becoming a legacy system. There is no immediate risk for Enterprises that deal with small file sizes, as Base64 is still an acceptable technology, as long as your ESB still supports it, but it will not scale well into the future, especially with High Res/HD content becoming mainstream on the horizon.

The problem arises when your Enterprise makes the decision to embrace MTOM, and when the Enterprise architects are deciding which projects will provide the file attachment functionality.

The question they will be asking is your Siebel project MTOM capable and ready?

Searching through UTF-16 files

In a typical day of a Siebel EAI developer, we are required to investigate problems with XML/XSLT files, and unless you know the system inside out, and outside in, you may encounter XML fragments that look unfamiliar.

For example, your ESB team sends you an XML that they claim to have been sent from your system, but from looking at the XML you dont recognise the interface that sent it. However you reason that, if you could search through all the XSLT files in Siebel you could track down the interface that could be generating this file.

The most obvious tool at the developers disposal is Windows Find, this is the defacto method of searching for files or text in files in a Windows OS, but beware because this method will not yield any results when searching through UTF-16 files. Surely Unix users would have more reliable tools in their arsenal. What about Grep?

Grep belongs to the Unix family of tools who's sole purpose is for searching, so how does this command fair with UTF-16 files? Fire up any Grep tool within reach and search for "xml" within your directory of UTF-16 XSLT files.

Here were my results.





FAILPASS
Windows BareGrepX
Mac GrepX
HP Unix GrepX


To understand this behaviour, open a UTF-16 file in a binary editor, you'll see that there is a "00" byte (shown as a space character) between every character, this is how UTF-16 files are physically represented in the file system, but open it in Notepad or your favourite unicode aware editor, and it will magically render the UTF-16 code points into your familiar readable XML.

With that understanding, it becomes clear why text matches will not work in programs that use a non UTF-16 regex engine.

This is a problem for any unsuspecting EAI developer and Siebel project that relies heavily on XSLT for its request/response Integration, but knowing thy enemy is half the battle already won, so here are a few tools to deal with UTF-16 files.

1. Windows findstr
eg. findstr /s /c:"siebelmessage" *.xslt

This command can match strings in UTF-16, and can be used to search directories.

2. Notepad++
3. TextPad

The above two text editors, could search entire directories, and match for text within UTF-16 files.

With this in mind, I'm sure you could easily find other utilities and programs that can scour through UTF-16 files.

It is a little unsettling to discover that you cannot rely on such ubiquitous tools such as Find or Grep for your Integration needs, but hopefully the next time you get that foreign XML in your inbox, you'll already have a hand on a nearby UTF-16 scraper.

XSLT 1.0 upper-case() & lower-case()

In the last article, we looked at using XSLT "includes", and along the way, encountered a requirement to convert a string to uppercase in XSLT 1.0.

XSLT 2.0 users have the luxury of upper-case() & lower-case(), but oddly, functions for converting characters to upper and lowercase are not available in XSLT 1.0.

XSLT isnt the mother language of most Siebel professionals, or if you dont use it everyday, youre going to get rusty (i know i am), so this walkthrough should appeal to a lot of readers.

Lets take a look at our XSLT functions file from last time
<!--XSLTfunctions.xslt-->
<?xml version="1.0" encoding="UTF-16"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:variable name="smallcase" select="'abcdefghijklmnopqrstuvwxyz'" /> 
    <xsl:variable name="uppercase" select="'ABCDEFGHIJKLMNOPQRSTUVWXYZ'" /> 
    <xsl:template name="FORMATBOOLEAN">
        <xsl:param name="BooleanChar"/>
        <xsl:choose>             
            <xsl:when test="translate($BooleanChar, $smallcase, $uppercase)='Y'">true</xsl:when>
            <xsl:when test="translate($BooleanChar, $smallcase, $uppercase)='YES'">true</xsl:when>
            <xsl:when test="translate($BooleanChar, $smallcase, $uppercase)='TRUE'">true</xsl:when>
            <xsl:otherwise>false</xsl:otherwise>
        </xsl:choose>
    </xsl:template>
</xsl:stylesheet>


It works by taking a string value, converting it to uppercase and comparing it to 3 static values, but with a little bit of effort, we can optimise it to be more efficient. Instead of calling translate on the same variable 3 times to get the same result. We will engineer it so its only called once.

We start by creating our own custom UPPER and LOWER functions
    <xsl:template name="UPPER">
        <xsl:param name="text"/>
        <xsl:value-of select="translate($text, $smallcase, $uppercase)"/>
    </xsl:template>
    
    <xsl:template name="LOWER">
        <xsl:param name="text"/>
        <xsl:value-of select="translate($text, $uppercase, $smallcase)"/>
    </xsl:template>   


In XSLT we cannot reassign the value of a variable once it is declared, so we create a new variable called $BooleanCharUPPER,and call the UPPER function from inside our FORMATBOOLEAN function and return the result.
    <xsl:template name="FORMATBOOLEAN">
        <xsl:param name="BooleanChar"/>

        <xsl:variable name="BooleanCharUPPER">
            <xsl:call-template name="UPPER">
                <xsl:with-param name="text">
                    <xsl:value-of select="$BooleanChar"/>
                </xsl:with-param>
            </xsl:call-template>
        </xsl:variable>  
  
    </xsl:template>


Finally we evaluate the $BooleanCharUPPER against our static values.
        <xsl:choose>             
            <xsl:when test="$BooleanCharUPPER='Y'">true</xsl:when>
            <xsl:when test="$BooleanCharUPPER='YES'">true</xsl:when>
            <xsl:when test="$BooleanCharUPPER='TRUE'">true</xsl:when>
            <xsl:otherwise>false</xsl:otherwise>
        </xsl:choose>  


We can now use these functions in any XSLT, where we've included them
        <xsl:call-template name="UPPER">
            <xsl:with-param name="text">
                <xsl:text>bad boys blue</xsl:text> 
            </xsl:with-param>
        </xsl:call-template>

Result: BAD BOYS BLUE

        <xsl:call-template name="LOWER">
            <xsl:with-param name="text">
                <xsl:text>BAD BOYS BLUE</xsl:text>
            </xsl:with-param>
        </xsl:call-template>

Result: bad boys blue

Here is the final XSLTFunctions.xsl file.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  
    <xsl:variable name="smallcase" select="'abcdefghijklmnopqrstuvwxyz'" /> 
    <xsl:variable name="uppercase" select="'ABCDEFGHIJKLMNOPQRSTUVWXYZ'" /> 
    
    <xsl:template name="UPPER">
        <xsl:param name="text"/>
        <xsl:value-of select="translate($text, $smallcase, $uppercase)"/>
    </xsl:template>
    
    <xsl:template name="LOWER">
        <xsl:param name="text"/>
        <xsl:value-of select="translate($text, $uppercase, $smallcase)"/>
    </xsl:template>   
    
    <xsl:template name="FORMATBOOLEAN">
        <xsl:param name="BooleanChar"/>
        <xsl:variable name="BooleanCharUPPER">
            <xsl:call-template name="UPPER">
                <xsl:with-param name="text">
                    <xsl:value-of select="$BooleanChar"/>
                </xsl:with-param>
            </xsl:call-template>
        </xsl:variable>        
        <xsl:choose>             
            <xsl:when test="$BooleanCharUPPER='Y'">true</xsl:when>
            <xsl:when test="$BooleanCharUPPER='YES'">true</xsl:when>
            <xsl:when test="$BooleanCharUPPER='TRUE'">true</xsl:when>
            <xsl:otherwise>false</xsl:otherwise>
        </xsl:choose>        
    </xsl:template>
   
</xsl:stylesheet>


Bad Boys Blue, are a eurodance group that came to fame around the same era as Modern Talking and Sandra back in the 80s, and like many of their contemporaries, their music evolved from the more traditional europop to a more upbeat eurodance style. Its hard to pick a favourite BBB track, but check out "You're a woman, i'm a man" for a little taste of their magic.