Underestimating the Underdog - Siebel 16 & The Steve Watson Story



As part of our syndication arrangements with the Siebel Hub team, the following article has been kindly been allowed to be published here. Since a large portion of the Impossible Siebel readers are Americans, this Siebel event should have particular relevance.



As we all know Siebel used to be the talk of the town, the cock of the walk, the top dog in CRM. Sadly, that is no longer the case. Siebel is now the underdog. But this dog can still hunt!

Certainly Oracle is doing its part. The success of Open UI has sparked a Siebel Renaissance. Now Oracle is telling certain customers they can expect Release 16 in April.

To mark this achievement, along with Eagle Creek, the Siebel Hub has put together a very special event for April 13th. It will feature an update on the state of Siebel, a briefing on Release 16, and a special guest former Denver Bronco Steve Watson. Because you can’t spend too much time with world class people!

Like Siebel, many people were tempted to count Steve out during his football career. He was cut from his junior high team. No major college recruited him. He was not drafted by the NFL. But he persevered to become the league’s leading receiver.

It should be a great program; but it would be even better if you came.

Where:

Great Northern Tavern
8101 E Belleview Ave, Suite E
Denver, CO 80237

When:
Wednesday April 13th, 2016 5:00 P.M. - 8:00 P.M.
Or
http://info.eaglecrk.com/2016-CO-Siebel-Event-in-Denver-April_CO-Siebel-Event.html

Please feel free to extend this invitation to anyone who might be interested in attending.

Asynchronous Workflow With Real User Login


Background


A common requirement amongst many Siebel customers, is to have the ability to run an asynchronous workflow under the real user's login.

This issue is highlighted in the following Oracle support web document, and is representative of the problems faced by many customers.

NOTE:494815.1 Setting CREATED_BY to the user that invoked the workflow process asynchronously

The customer stated that they have an asynchronous workflow, and wanted to know if it was possible to set the creator login of a server request. The solution offered by the support personnel was to run the work flow in synchronous mode.

Asynchronous workflows are used in Siebel to offload work from the user session to perform background processing, however that leaves an undesired effect of stamping the record with SADMIN. This behavior obscures information about the creator, updater, and prevents audit trail from being utilized.

Workflows can be run synchronously as suggested in the Oracle support web document above, but it has the rather unwanted effect of blocking the UI.

Fortunately, for customers who can't accept running background tasks synchronously, there is a viable solution, but it requires a little bit of effort.

Requirement


To re-affirm the requirement, we are going to walk through a solution that allows any workflow or business service in Siebel to be run asynchronously, under the actual users credentials, and without actually supplying the password!

If we step back, and look at the problem from an architectural perspective, we know that when the user logs into the Application and executes any type of code such as eScript, or workflow, it will run synchronously under the users application object manager, and it will operate under the users own credentials.

The only supported method to run code and avoid blocking the UI, is to run code under a background process, however any code that is run in the background will operate under SADMIN by default.

The first challenge is to find a supported method to execute code in a background session under the current users credentials. Experienced Siebel Integrators in the audience can probably guess, that we can achieve this with any Siebel API that supports SSO, or utilize the inbuilt impersonation capability of certain Siebel components.

The two most suitable choices are:
1. Siebel WS
2. Siebel JDB

I've highlighted these particular interfaces, because both provide APIs that can be invoked from a background session, and has the potential to allow arbitrary code to be dispatched, and run under a different set of credentials. As an added benefit, both of these APIs can be built to support load balancing for high availability.

With a suitable SSO API identified, we have the necessary key piece required to solve the above problem.

Solution Overview


The diagram below shows an overview of this proposed solution.

1. The user triggers an asynchronous task from the current user session, and is allowed to continue with their work in the application

2. In the background, A WFProcMgr component on the server executes the process, and calls a bespoke service called "JLE Session Transport Service" that acts as a wrapper for the transport.

3. A synchronous call is invoked over the transport, using the Siebel API identified above

4. The Siebel API instantiates a new session on behalf of the real user, executes the destination business service, and returns the results back to the background process.

Siebel API


An important implementation aspect worth considering upfront, is designing a transport agnostic API for the Siebel developer. The developer should be able to specify the desired subsystem, and the service should abstract the low level transport requirements, and perform the necessary acrobatics to make the call.

The following diagram shows how a facade is used for invoking the actual SSO interface.


A standard Siebel business service is defined, so it can be substituted in any existing workflow, or hooked into any existing Siebel trigger. The designer now has to implement the adapter code, and build the interface to communicate with the relevant SSO interface.

The implementation of interfaces to these Siebel APIs are out of scope of this article, as it requires different spectrum's of design, involve advanced integration, and require environment dependencies to exist, but the key considerations are provided below for customers who are interested in building this capability.

Depending on which SSO API is chosen, you may require an Integration specialist to implement a WS/BS dispatch wrapper, or a Siebel EAI/Java specialist to implement the Siebel-JDB bridge.

Please consult with your Siebel Integration Architect for more localized implementation advice.

Input Arguments


To support a plug and play design, I propose that all input arguments and child properties passed into this service would be dispatched to the remote business, in the same way as any other business service in Siebel including workflows.

This design allows us to go into tools, and retrofit this capability to any business service, without redevelopment effort. It is also deliberately designed to avoid hierarchy changes required for the correct invocation of the destination business service.

To control the dispatch, lets define a custom child hierarchy, which contains 3 key elements.

1. The user login to impersonate
2. The destination business service to dispatch to
3. The destination business method name

The diagram below illustrates the property set structure, with the required information to dispatch the call



This special PropertySet can sit at any index, however it should be removed before it is dispatched to the remote business service, in-case there is logic that is sensitive to property sets with specific indexes.

PropertySet Serialization


PropertySets are a Siebel proprietary representation of objects that only exist in memory. In order to send a PropertySet out of Siebel, and receive it back in, it has to be serialized, de-serialized, and potentially encoded to match transport constraints.

XML is usually used to transfer data between different systems, and Siebel provides methods to convert the PropertySet to XML, and back, however a more efficient method is to utilize Siebel's own PropertySet to text encoding format, or a utilise a custom JSON parsing engine in eScript. This minimizes the size of the message, results in less IO, and ensures that request is sent as efficiently as possible.

Error Handling


This is the most critical aspect of the design, as every component in the design has to be proactive in handling, and bubbling errors back up the chain.

The caller is effectively making a synchronous request/response call between two sessions. Errors at the end of the chain in the impersonated session, has to be propagated through the transport, to the calling session, which has the responsibility to handle and log any errors.

It is imperative that each component have guards in place to handle environmental, transport, and general unhanded exception scenarios, which could cause the call to fail.

This can happen for a variety of reasons including

Component is offline
Component Maxed Task
Connection timeouts
Failed authentication
Out of resource

The Siebel JDB interface has OOTB capability with extra options for configuring, settings such as retry, and timeouts, to counter some of the above issues. Customers choosing the Siebel WS API will have to consider implementing the appropriate mechanisms to guard against transient errors.

Load Testing


Both the Siebel WS, Siebel JDB options can be configured for load balancing through virtual server definitions. This capability allows this solution to scale easily. In practice this only works natively for the WS option, the JDB interface seems to only get load balanced if sessions are spawned from the thread. An as alternative, the load balancing responsibility for the JDB interface can be offloaded to the SRProc component.

Its important to understand the current expected load, and forecast the expected number of tasks that are needed. Every invocation can potentially spawn a new session, which could effectively double the amount of threads that the system has to spawn, along with the increased resource utilization, so it would prudent to involve your performance test team, and environment specialists to ensure that the application is tuned to handle the extra load.

Maintaining Impersonation


One last consideration in this design, is that once the user dispatches the work flow to be picked up in the background, no part of the invocation chain is allowed dispatch to another server process.

The impact is that the developer needs to ensure that all work flows are run inside the impersonated object manager. If a server request is initiated, that would then dispatch to a server thread, breaking the impersonation for that particular thread.

Conclusion


The above capability was designed and delivered for a customer who needed to meet regulatory audit requirements, to capture the identity of the users when they performed bulk CRUD operations across the application.

In the past customers who needed to meet this requirement would have chosen to take a hit in performance, and kill the user experience, while expensive background operations are forced to run in the foreground, or resort to creating extra columns across entities in the application to separately capture the identity of the user.

With a little bit of effort, Siebel customers now have the option of creating a custom impersonation component, that can run asynchronous tasks under a real user.

Open UI: Protocol Handler

Getting rid of Active X usage is a priority on Open UI projects, because it allows Siebel customers to embrace more modern browsers, and remain compliant into the future, but one of the last bastions of Active X support in a Siebel Open UI project stems from a requirement to launch executables inside an internal domain.

Launching a program on the users local machine, requires browser script code similar to the following to be implemented.

var sCmd = "notepad.exe text.txt";
var shell = new ActiveXObject("WScript.Shell") ;
shell.Exec(sCmd);

As part of an Open UI upgrade review to remove browser script, I recommended to a Siebel customer that they implement a custom protocol handler to get around the above dependence on Active X.

Protocol handlers are cross platform, and are supported by the major browsers. Although the term protocol handler is obscure amongst general users, its usage is quite prolithic. Here are some common examples of protocols which are handled by browsers

http://
https://
ftp://

The above protocols are trapped and handled internally by the browser, while other protocols are passed down to the OS to be handled by the default program.

mailto://
itmss://

When mailto:// is typed in to the browser address bar, it launches the default program which is registered to handle emails. Apple users will also recognize the second protocol itmss://, as it is used to launch iTunes, the subtle suggestion here is, we could use the same technique to launch any executable.

Each platform has its own method of implementing protocol handlers, so if you are lucky enough to support mac users on Siebel, you’ll need to look into the appropriate vendor documentation. The MSDN reference provides a good digest for those wanting to implement protocol handlers in windows.

https://msdn.microsoft.com/en-us/library/aa767914(v=vs.85).aspx

To implement a handler for a new protocol named jle://, a simple reg file can be created and imported to into the registry.

Windows Registry Editor Version 5.00
 
[HKEY_CLASSES_ROOT\jle]
@="URL:jle protocol"
"URL Protocol"=""
 
[HKEY_CLASSES_ROOT\jle\DefaultIcon]
@="C:\Apps\protocol_handler.bat"
 
[HKEY_CLASSES_ROOT\jle\shell]
 
[HKEY_CLASSES_ROOT\jle\shell\open]
 
[HKEY_CLASSES_ROOT\jle\shell\open\command]
@="C:\Apps\protocol_handler.bat" "%1"

Replace “jle” with your own protocol, and substitute the path to “protocol_handler.bat” according to your local path. “protocol_handler.bat” is a test file that can be used debug the arguments, to see what gets passed to the invoked program, but you may want to use something more sophisticated than a batch file.

rem protocol_handler.bat file
@echo off
echo %1
pause


Once thats setup, type the the new protocol + program arguments into the browser address bar and press enter. Here’s a screenshot of the protocol handler in action.



When designing the protocol handler, keep in mind that you may want to implement a lookup system, to maintain the flexibility of invoking different programs, without having to register new protocols.

This approach was used on an Open UI upgrade to negate the dependence on ActiveX in IP2012+. If you are looking to solve similar requirements on your project, it is important to keep the following points in mind.

Pros Cons

  • No Active X
  • Works across all browsers, and platforms
  • Does not need elevated privileges
  • Launch only registered programs

  • Requires desktop deployment
  • Requires non Siebel expertise


  • The Harness: Use Cases


    The Harness is a modern Siebel development tool, that was built to help a customers Siebel development team to maximize their efficiency, by reducing the amount of time that is lost through the build, compile, and unit test cycle. It was developed off the necessities of integration development, and evolved from some essential use cases.

    The Harness is deployed as part of the standard developers tool set, and is currently available to all integrators, configurators and production support staff, along side other necessary tools such as database clients, XML/code editors, source control tools.

    It has been about two years since the first alpha version was built, and it has matured into a product that is used for development and production support.

    The Harness was originally introduced to readers, in The Harness, and subsequently in The Harness Case Study, in this article, we go through the use cases, and provide videos walk-throughs of common usage scenarios.

    Use Case 1 - Expression testing


    Siebel expressions can be found universally in Siebel, but they can be notoriously difficult to troubleshoot.

    For complex expressions, it is easy to make a mistake, leave out a bracket somewhere, or mis-spell the reference to a field or function, and when Siebel expressions are configured incorrectly, they famously error out silently. This adds significant overhead to the development process, because each change has to be iteratively compiled into the SRF.

    The Harness provides an Expression module, that allows developers to simulate expressions in real time

    The Harness VS Vanilla Siebel


  • Provides the ability to simulate Siebel expressions in real time
  • Supports documented Siebel expressions
  • Supports undocumented Siebel expressions
  • Supports BC Field, and WF property notations


  • Expressions cannot be simulated in Siebel
  • Use Case Video: Expression Tester - Fun with Timestamp
    https://www.youtube.com/watch?v=JNH6CnGbAGU

    Use Case Video: Expression Tester - Lookup Function
    https://www.youtube.com/watch?v=w26Ae9Tnfjc

    Use Case 2 - Run eScript on the fly


    One of the pain points of Siebel development, is the need to make iterative changes to eScript, which requires the developer to compile the changes into the SRF, launch the thick client, and execute the code to build and unit test code. This is another task that adds significant development overhead.

    The Harness allows developer to build and test eScript on the fly.

    The Harness VS Vanilla Siebel


  • Write, test eScript on the fly using a web browser
  • Supports ST and Non ST Script engine
  • Supports custom script libraries
  • eScript can be run locally, on the server, or asynchronously
  • Code indenting, highlighting, JS code hints for Non ST engine
  • Library feature integrates with JSDoc comments for documentation
  • Code can be generated from predefined templates
  • Can be used to inspect runtime object variables


  • Traditional eScript needs to be compiled into the SRF, re-triggered, and retested after each change
  • Developers also have the option of using the notepad like client side business service editor
  • Use Case Video: Script Module - Run eScript on the fly
    https://www.youtube.com/watch?v=jERcYcguYXA

    Use Case Video:  Script Module  - Library Search
    https://www.youtube.com/watch?v=KNaSWYiIAfo

    Use Case 3 - A modern Business Service simulator 


    The BS SIM module provides the ability for developers to simulate business services with current application context.

    It also provides modern features such as auto complete, and input property helpers, that makes testing Siebel business service behavior much more fluid.

    The Harness VS Vanilla Siebel


  • Auto-complete data for business service names, and business service methods.
  • Auto-complete data for business service input arguments, workflow process inputs, and BRP Process inputs
  • Supports Siebel expression notation
  • Converts XML into SiebelMessage/PropertySet/Hierarchy, as an input into the service
  • Simulate any business service with current application context
  • Can be configured with project specific form prefills
  • Test cases can be saved, restored and persisted between sessions


  • Using Business Service Simulator the developer has to click new record, open the Pick applet, and use a wildcard search to get the service name, or method
  • Similarly for input property name/value, the developer has to click new record, open the child applet, click new record, and enter in a each name/value pair
  • No input property argument is provided
  • Complex hierarchies has to be manually constructed, or shifted from the back of another business service invocation
  • Simulation only works under the Business Simulator BO
  • Test cases history is not available in the UI, it has to be saved/restored from a file system location.
  • Use Case Video: BS Sim Module - Autocomplete Fields
    https://www.youtube.com/watch?v=xdZKsg4T1kQ

    Use Case Video: BS Sim Module - Form Prefill
    https://www.youtube.com/watch?v=Y0lsRU9hNEs

    Use Case Video: BS Sim Module - History, with eScript generation
    https://www.youtube.com/watch?v=umfnsBAGaXs


    Use Case 4 - Manipulating the data layer


    The BC module provides a tool for the developer to interrogate the configuration behind the current view, as they navigate the Application. The Harness automatically detects the application context, and pre-fills the necessary the form fields to manipulate the current record.

    This module a provides developers with the ability to create, update, delete, and query for data behind the UI.

    The Harness VS Vanilla Siebel


  • Detects UI context, and allows CRUD operation with a web UI
  • Supports non UI context objects
  • Tools configuration is exposed through the BC module UI


  • Developers have to use a combination of "About view", a SQL tool, and Siebel Tools
  • Use Case Video: BC Set Readonly Field
    https://www.youtube.com/watch?v=mu5hJvqyl9I


    Use Case 5 - Learning the Open UI API


    The OUI module provides the Open UI developer with a web UI, that integrates with the Siebel session, displays information about the current Screen, View, BO, BC, Applets, and controls, and allows developers to drilldown into the OUI API.

    Commands can be executed against the Siebel session using the inbuilt console, and the auto-complete box allows the developer to quickly search through multiple levels of API name spaces, and custom libraries.

    The Harness VS Vanilla Siebel


  • About View with object inspection
  • Learn the Open UI API by browsing on screen objects.
  • Console with access to the Siebel session
  • Indexes and provides auto-complete search for multiple script libraries
  • Automatically probes and displays information, from methods that return data


  • Developers can manually look at source code, or memorize documentation
  • Open UI developers still have to use About View", and lookup the corresponding configuration in Tools
  • Use Case Video: Open UI API Browser
    https://www.youtube.com/watch?v=n-CLyrc_YA8


    Other modules


    MQ/HTTP Module

    The Harness originated from this humble module. The MQ/HTTP module is an essential module for Siebel EAI/middleware developers. It provides a real time interface for sending/receiving messages across message transports. This specialist module provide important prefills on transport parameters, and message templates, that can be used to quickly setup data, and interrogate interface behavior, with XML aware syntax highlighting.

    IO/Msg Module

    The IO/Msg module is another essential tool for Siebel EAI/middleware developers, that allow messages to be generated, inspected, upserted/inserted/synchronised and converted on the fly.

    Server Health Module

    The Server Health modules probes the server to provide important information such as the machine that the session is connected to, server disk, memory, and CPU utilization. This has been found to be useful in pin pointing problems in resource constrained environments.

    FAQs


    Does it work with HI/Open UI?

    The Harness was originally designed to work in HI, and has since been ported over to Open UI.  It has been tested against HI 8.1, OUI IP2012, IP2013, IP2014, IP2015

    Does it replace Siebel tools?

    No. The Harness augments Siebel Tools, however depending on the role of the developer, they could spend a significant part of the day in The Harness testing interfaces, and writing unit test cases, reverting to Tools to create the necessary foundation objects.

    How long does it take to build such a tool?

    2 months to build an alpha version from scratch.

    What kind of skill set do I need to build this tool?

    Siebel professionals with expertise in eScript, Web professionals with strong JS, UX, and Java is also helpful

    I would love to have it as part of my collection, can you send me a copy?

    Unfortunately for third party Siebel tool collectors, The Harness is an internal product.

    Where can I download it?

    The Harness cannot be distributed in its current form.

    Can I trial The Harness?

    The Harness doesn't have a trial mode

    Can I take this tool onto my next project?

    The Harness contains specific customer code, and integrates tightly with a client library. Key parts of the program has to be ported, and a suitable core library has to be established, for it to be used on a new project.

    Contact me on Linkedin, if you are interested in knowing more.

    Advanced Siebel Integration - A SOA Review



    A recent Oracle SOA review into one of my customers Siebel Integration capability, rated it as highly advanced. The innovative systems, processes, and tools that were put in place, evolved immensely over the years, and enabled the customer to build, deploy, test and maintain their interfaces quickly and with little effort.

    In this organization, the Siebel team provided the front line EAI development, and supported a large portfolio of services across numerous touch points and multiple transport types including MQ, WS, File, HTTP. While adhering to architectural principles such as SOA, P2P, Pub/Sub, this Siebel system plays the role as a consumer, subscriber, publisher, and router of services.

    The environment mix, architecture, and scale may differ in your organization, but there are some fundamental challenges that all Siebel Projects face.

    High Level Challenges

    - Simplify and standardize Integration
    - Reduce Integration expenditure
    - Agility and Time to market

    Simplify and standardize Integration

    From a high level, there are a variety of messaging requirements that we may encounter, including Request/Response, Notification, Async callbacks, Pub/sub, then we have various transports to handle, which may include HTTP, Message queues, file formats. We then have to choose between the various transformation options, and figure out how to generate our message with the correct headers, deal with message dispatch and error handling.

    There are a variety of implementation options, and each developer will have their own approach to solving the above problems, which can lead to non standardized solutions, that overtime becomes difficult to maintain. Reducing the complexity, and standardising the way we implement integration requires a system that helps us to deliver a consistent solution to dealing with the above scenarios. 

    A key component of this system is establishing a Siebel Integration competency centre (ICC), which acts to standardise Integration solutions. An ICC can be defined as team that is responsible for the Siebel EAI needs across different streams and projects. 

    Another vital component is the integration framework. An integration framework encapsulates all common integration logic, implements standard design patterns, minimises customisation, coding, and minimize deviation from the standard design entry/exit points. A good integration framework provides the ability to integrate a lot of different technologies, and is instrumental in enabling projects to simply their Integration.

    Reduce Integration expenditure

    Some organisations approach integration on a case by case basis. The work is put together in an adhoc manner to achieve a certain result, and the grunt work is then repeated on the next project. Issues such as maintenance, error handing, and robustness, unfortunately is an after thought. This causes a lot duplication, waste, increases cost, maintainability, and risk.

    An alternative approach is to look at introducing an integration factory. The concept of an integration factory, is not dis-similar to how car factories mass produce vehicles using standard core components, where robots perform the heavy lifting and mundane tasks on a production line, and engineers move in to perform the specialist tasks that require finesse and skill. This results in a consistent and quality deliverable, that is cost effective.

    In the same way a Siebel integration factory, can delegate or automate certain integration chores, to free up the Siebel Integrator to perform more value added tasks. This factory consists of a set of tools and processes which allows projects to churn out interfaces using a cookie cutter approach to reduce integration build and test.

    Under this approach, less experienced integrators can be used to safely deliver interface work, because the factory enforces a consistent approach to the build. Contrast this with an adhoc build approach, where a less experienced developer is provided with 101 ways to implement something simple like validation, and countless ways to trigger, and implement automation. We can imagine the outcome will be inconsistent, to say the least, and is likely to result in increase costs for the next project who has to maintain this work.
     
    An Integration factory provides tools to work with messaging requirements, scaffolding for quick deployment of interfaces, and a controlled process to produce interfaces. Integration artefacts such as VBCs, Inbound, Outbound, Request/Response interfaces all have a certain flavour, and are produced with the same common components, and are orchestrated to meet a requirement. 

    By applying standardisation, establishing reusable components, and using proven design patterns to solve known problems, we can reduce the cost and risk of implementing integration.

    Agility and Time to market

    Many Siebel projects need to support rapidly changing business requirements, and deliver on future initiatives at the same time. Integration is traditionally seen as risky, costly, and usually delegated to major releases, which limits the ability to be agile and deliver solutions quickly.

    One of the keys to agility in Siebel is to keep logic out of the SRF, and fortunately Siebel provides a lot of options to configure EAI capability on the fly.

    Client side business services

    Client side business services allow Siebel customers to configure and deploy technical logic to a run time database. This capability allows project to change logic on the fly, and maintain a 24/7 application. There are customers that purely use Client side business service because of the agility advantages over eScript.

    Siebel Workflows

    Siebel workflows are the work horse of automation in Siebel, it provides a nice interface with a visual representation of customer logic. Workflows can be configured and deployed to a run time database, allowing projects to switch workflow definitions on the fly, and apply logic changes without downtime.

    Business Rule Processor (BRP)

    BRP was designed as declarative replacement for eScript, and allows projects to build application logic using a browser. These rules are then executed by a highly optimised engine that works beneath the Siebel BO layer to maximise logic execution. BRP rules can configured and deployed using the Siebel thin client.

    Data Mapper

    Datamapper is Siebel’s declarative engine that performs an object to object transformation. Data map transformations can be also configured using the Siebel thin client.

    Runtime Integration Objects

    Integration objects are representations of a model in Siebel, which is usually built using Siebel tools, and compiled into a SRF. However, with an appropriately connected thick client, Integration object definitions can be deployed to a run time database. When runtime Integration Objects are combined with the right mix of EAI technologies, Siebel customers can build and deploy interfaces without downtime.

    XSLT

    Siebel provides XSLT 1.0 as standard out of the box, but customers can easily implement an XSLT 2.0 engine to get access to an expanded set of transformation options, and speed benefits. XSLT provides a powerful declarative transformation alternative to eScript or Data mapper. 

    The Siebel product has a solid range of construction tools out of the box, that allow for changes to be deployed without downtime. Siebel customers that make best use of these technologies, have the opportunity to be very agile and reduce the time to market for their solutions.

    Summary

    In this article we’ve introduced the high level challenges of Siebel Integration that organisations  face. This prepares us for the next set of articles, where we’ll dig deeper, go through real world scenarios, consider solutions that are aligned to the principles introduced above, walk through the features of a Siebel integration framework, and address the following low level challenges.

    - Reducing technical burden
    - Dependency on external interfaces
    - Working in disconnected environments
    - Working within constraints of Siebel product
    - Reduce scripting and hard coded references
    - Improving code quality 

    Oracle Siebel Open UI Developer's Handbook Review


    Not so long ago, I received a book in the post from our good friends from Oracle (Ford, D & Hansal, A & Leibert, K & Peterson, J, (2015) Oracle Siebel Open UI Developer's Handbook: Oakamoor, P8Tech). I spent a weekend scrutinising the book and came away thoroughly impressed. So here is my perspective of the book.

    As one of the early adopters of Siebel Open UI, I had to learn the product quickly, train internal  developers, and establish a build process for working with this new technology.  The tools, scaffolding, processes and Open UI knowledge was then passed onto the next generation of developers.

    A lot of effort was put into establishing this internal competency, as it was done at a time when knowledge of Open UI was scarce, Open UI instructors were themselves being trained, so the best way to learn was to look through the source code, and browse the API tree using the browser console. 

    Now days training can be arranged quite easily, there are numerous blogs that spread Open UI knowledge, and you could still learn a lot by looking at the source code, so does this new book add any value? I contend that the "Oracle Siebel Open UI Developer's Handbook" provides significant value for the serious Open UI developer.

    "Oracle Siebel Open UI Developer's Handbook” is a comprehensive book, and its likely that you’ll need to read over many times to really gain benefit from it, there are sections that you’ll skim over because it may not be relevant in your current role, but its comprehensiveness is part of its broad appeal. The book covers Open UI basics, manifest, PM/PR, architecture, installation and deployment, visualisations, CSS themes, advanced scripting, integration and even mobile development. 

    Much has changed since I delved into the product that was then IP2012. The whole development team was sent on Advanced Open UI training, and since then only a handful of lucky people got to work with the technology. The half of the team that actively works with the technology are certainly more skilled than the other half that hasn’t use their training. This book  appeals to both markets, as it is aimed squarely at the developer, and doesn’t make any assumptions about the level of the reader.  

    Siebel projects around the world would have a similar mixed bag of Open UI knowledge. IP2012 is now superseded by IP2013, and IP2014 and the product has evolved quite drastically, so your skills might not be where you think it is. This book provides a great chance for Open UI professionals to learn the latest developments from the Open UI product or solidify their existing knowledge, without going through further training.

    Alternatively developers can invest a bit of time going through bookshelf, and trawl through all the articles on the web to self learn Open UI, it is certainly possible, but learning Open UI is not a natural stepping stone for many Siebel developers, and for the seasoned developer like myself, who is quite time poor, I found the book to be an excellent investment of my time, and wished I had it when I first learned Open UI.

    This book adds great value, as you learn Open UI from the 4 of the best Open UI experts in the world. If you like @lex’s work on Siebel Essentials (Siebel Hub), then this book is more of the same awesomeness, but packaged for the Open UI professional.

    Get it here from Amazon

    Open UI: Testing with Selenium

    Selenium is a browser automation tool, that allows web projects to automate repetitive tasks. As an early pioneer of Open UI, I found this new capability to be incredibly useful for achieving continuous integration. I gave readers a hint of this last year, in the article "Open UI - Build Process", which left readers with a list of ingredients, and a plan to manage their build automation.

    Now that more Siebel projects around the world are on embarking on upgrading their Siebel Applications to Siebel Open UI. Its about time we advance this concept into something more tangible that Clients can benefit from.

    Picture this... as part of the Open UI upgrade, you are required to navigate to 1000+ views in Siebel and test for WCAG defects in all 3 major browsers. That isn't so exciting.

    Testing for Open UI defects requires a high level of thoroughness to ensure that entire your Application is compliant, and remains compliant in the future. The move to Open UI will expose poor configuration practices that might have been passable in HI, but will break in Open UI. Open UI will also introduce defects as a side effect of the upgrade.

    The simple approach to this problem is to brute force it, and assign a team of developers to navigate to each view to analyse for technical defects. But this isn't really viable, as a long term strategy, a smarter approach is to use web automation.

    Web automation brings to mind images of robots that are used to scrape web sites, harvest email addresses and index content, but Siebel developers have a more important itch to scratch: Testing the Siebel UI.

    The obvious use case is to run continuous integration testing, to ensure that new builds are thoroughly tested over night. A more advanced crawler can be built to perform functional testing of the applications main areas, but the scope of this article is to show how you can build your own Open UI crawler that can be used to programatically navigate to each view, and optionally validate your application.

    Theres no shortage of tools in the market that is available to perform this sort of work, but if we narrow our criteria to open source web automation solutions, the Selenium web driver makes a pretty good choice, as it's also set to become a W3C recommendation. Selenium works with all the major browsers, is compatible with your favorite programming language, and it is also free.

    This article will provide you with a sample application that implements Open UI automation with Ruby, but you can extract the lessons learnt from this article, and implement them in your language of choice.

    I've chosen to use Ruby as my language, because there is solid support for web automation. Ruby has gems that allow the developer to easily parse the DOM, make selections using CSS or XPath syntax, deal with dialog boxes, and take screen shots of problem views. Combining with this Watir, which is a selenium wrapper in Ruby, allows the developer to build the automation quickly in a light weight language.

    The solution for your project may be different, if you require enterprise support, headless servers, or if you plainly prefer to stick to your language of choice, because of your available skill set, then the right tool for your circumstance will be different. The most important ingredient here is Selenium, or in this case Watir, which is a Ruby flavor of Selenium.

    Selenium

    This is the Browser automation API that allows you to control your browser programatically

    http://www.seleniumhq.org/

    " Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well."

    Browser Support

    Chrome, Firefox, IE...

    http://www.seleniumhq.org/about/platforms.jsp#browsers

    Remote control drivers

    Javascript, Java, C, Ruby, Perl, PHP...

    http://www.seleniumhq.org/about/platforms.jsp#programming-languages

    Open UI

    The crawler can be run against a thin client, or against a local thick Client running Open UI.

    Ruby Implementation

    Knowing Ruby isn't a prerequisite to understand this concept. Ruby is a pretty readable language, and I've added a good dosage of comments to explain what the code is doing so those from non Ruby backgrounds can follow.

    #import required libraries
    require 'rubygems'
    require 'watir-webdriver'
    require 'cgi'
    
    #Parametise the URL that will be used to navigate to the Open UI application
    #This should really be configured for a server URL for real testing
    #but im using a local URL for this example
    sURL = "http://localhost/start.swe?"
    sPass = nil
    sUser = nil
    
    #Open a new chrome browser session
    browser = Watir::Browser.new :chrome
    
    #navigates to the URL defined above
    browser.goto sURL
    
    #maximise the window so we can see all entire view
    browser.window.maximize
    
    #This block simulates the user login process for thin client connection
    #else it is bypassed for the thick client
    if sUser!=nil && sPass!=nil
         #fill in username in the user name field
         username = browser.text_field(:name, "SWEUserName")
         username.set sUser if username!=nil
         #fill in password in the password field
         password = browser.text_field(:name, "SWEPassword")
         password.set sPass if password!=nil
       
         sleep 1
         #click the login button
         browser.link(:text =>"Login").when_present.click
    end
    sleep 5
    
    

    Congratulations!. At this point we have just created a simple crawler that has logged into an Open UI Application.

    Next we want to tell it how to navigate the application, by going to the sitemap, read all the views, and parse all the JS links embedded in the view elements. This is done pretty easily in Ruby.

         #use CSS to locate sitemap icon, and click it 
         browser.element(:css => "li[name=SiteMap] > img").when_present.click
    
         #use CSS selector to read all the views in sitemap into an array
         #these links also contain an onclick attr that allows to emulate a view navigation
         links =browser.css("span[class='viewName'] > a")
    

    Finally, we loop through every link, execute the JS code to perform the navigation, and optionally perform any automation.

         links.each do |link|
         params = CGI.parse(link.attr("onclick"))
         sleep 5
         #Goto view
         browser.execute_script( link.attr("onclick") )
         end
    

    Thats as simple as it needs to be.

    To fire it up with a dedicated client, you'll first need to open up an existing session to start the local web server. When the above code is run, it will launch a new browser connecting to the existing session.

    For thin client connections, it is not necessary to pre-launch the session, the above code will instantiate a new browser session, and will login in normally.

    A crawler like the program above, can navigate the application, verify each view and flag views that have issues.

    Building a simple crawler is a easy, and is something that you can give to an energetic graduate to perform in a day, however if you require a crawler that can be a workhorse for your Continuous Integration strategy, then it needs to be a little more robust, and more scalable than the simple example above, or if your needs warrant a more specialized crawler, that can check for and enforce WCAG compliance in your application, then you'll ideally need a plugin system that can allow you to easily add new validators. If want to go a little further, and build in functional testing capabilities, then could build an API bridge to facilitate a DSL to build more readable test cases.

    Open UI + selenium opens up these exciting opportunities. For my client, having a nightly build process + continuous integration, and WCAG reporting, ensures they have a strict standards compliant UI and a more stable application for every deploy.

    This article provides the necessary ingredients, and a simple recipe for other Siebel customers to follow the same path.

    Further Reading

    Selenium Web driver
    Continuous Integration
    WCAG

    More on Open UI

    Serious Open UI developers should be on the watch, for the up comming Open UI book from some very distinguished authors

    Siebel Open UI Developer's Handbook

    Open UI: Grunt and JSHint

    Previously we looked at how Grunt can improve your Open UI development experience, by automatically reloading the browser, every time a source file is changed. This time we take another step, and configure an automated code check using Grunt and JSHint.

    Think of how Siebel Tools prevents you from compiling, or checking in poorly constructed eScript. JSHint will do the same (if not better) job of making sure your client side code is free from language errors, and potential problems.


    Starting from where we left off in Open UI: Grunt and Livereload, we should have an operational grunt job that reloads our browser. To add JSHint to the mix, we need to modify the package.json file slightly, to include one new line to install this new dependency

    {
      "name": "SiebelGrunt",
      "version": "0.0.1",
      "devDependencies": {
        "grunt": "~0.4.1",
        "grunt-contrib-watch": "^0.6.1",
        "grunt-contrib-jshint": "^0.10.0"
      }
    }


    With your command prompt at the directory with this package.json file, type the following command to pull down this module

    npm install

    Now we need to configure the Grunt build process, to register this new task, and configure it so it only checks changed files.

    module.exports = function(grunt) {
         var sJSPath = '../PUBLIC/enu/*/SCRIPTS/**/*.js';
        grunt.initConfig({
            pkg: grunt.file.readJSON('package.json'),      
              jshint: {
                   scripts: [sJSPath]
              },

              watch: {
                   scripts: {
                        files: [sJSPath],
                        tasks: ['jshint'],

                        options: {
                             nospawn: true,
                             livereload: true
                        },
                   }
              }
        });
        grunt.loadNpmTasks('grunt-contrib-watch');
        grunt.loadNpmTasks('grunt-contrib-jshint');

         grunt.event.on('watch', function(action, filepath){
              grunt.config(['jshint', 'scripts'], filepath);
         });

         grunt.registerTask('default', ['watch']);
    };

    To explain what I've done, this newly added block registers JSHint, and uses the file path that we defined previously
              jshint: {
                   scripts: [sJSPath]
              },

    Next, I register the 'jshint' task with the 'watch' task, which will run JSHint every time a file is changed.
                        tasks: ['jshint'],

    The only problem with this configuration is that, JSHint has no idea which file was modified under the watch task, so JSHint will run over all 1500+ files under Siebel scripts and lint them all. If this is run without a filter, your command prompt will hang, and after a while it will come back with a window similar to this.



    JSHint reports 15K plus errors across all the files under the Siebel scripts folder, but its not all bad, some of the code under Siebel/scripts is 3rd Party which can raise some errors, and the rest will have been through Oracle's own lint process, and then minified before delivery to customers. We should only be concerned with the files that we have control over.

    To make JSHint validate a single file, I added the following block.
         grunt.event.on('watch', function(action, filepath){
              grunt.config(['jshint', 'scripts'], filepath);
         });

    This configures a listener on the watch event, and passes in the name of the modified file to JSHint. Now we are ready to roll. Fire up the grunt process.

    grunt

    Switch back to your code editor, open up postload.js, and type in every body's favorite command "asdf"


    Watch will automatically fire JSHint, and report back on the errors, with the line number and a description of the problem. Note that we lost our purple background because our script failed to parse.

    Alright, now lets fix that bug


    We see that JSHint has reported "1 file lint free", and our browser has automatically reloaded in the background with our purple background again. 

    JSHint won't make your Siebel developers better web developers. It will only ensure that any code that is written is written with a consistent syntactic style, and that critical language errors are picked during build and unit test, but it will provide you with the coordinates of any errors that is picked up. 

    JSHint is a great safety net, but remember its recommendations can be ignored. To enforce this on a broader scale JSHint can be configured to run against the entire code base on a regular basis, and its results can be enforced by a QA process.

    In the case above, a small typo caused 12 errors resulting in missing background color, but in other scenarios, these kinds of errors could cause more serious issues, which illustrates the importance of a code quality tool like JSHint.

    Hopefully this article has piqued your interest in Client side build automation. Are you using similar tools on your project, or have you adapted a build process that you would like to share with the Siebel community?.

    Send in your comments.

    Open UI: Grunt with Livereload

    With Siebel Open UI development we naturally spend the majority of our time working on the browser. Part of that requires clearing the browser cache, and hitting F5 to reload the browser.

    Sometimes we reload the browser, and think afterwards, did I really clear the cache that time? Maybe not... so we the clear cache, and reload the browser again. Seconds later, we spot a typo in our code and the endless cycle continues. Wouldn't it be good if we can focus on writing the code, and let something else clear the cache and reload the browser? 

    This is where tools like Grunt can help.

    1. Download No Cache

    "No Cache" is a Chrome extension that I wrote to automatically clear your browsers cache. It is available for free on the chrome store. There are 1 click solutions out there, but I prefer to have an automatic background process take care of this.

    1. Right click on the No Cache icon, and choose "Options"
    2. On the "Cache" tab select "Enable for filtered requests"
    3. On the "Filters" tab select the input box, type in "start.swe" and click the + icon


    This tells No Cache to clear the cache for every request Siebel makes, this should work on the local or thin client. If you use another browser and have your own favorite clear cache plug in, then that will work as well.

    2. Install Grunt

    Grunt is a JavaScript task runner that we can configure to reload our browser, but first we need to download and install Node.js from this website


    Its not necessary to know what Node.js does, just know that it provides us with the environment to run Grunt. Once you've installed Node.js open a command prompt and type the following line to install the grunt command line globally.

    npm install -g grunt-cli

    Next we need to decide where to put our Grunt configuration. I like to put it under the Siebel/Client folder because I backup the PUBLIC folder, and don't want copies of Grunt and its modules copied along with this backup.

    Open a command prompt, cd to your Client folder, and follow the next set of instructions to install and configure grunt on your local machine.

    mkdir grunt
    cd grunt

    Create a new file in the same folder called package.json, with the following contents

    {
      "name": "SiebelGrunt",
      "version": "0.0.1",
      "devDependencies": {
        "grunt": "~0.4.1",
        "grunt-contrib-watch": "^0.6.1"
      }
    }

    back at the command prompt, type the following command to download the required dependencies defined in the file above

    npm install

    Next create a new file called Gruntfile.js, with the following contents

    module.exports = function(grunt) {
         var sJSPath = '../PUBLIC/enu/*/SCRIPTS/**/*.js';
        grunt.initConfig({
            pkg: grunt.file.readJSON('package.json'),
              watch: {
                   scripts: {
                        files: [sJSPath],
                        options: {
                             livereload: true
                        },
                   }
              }
        });
        grunt.loadNpmTasks('grunt-contrib-watch');
         grunt.registerTask('default', ['watch']);
    };

    This file is important, so lets run through it line by line

    >     var sJSPath = '../PUBLIC/enu/*/SCRIPTS/siebel/custom/**/*.js';

    This path specifies the folder and file pattern that Grunt will watch for changes. Essentially we want to watch for any changes to JS files in the Siebel script folders

    > watch: {

    This line defines a "watch" task, there are other tasks that can be run. Eg code lint/minification/, but I've kept it simple for this example.

    > files: [sJSPath],

    configure the files to watch using the path statement above

    > livereload: true

    This option will spawn a background process that will cause your browser it refresh automatically. To Integrate our browser with this background process, there are 3 options as described here


    The browser extension is the most suitable method for development, so go ahead and download it with these instructions.

    To start it all up, go back to the command prompt and simply type
    grunt

    To test it out, goto SCRIPTS, open up any JS file like "postload.js", modify it as below, and watch your browser reload magically.



    I've presented the bare minimum required to get up and running with Grunt, but once you have that setup, you no doubt want to look into what other things Grunt can do for you.

    A good place to start is..




    Siebel cScript




    Introduction

    eScript is Siebel’s de-facto scripting language, but in an alternate universe, Siebel could have called its scripting language cScript, in reference to its C heritage. eScript is based on the ECMA Script standard (from which JavaScript is also based upon), and shares the same C family style syntax, but it also has uncanny similarities to C that are unparalleled by other ECMA Script derivatives.

    Here are 5 good reasons, why Siebel could have called its scripting language cScript instead of eScript.

    1. Compiled Code

    eScript code is compiled, and runs outside of a browser, just like C.

    2. Pre-processors

    Pre-processor directives tell the compiler to perform specific actions before the code is actually compiled.

    A typical C program is constructed like so:

    #include <stdio.h>
    int main(void)
    {
        printf(“hi”);
    }
    

    #include is an example of a pre-processor directive.This tells compiler to import the standard IO library and make it available to the program.

    Veteran Siebel programmers will also recognise that the same syntax is used in eScript for EAI purposes.

    #include "eaisiebel.js"

    This pre-processor isn't explicitly documented under the main eScript reference, but it is hidden away under the following section.

    Siebel eScript Language Reference > Compilation Error Messages > Preprocessing Error Messages
    http://docs.oracle.com/cd/B40099_02/books/eScript/eScript_Troubleshooting6.html#wp1012712

    It is also mentioned in

    Business Processes and Rules: Siebel Enterprise Application Integration > Data Mapping Using Scripts > EAI Data Transformation
    http://docs.oracle.com/cd/B40099_02/books/EAI4/EAI4_DataMapUSScripts3.html

    A small caveat. The #include directive can be written in the following two styles in C.

    #include <eaisiebel.js>
    #include "eaisiebel.js"

    The <> brackets tell the compiler to look in a predetermined directory, while the “”(two double quotes) tell the compiler to look in the current directory as the c program, however in Siebel, when the “” is used, Siebel will look in Tools\Scripts\ for the include file. (Siebel v8+)

    3. Libraries

    In the above code sample under Pre-Processors, we saw the usage of a function called “printf”. printf outputs a formatted to the standard console. This isn’t part of the core C language, but was made available via the include pre-processor directive, which imports this command into the program from standard libraries.

    Standard C libraries from stdio.h, stdlib.h, are also available in Siebel, but there’s no need to include them, Siebel has conveniently wrapped these libraries in Siebel and exposed them through the Clib global object.

    eg.
    Clib.rsprintf – returns a formatted string to the supplied variable.
    Clib.fscanf – read data from file stream

    The complete Clib object reference can be found here
    http://docs.oracle.com/cd/B40099_02/books/eScript/eScript_JSReference109.html

    4. Reference Operators

    The following example shows how reference operators are used in a C program.

    #include &lt;stdio.h>
     
    void sub(int* main_int)
    {
        *main_int=10;
        printf("Address sub:\t %d\n",&*main_int);
    }
     
    int main()
    {
        int myInt=5;
        printf("Address main:\t %d\n",&myInt);
        printf("Before Value:\t %d\n",myInt);
        sub(&myInt);
        printf("After Value:\t %d\n",myInt);
    }
    //Results
    /*
    Address main:    2293340
    Before Value:    5
    Address sub:     2293340
    After Value:     10
    */
    


    In the above code, an int is given the value 5, its address reference in memory is provided to a sub function using a reference operator

    sub(&myInt);

    The sub function takes this reference, goes to its address in memory, and modifies the original value.

    We can the same functionality in eScript with the following code

    function sub(&main_int)
    {
        main_int=10;
    }
     
    function main(Inputs, Outputs){
        var myInt=5;
        Outputs.SetProperty("Before Value",myInt);
        sub(myInt);
        Outputs.SetProperty("After Value",myInt);
    }
    //Results
    /*
    Before Value:    5
    After Value:     10
    */
    

    Note: The reference operator in this case is declared in the sub function input argument. This tells Siebel to treat &main_int as a pointer to the original value, instead of a copy.

    5. Memory pointers

    Like all relatives, eScript and C share common ancesters. In this case, the commonality is in the sharing of memory pointers.

    The following example shows how an invoked C program can modify data inside eScript. This example is sourced from the bookshelf document on SElib
    http://docs.oracle.com/cd/B31104_02/books/eScript/eScript_JSReference253.html

    In summary, a couple of buffer objects are created.

        var P_CHURN_SCORE = Buffer(8);
        var R_CHURN_SCORE = Buffer(8);
    


    SElib is used to call a DLL, passing in the reference to the buffer

    SElib.dynamicLink("jddll.dll", "score", CDECL,
    …
        P_CHURN_SCORE,
        R_CHURN_SCORE,
    );
    


    The following C snippet, de-references the pointers, and modifies the data at the memory address that was allocated in the above eScript code.

    #include 
    _declspec(dllexport) int __cdecl
    score (…
        double *P_CHURN_SCORE,
        double *R_CHURN_SCORE
    {
        *P_CHURN_SCORE = AGE + AVGCHECKBALANCE + AVGSAVINGSBALANCE;
        *R_CHURN_SCORE = CHURN_SCORE + CONTACT_LENGTH + HOMEOWNER;
        return(1);
    }
    


    Clib vs eScript

    There is an obvious overlap between CLib and eScript functions, which causes some confusion on which is better to use. The recommendation from Siebel, is to use the eScript equivalent over Clib.

    The CLib library is useful for OS level operations, and those scenarios where you need to operate at the memory level, but such usage is usually confined to edge cases, otherwise eScript provides a safer and more familiar playground for Siebel professionals.

    Conclusion

    Siebel designed eScript as a hybrid of ECMA Script and C, providing customers with the (not so well documented) ability to extend eScript beyond its natural abilities. Although everyday configurators may never need to touch the Clib libraries, use pre-process instructions, utilize reference operators or even call custom DLL's, but understanding these advanced features, and recognizing the C heritage provides insights for experienced developers, to maximize, and extend the product beyond its out of the box capabilities.

    Finally, as for the great idea of renaming eScript, unfortunately the cScript moniker is already taken (most notably by cscript.exe from MS), but you could also argue the same for 'eScript', which is claimed by these other software companies.

    eScript - Eclipse scripting language
    http://wiki.eclipse.org/FAQ_What_is_eScript%3F

    EScript - C++ Scripting language
    http://escript.berlios.de/pmwiki/pmwiki.php

    Escript – Python programming tool
    https://launchpad.net/escript-finley/

    escript – Erlang scripting language
    http://www.erlang.org/doc/man/escript.html

    eScript.api - This is the name of an Adobe plugin
    http://www.processlibrary.com/en/directory/files/escript/403646/

    As a consolation, we could designate eScripters who primarily use Clib, cScripters. I think this might catch on =)