Introduction
When we think of the top Integration tools used in a Siebel project, typical answers such as SOAP UI, Oxygen, XML Spy, Wireshark, Fiddler, MQ Harness, and even Hex Editors may come to mind.
But there is an even more important tool which is missing. You've probably guessed it, what list would be complete without an XML logger?.
Not to be confused with an eScript logger that prints a property set or a WF step that simply writes the XML to a text file. An XML logger is a component, that is part of a more larger Error Handling Framework, which should provide the capability to associate the XML message with the related error information.
Importance of an XML Logger
Once upon a time, I took over the reigns of an Error Handling framework that had an Achilles heel: it chopped off every single EAI message at the 2000 character mark, and all that was captured was a partial message header full of name spaces. Not only was the payload discarded, only part of the message wrapper was captured.
For the benefit of non EAI professionals reading this article, imagine that as a kid, someone gave you a lollipop, except that there was no candy, not only that, half the plastic wrapper was torn off, so you don't know what flavour it was, or even the name of the bastard who gave it to you. This situation could bring any kid or EAI professional to tears.
For an EAI professional, living without XMLs is like working blindfolded, it's almost impossible to pinpoint where failures are occurring.
If the system didnt log the outbound message, the developer would have to spend time recreating the test case, and simulate what the message "could" have looked like at a point in time, assuming the problem is reproducible. Without access to the full inbound message, your hope would rest on the external system providing the message, that your system should have rightfully logged.
Having to live with poor error handling, is a hard lesson to learn, but it does illustrate a common need for projects to have access to all Inbound and Outbound messages across all Interfaces.
EAI XML Queuing Service
Siebel provides a solution in the form of the "EAI XML Queuing Service"
http://docs.oracle.com/cd/B31104_02/books/ConnSAP/ConnSAPeAIQueue12.html
Professionals in the Siebel Community have already discovered this business service has the capacity to act as an XML logging tool.
★ http://siebelexplored.blogspot.com.au/2011/01/exception-handling-using-eai-queue.html
★ http://siebeltips.wordpress.com/2009/02/19/eai-queue-for-error-handling/
★ http://geeksbloggingat.com/2009/05/03/siebel-eai-queue-usage/
To summarise, this out of the box business service can be used to log a SiebelMessage to the Siebel file system for storage and retrieval.
The EAI Queuing service has the following benefits
★ Can store more than 2000 characters! Hooray
★ Message can be logged with Error Information
★ Can be used to replay a transaction
★ OOTB
However, if you have a diverse portfolio of services to interact with, where the SiebelMessages have to conform to Enterprise schemas, or even third party proprietary formats, then you will need the capability to log the raw interface message. This could be in the form of XML, CSV, flat file, or it could be SiebelMessage with a different wrapper.
The problem with the "EAI XML Queuing Service" is that it only supports logging of a valid SiebelMessage, which might seem to dismiss this business service as a credible solution, but there is an easy work around. A simple XML based IO can be created, with one IC, and one Field called "Payload". The XML can simply be injected into this Payload property to store your XML. This XML based IO simply acts as a wrapper to allow the XML to be accepted by the EAI XML Queuing Service.
This works, but it does have a few major drawbacks.
★ The wrapper has to be stripped to reveal the Raw Message
★ The raw message, wont actually be raw anymore. XML cant be nested in other XML without being interpreted as part of the schema, so Siebel will automatically HTML encode your raw XML, so it can be accepted as text.
★ The file type is TXT, and the file name cannot be controlled
★ There is a 1-1 relationship between the error and the message
★ The available fields to store error information is limited
There are major limitations with using this OOTB business service as a standalone solution, because in order to exchange messages with external teams, the EAI Queued message has to be stripped of the wrapper, and decoded before it can be used. EAI Developers can perform this task without complaint, but it is hardly accessible to other teams.
Eg. Testers who have access to the raw messages, can raise defects, and exchange the raw XMLs with external teams directly, but wont necessarily have the tools or knowledge to decode XML, so providing access to the raw XML provides convenience and saves time.
Using the EAI Queue Item as a means to store error details, also means you are limited to storing one message per error record, unless a method is devised to flatten the SiebelMessages, and store them as properties in a wrapper IO, which comes back to the decoding problem described above.
The other issues described in the points above are fairly obvious, and don't need any further narrative, so lets move on, and look at the ideal solution.
XML Log Details
In the last article, I provided a list of error details that should be captured. Here is a revised list of the error details that should be captured when an Integration error occurs. I've expanded on the original list to reveal the specific details an Error Framework and XML logger should handle for Integration (highlighted in blue).
★ User Name
★ Error Time
★ Host Name
★ Primary Id
★ Internal Error Code
★ Error Message
★ Error Category
★ Transaction Reference
★ Transaction Time
★ Calling Function/WF
★ Stack Trace
★ External Error Type
★ External Error Code
★ Transport Time
★ Transaction Id
★ Transport and Transport Parameters
★ XML Logs
Outbound XML
Outbound SiebelMessage
Inbound XML
Inbound SiebelMessage
It should be clear at this stage, that we'll need a custom table, and a parent child relation between the error message, and the interface messages. It should also be clear that the EAI Queuing Service works OOTB, but dosnt provide the flexibility for complex EAI error handling requirements.
There is a lot to take in, so in the next article, we go through the solution to overcome these limitations, and introduce a new capability, that will allow you to have full visibility of the interfaces firing around you.
Thanks ! I see you have referenced my blog in this article as well. Thanks for that :)
ReplyDeleteUsing EAI Queue in Siebel has another disadvantage. The XML message is stored as an individual file in Siebel file system which increases the disk I/O usage impacting performance directly. As well an enterprise solution using such approach should have a good archiving solution in place to keep the system health green.
Cheers,
NanPats
Hi NanPats
ReplyDeleteThats a valid concern, bad file logging design can bring a system to its knees, but IMHO I see the file attachment as an advantage. A small hit in performance to capture critical information when an error occurs is quite justified.
Readers take note on the excellent archiving advice! =)
Jason
Great Articles!!
ReplyDeleteHow is the next part coming along?
I have a proof of concept to build :)
Hi, Thank you for your interest. The next article is scheduled to be published in two weeks.
ReplyDeleteHi Jason
ReplyDeleteI have made XML logging a component of the eScript Framework I blogged about quite a while ago (I guess it is due for an update):
http://thesiebelscholar.blogspot.com/2010/05/escript-framework-logging.html
The premise of my logging service is to store all data in the DB. This serves two points:
(1) Easy to relate Logging data to Txn records in various ways (Link integration and escript logs to a particular quote id being transacted upon)
(2) Do not need access to the file system to see log data which is especially useful when dealing with Unix app servers
To overcome the file length limitations you mention, what I did was to break the text XML across multiple detail records. I then use a business service to concatenate all these detail records into a single calculated field so it is displayed in the UI. I basically copy and paste the XML data into a better Notepad++ so I can use Pretty Print on it, though I have been exploring how I can use browser script to launch a web editor to do this (browser script is not my strong suit)
I will try to spend some time updating the blog so people can see an alternative approach
Mik
Hi Mik,
ReplyDeleteThanks for your detailed comment. I look forward to seeing your alternative approach.