In my
LinkedIn group, I recently suggested that converting an XML from UTF-8 to UTF-16 using the Transcode Service was impossible.
This myth was quickly busted by
Mr Unleashed, so here is the background for everyone else who's encountered this problem.
Do a search on support web, and you'll come across this article
How to Read UTF-8 encoded files using Script [ID 602632.1]The above article concludes by saying that after you've read the UTF-8 file, you need to convert it to UTF-16 to use in Siebel. However, it acknowledges that the Transcode Service cannot do this.
"According to the Bookshelf, the Transcode Business Service should offer this kind of conversion. However there is Bug 12-KTEC6N "Transcode Service does not offer conversion from UTF-8 to UTF-16 encoding" has been logged to indicate this functionality is not currently available."and it also offers the following hardcore workaround
"To do this you could build a Com Callable Wrapper on the System.text.UnicodeEncoding .NET class and use the GetBytes method to convert a UTF-8 string to UTF-16 encoding.
For a non-Windows environment you could write a custom function that takes care of character encoding in C or C++ and make it available using a DLL. This could then it can be invoked from eScript using the SElib.dynamicLink method."Another article on Support web also picks up on this behaviour
Problems of Reading UTF-8 file [ID 539077.1]It also notes
"The Change Request number 12-KTEC6N ("Transcode Service" does not offer conversion from UTF-8 to UTF-16 encoding) has been logged for addressing this wrong behavior."[Siebel and UTF-16]UTF-16 messages are roughly equilivent to double the size of UTF-8 messages, and to provide a bigger picture on what this actually means, consider this.
You've just designed to your application to receive messages from an external system. Siebel only talks in UTF-16, so you ask your enterprise to only send you messages in UTF-16 format.
In the beginning these messages were small, and didnt pose any problems, as your organization grew, so did the number and size of the messages.
You then reach a point when users complain that the application takes to long to respond, and your enterprise asks you why everyone else can talk in UTF-8, and Siebel cannot?
[Solution]Fortunately there is a simple solution, the Transcode Service does actually work.
It will successfully convert a message from UTF-8 to UTF-16, but does not change the encoding tag, so you'd be forgiven if you thought it didnt work.
To verify that it does infact convert the message into UTF-16, you can use any HEX editor to inspect the file and check for the
Byte Order Mark (BOM) character.
This is also documented in the following article
Why does the Convert method of Transaction Service BS not adjust the "encoding" attribute of the XML header to reflect the target encoding? [ID 852822.1]To fix the prologue, you could use a simple business service to perform a replace on the encoding tag, or if you are using XSLT, you just need to ensure that your XML output has the UTF-16 tag.
I've heard of many projects who stick to UTF-16 for the above reason. Its not easy finding information on support web, the information that you do find can conflict with each other, and sometimes the articles are not accurate.
This was an interesting find by colleage PW, which will surely affect a lot of developers. 
