How To Handle / Parse / Split Large XML Files ?

We understood that we cannot fit a 10GB file into only 1GB of memory or 4GB of memory i.e. RAM size.

Even for the 77Mb file it might not be enough depending on your implementation.

Memory issue is a common problem for applications that need to handle large XML files.

The XML parser of BW works roughly like a DOM parser.

Meaning, it loads the whole XML file into memory. This is not really design for working with huge XML.

For reading a large XML file you will need to use a different XML parser that does not load the whole XML (like stax or sax or other ?). For this you need to use "java code".

Design steps:

Step1 : Get the stax jar file and java code from open source links stax xml parser.

Step2:  Do the necessary steps for copy the jar file in tibco library folders.
Step3:  Design one subprocess with java Code and configure the parameters as required for Jar file.

Step4:  Design the Main Process File Pollar, callprocess and Parse XML.

Step5:  Select the subprocess (which we designed in Step2) in main process call process.

Solution example posted in this blog with title BW solution for Large XML Files.



 

How to handle / read a large file in TIBCO BW ?

1) Best practise would be to use is File adapter, If we need to process a large file of size around 25MB.

The Tibco file adapter parses the file and send it over to Tibco BW Engine. Whenever the filesize is larger than 25MB the Tibco BW process hangs or give java.lang.OutOfMemoryError.
 
2) If the file is delimeter separated type (fixed format or comma separated), then we can use "Parse Data" palette to read the file in subsets. check the option "Manually specify start record". we can make a loop for this palette, where we can pass the 'startRecord' value as 100, 200, 300 in each loop. So that each loop reads the file with specific 100 lines. Then we can process these 100 lines within loop.

3) we may need to increase Heap Size as well in run time,we can see the setting in tibco administrator while deploying the project.

For heap size go to process service instance of that project->monitoring->server settings->maximum heap size

For threads(max jobs) go to process archive.par->advanced below we can see TIBCO BW Process Configuration there we can set the max jobs for this read file process to 8.

we can try setting EnableMemorySavingMode=true . 

Error : out-of-memory errors received by a Wait For JMS Topic Message

How should we fix out-of-memory errors received by a Wait For JMS Topic Message due to events with no Candidate Key matches?

A. A shorter process timeout should be configured in the Input tab of the activity.

B. A shorter event timeout should be configured in the Message Event tab of the activity.

C. The size of the LRU cache should be reduced in the Advanced tab of the activity.

D. The Candidate Key should be configured as "match-any."

The correct answer is option B.

Explanation:

Event Timeout in Message Event Tab specifies the amount of time (in milliseconds) a message waits if it is received before this activity is executed.

If the event timeout expires,an error is logged and the event is discarded.

If no value is specified in this field, the message waits indefinitely.

If we specify a shorter timeout,then we can meet the requirement given in the question.

Error: java.io.IOException: An AXSecurityException was thrown while trying to create the server socket on the port: xxxx

This exception indicates that a secure server socket could not be opened at the given port xxxx.

In order to initialize the server socket, a valid private key and pass phrase is required.

If you are using JKS keystore as an identity then make sure the keystore also contains the private key for the leaf certificate.

For example,the command
   
>>keytool –list –v –keystore     <keystore name>
   
will output the contents of the keystore on the console:
   
Check the Entry type: trustedCertEntry or Entry type
    If  it has  “trustedCertEntry” which means that this keystore  does not have the private key corresponding to the above certificate.

If the above keystore have “Entry type” as  “keyEntry” then its fine.

   
Resolution:

Create the key and import the signed certificate in the same keystore using the following commands:
   
1. keytool -genkey -v -alias <alias> -keysize 1024     -keypass <key password> -keystore <keystore name>     -storepass <keystore password> -validity <no. of days     of validity>
   
2. keytool -certreq -v -alias <alias> -keystore     <keystore name> -keypass <password for the keys>     -storepass <keystore password> -file <certificate     request file name>
   
3. keytool –import –v –alias     <alias> -keystore <keystore name> -storepass     <keystore password –file <signed certificate in a file>
    

SSL Tracing Information - Enable

For troubleshooting any problem related to SSL configuration in BW, it helps to enable the following tracing  properties:

Trace.Task.*=true (client-side SSL tracing information is made available)

Trace.Startup=true
Trace.JC.*=true

Trace.Engine=true
Trace.Debug.*=true

bw.plugin.http.server.debug: true (server-side SSL tracing information is made available)

We can specify the tracing  properties in a custom properties file anywhere on your file system (e.g. in C:\test\props.cfg) then reference the file in  your C:\tibco\designer\5.3\bin\designer.tra file using the property
   
java.property.testEngine.User.Args –p     c:/test/props.cfg
   
After updating your designer.tra file, we must restart Designer in order for the updates to take effect.