How To Handle / Parse / Split Large XML Files ?

We understood that we cannot fit a 10GB file into only 1GB of memory or 4GB of memory i.e. RAM size.

Even for the 77Mb file it might not be enough depending on your implementation.

Memory issue is a common problem for applications that need to handle large XML files.


The XML parser of BW works roughly like a DOM parser.

Meaning, it loads the whole XML file into memory. This is not really design for working with huge XML.

For reading a large XML file you will need to use a different XML parser that does not load the whole XML (like stax or sax or other ?). For this you need to use "java code".

Design steps:

Step1 : Get the stax jar file and java code from open source links stax xml parser.

Step2:  Do the necessary steps for copy the jar file in tibco library folders.
Step3:  Design one subprocess with java Code and configure the parameters as required for Jar file.

Step4:  Design the Main Process File Pollar, callprocess and Parse XML.

Step5:  Select the subprocess (which we designed in Step2) in main process call process.

Solution example posted in this blog with title BW solution for Large XML Files.



 

How to handle / read a large file in TIBCO BW ?

1) Best practise would be to use is File adapter, If we need to process a large file of size around 25MB.

The Tibco file adapter parses the file and send it over to Tibco BW Engine. Whenever the filesize is larger than 25MB the Tibco BW process hangs or give java.lang.OutOfMemoryError.
 
2) If the file is delimeter separated type (fixed format or comma separated), then we can use "Parse Data" palette to read the file in subsets. check the option "Manually specify start record". we can make a loop for this palette, where we can pass the 'startRecord' value as 100, 200, 300 in each loop. So that each loop reads the file with specific 100 lines. Then we can process these 100 lines within loop.


3) we may need to increase Heap Size as well in run time,we can see the setting in tibco administrator while deploying the project.

For heap size go to process service instance of that project->monitoring->server settings->maximum heap size

For threads(max jobs) go to process archive.par->advanced below we can see TIBCO BW Process Configuration there we can set the max jobs for this read file process to 8.


we can try setting EnableMemorySavingMode=true . 



Error : out-of-memory errors received by a Wait For JMS Topic Message

How should we fix out-of-memory errors received by a Wait For JMS Topic Message due to events with no Candidate Key matches?

A. A shorter process timeout should be configured in the Input tab of the activity.

B. A shorter event timeout should be configured in the Message Event tab of the activity.

C. The size of the LRU cache should be reduced in the Advanced tab of the activity.

D. The Candidate Key should be configured as "match-any."

The correct answer is option B.

Explanation:

Event Timeout in Message Event Tab specifies the amount of time (in milliseconds) a message waits if it is received before this activity is executed.

If the event timeout expires,an error is logged and the event is discarded.

If no value is specified in this field, the message waits indefinitely.

If we specify a shorter timeout,then we can meet the requirement given in the question.




Error: java.io.IOException: An AXSecurityException was thrown while trying to create the server socket on the port: xxxx

This exception indicates that a secure server socket could not be opened at the given port xxxx.

In order to initialize the server socket, a valid private key and pass phrase is required.

If you are using JKS keystore as an identity then make sure the keystore also contains the private key for the leaf certificate.

For example,the command
   
>>keytool –list –v –keystore     <keystore name>
   

will output the contents of the keystore on the console:
   
Check the Entry type: trustedCertEntry or Entry type
    
If  it has  “trustedCertEntry” which means that this keystore  does not have the private key corresponding to the above certificate.

If the above keystore have “Entry type” as  “keyEntry” then its fine.

   
Resolution:

Create the key and import the signed certificate in the same keystore using the following commands:
   

1. keytool -genkey -v -alias <alias> -keysize 1024     -keypass <key password> -keystore <keystore name>     -storepass <keystore password> -validity <no. of days     of validity>
   

2. keytool -certreq -v -alias <alias> -keystore     <keystore name> -keypass <password for the keys>     -storepass <keystore password> -file <certificate     request file name>
   

3. keytool –import –v –alias     <alias> -keystore <keystore name> -storepass     <keystore password –file <signed certificate in a file>
    

SSL Tracing Information - Enable

For troubleshooting any problem related to SSL configuration in BW, it helps to enable the following tracing  properties:

Trace.Task.*=true (client-side SSL tracing information is made available)

Trace.Startup=true

Trace.JC.*=true

Trace.Engine=true

Trace.Debug.*=true

bw.plugin.http.server.debug: true (server-side SSL tracing information is made available)

We can specify the tracing  properties in a custom properties file anywhere on your file system (e.g. in C:\test\props.cfg) then reference the file in  your C:\tibco\designer\5.3\bin\designer.tra file using the property
   
java.property.testEngine.User.Args –p     c:/test/props.cfg
   
After updating your designer.tra file, we must restart Designer in order for the updates to take effect.




Simple Example tibco hawk rule

we can use one rule to stop multiple ADB instances and/or applications.

In 'TestEditor' we can add multiple actions. In 'Action Editor' choose 'Method' for action type and search for adapters' microagent where we can select 'stopApplicationInstance' method.

Here is one simple example.

<?xml version="1.0" encoding="UTF-8" ?>
<ruleBase>
    <version>4.6.0</version>
    <name><![CDATA[stopRule]]></name>
    <schedule></schedule>
    <author>XPMUser on host at 09:56 uto, lip 7, 2011</author>
    <lastModification>XPMUser on host  at 09:57 uto, lip 7, 2011</lastModification>
    <comment><![CDATA[]]></comment>
    <rule>
        <name><![CDATA[Self:getUptime():60]]></name>
       <schedule></schedule>
        <overRuling>0</overRuling>
        <dataSource>
            <microAgentName><![CDATA[COM.TIBCO.hawk.microagent.Self]]></microAgentName>
            <methodName>getUptime</methodName>
            <interval>60000</interval>
        </dataSource>
        <test>
            <name><![CDATA[(Total millisec > 0)]]></name>
            <schedule></schedule>
            <operator class="COM.TIBCO.hawk.config.rbengine.rulebase.operators.GreaterThan" >
                <operator class="COM.TIBCO.hawk.config.rbengine.rulebase.operators.RuleData" >
                   <dataObject class="java.lang.String" ><![CDATA[Total millisec]]></dataObject>
                </operator>
                <dataObject class="java.lang.Long"  value="0" />
            </operator>
            <consequenceAction>
                <name><![CDATA[stopApplicationInstance()]]></name>
                <schedule></schedule>
                <microAgentName>COM.TIBCO.ADAPTER.adr3.SomeAdapter</microAgentName>
                <methodName>stopApplicationInstance</methodName>
                <properties>
                </properties>
                <performOnceOnly/>
                <escalationTime>0</escalationTime>
            </consequenceAction>
            <consequenceAction>
               <name><![CDATA[stopApplicationInstance():1]]></name>
                <schedule></schedule>
                <microAgentName>COM.TIBCO.ADAPTER.adr3.SomeOtherAdapter</microAgentName>
                <methodName>stopApplicationInstance</methodName>
                <properties>
                </properties>
                <performOnceOnly/>
                <escalationTime>0</escalationTime>
            </consequenceAction>
            <trueConditionPolicy>
                <trueCountThreshold>1</trueCountThreshold>
            </trueConditionPolicy>
            <clearOn>
               <clearOnFirstFalse/>
            </clearOn>
        </test>
    </rule>
</ruleBase>





What are the monitoring best practices ?

Using TIBCO Hawk, we can have these practices to achieve an efficient and effective way of monitoring the system and  applications.

Process existence

Create a rulebase based on the process existence. This is to check or test whether there are not enough or too many process instances that are running or you can use this rulebase to automatically restart a failed process.

Monitors the process resource utilization

Creating a rulebase that monitors the process resource utilization. This is to make sure that the use of the resources such as the memory and the CPU don’t get too big overtime, which might affect the system’s performance.

LogFile or Event Log Monitoring

Create a rulebase for your LogFile or Event Log Monitoring. Through this, you can make or define another rule based on the entries of those activities logged on which you’ll be able to perform necessary actions.

Checking for the existence of an Application Microagent also needs to be monitored. This is useful in a situation where you want to detect if the application that you are monitoring is not responding or not communicating with the agents.






Advanced features of TIBCO EMS

The advanced features of TIBCO EMS include JMS capability to non-Java programs such as C and C# Client API.

It also provides an administrative API which is used for managing and monitoring the EMS server and its components.

It can be configured for connection to other EMS servers such as fault-tolerance and routing so that multiple EMS servers can participate in the server operations.

It can also be connected to other TIBCO components specifically TIBCO Rendezvous and TIBCO SmartSockets.

It offers destination bridging which makes it easier to send the same message from one destination to other destinations.

It has JNDI and SSL support to provide security over the EMS server and its components. Also, it can be integrated into third party application servers.

  © Blogger templates The Professional Template by Ourblogtemplates.com 2008

Back to TOP