How To Handle / Parse / Split Large XML Files ?

We understood that we cannot fit a 10GB file into only 1GB of memory or 4GB of memory i.e. RAM size.

Even for the 77Mb file it might not be enough depending on your implementation.

Memory issue is a common problem for applications that need to handle large XML files.

The XML parser of BW works roughly like a DOM parser.

Meaning, it loads the whole XML file into memory. This is not really design for working with huge XML.

For reading a large XML file you will need to use a different XML parser that does not load the whole XML (like stax or sax or other ?). For this you need to use "java code".

Design steps:

Step1 : Get the stax jar file and java code from open source links stax xml parser.

Step2:  Do the necessary steps for copy the jar file in tibco library folders.
Step3:  Design one subprocess with java Code and configure the parameters as required for Jar file.

Step4:  Design the Main Process File Pollar, callprocess and Parse XML.

Step5:  Select the subprocess (which we designed in Step2) in main process call process.

Solution example posted in this blog with title BW solution for Large XML Files.



 

How to handle / read a large file in TIBCO BW ?

1) Best practise would be to use is File adapter, If we need to process a large file of size around 25MB.

The Tibco file adapter parses the file and send it over to Tibco BW Engine. Whenever the filesize is larger than 25MB the Tibco BW process hangs or give java.lang.OutOfMemoryError.
 
2) If the file is delimeter separated type (fixed format or comma separated), then we can use "Parse Data" palette to read the file in subsets. check the option "Manually specify start record". we can make a loop for this palette, where we can pass the 'startRecord' value as 100, 200, 300 in each loop. So that each loop reads the file with specific 100 lines. Then we can process these 100 lines within loop.

3) we may need to increase Heap Size as well in run time,we can see the setting in tibco administrator while deploying the project.

For heap size go to process service instance of that project->monitoring->server settings->maximum heap size

For threads(max jobs) go to process archive.par->advanced below we can see TIBCO BW Process Configuration there we can set the max jobs for this read file process to 8.

we can try setting EnableMemorySavingMode=true . 

Catch activity vs Error path

In TIBCO, DB Adapters are C based and BusinessWorks is java based applications.

Hence we can't catch all the exceptions from Adapter in BW using Error path.

So we need to use either "catch" or "Write to Log" to know the error in BW. 

How can change the Global Variables value at runtime ?

To change the Global Variables value at runtime, add a property like below in the application .tra file and restart the application.

tibco.clientVar.SharedResource/Connections/MaxSessions=16

xpath function "tib:render-xml" vs "Render XML" activity

• We can use tib:render-xml function instead of Render XML activity if we are not modifying XML structure.
• We can't set encoding with the function, where as with activity we can.
• By default, Render XML activity do the XML validation, not Schema validation.

  

BusinessWorks Process Design Methodology

Analysis

TIBCO BusinessWorks implicitly supports analysis and design by offering a set of objects representing services and activities as the basis for the project flow. The design team can use these objects during project design.

Domain Setup

A TIBCO administration domain is the set of software and hardware resources used by integration project.
Services Configuration

TIBCO BusinessWorks uses different types of services that can be accessed from within the process.

Process Design

Using the TIBCO Designer GUI, you create your business process using predefined activities and add

conditions and mapping as appropriate.

Deployment

The TIBCO administration domain supports a simple installation and deployment paradigm

Management and Monitoring

After the integration development team has configured and deployed the integration project, can use the TIBCO Administrator GUI for monitoring and management.