How To Handle / Parse / Split Large XML Files ?

We understood that we cannot fit a 10GB file into only 1GB of memory or 4GB of memory i.e. RAM size.

Even for the 77Mb file it might not be enough depending on your implementation.

Memory issue is a common problem for applications that need to handle large XML files.

The XML parser of BW works roughly like a DOM parser.

Meaning, it loads the whole XML file into memory. This is not really design for working with huge XML.

For reading a large XML file you will need to use a different XML parser that does not load the whole XML (like stax or sax or other ?). For this you need to use "java code".

Design steps:

Step1 : Get the stax jar file and java code from open source links stax xml parser.

Step2:  Do the necessary steps for copy the jar file in tibco library folders.
Step3:  Design one subprocess with java Code and configure the parameters as required for Jar file.

Step4:  Design the Main Process File Pollar, callprocess and Parse XML.

Step5:  Select the subprocess (which we designed in Step2) in main process call process.

Solution example posted in this blog with title BW solution for Large XML Files.



 

How to handle / read a large file in TIBCO BW ?

1) Best practise would be to use is File adapter, If we need to process a large file of size around 25MB.

The Tibco file adapter parses the file and send it over to Tibco BW Engine. Whenever the filesize is larger than 25MB the Tibco BW process hangs or give java.lang.OutOfMemoryError.
 
2) If the file is delimeter separated type (fixed format or comma separated), then we can use "Parse Data" palette to read the file in subsets. check the option "Manually specify start record". we can make a loop for this palette, where we can pass the 'startRecord' value as 100, 200, 300 in each loop. So that each loop reads the file with specific 100 lines. Then we can process these 100 lines within loop.

3) we may need to increase Heap Size as well in run time,we can see the setting in tibco administrator while deploying the project.

For heap size go to process service instance of that project->monitoring->server settings->maximum heap size

For threads(max jobs) go to process archive.par->advanced below we can see TIBCO BW Process Configuration there we can set the max jobs for this read file process to 8.

we can try setting EnableMemorySavingMode=true . 

Error : out-of-memory errors received by a Wait For JMS Topic Message

How should we fix out-of-memory errors received by a Wait For JMS Topic Message due to events with no Candidate Key matches?

A. A shorter process timeout should be configured in the Input tab of the activity.

B. A shorter event timeout should be configured in the Message Event tab of the activity.

C. The size of the LRU cache should be reduced in the Advanced tab of the activity.

D. The Candidate Key should be configured as "match-any."

The correct answer is option B.

Explanation:

Event Timeout in Message Event Tab specifies the amount of time (in milliseconds) a message waits if it is received before this activity is executed.

If the event timeout expires,an error is logged and the event is discarded.

If no value is specified in this field, the message waits indefinitely.

If we specify a shorter timeout,then we can meet the requirement given in the question.

Error: java.io.IOException: An AXSecurityException was thrown while trying to create the server socket on the port: xxxx

This exception indicates that a secure server socket could not be opened at the given port xxxx.

In order to initialize the server socket, a valid private key and pass phrase is required.

If you are using JKS keystore as an identity then make sure the keystore also contains the private key for the leaf certificate.

For example,the command
   
>>keytool –list –v –keystore     <keystore name>
   
will output the contents of the keystore on the console:
   
Check the Entry type: trustedCertEntry or Entry type
    If  it has  “trustedCertEntry” which means that this keystore  does not have the private key corresponding to the above certificate.

If the above keystore have “Entry type” as  “keyEntry” then its fine.

   
Resolution:

Create the key and import the signed certificate in the same keystore using the following commands:
   
1. keytool -genkey -v -alias <alias> -keysize 1024     -keypass <key password> -keystore <keystore name>     -storepass <keystore password> -validity <no. of days     of validity>
   
2. keytool -certreq -v -alias <alias> -keystore     <keystore name> -keypass <password for the keys>     -storepass <keystore password> -file <certificate     request file name>
   
3. keytool –import –v –alias     <alias> -keystore <keystore name> -storepass     <keystore password –file <signed certificate in a file>
    

SSL Tracing Information - Enable

For troubleshooting any problem related to SSL configuration in BW, it helps to enable the following tracing  properties:

Trace.Task.*=true (client-side SSL tracing information is made available)

Trace.Startup=true
Trace.JC.*=true

Trace.Engine=true
Trace.Debug.*=true

bw.plugin.http.server.debug: true (server-side SSL tracing information is made available)

We can specify the tracing  properties in a custom properties file anywhere on your file system (e.g. in C:\test\props.cfg) then reference the file in  your C:\tibco\designer\5.3\bin\designer.tra file using the property
   
java.property.testEngine.User.Args –p     c:/test/props.cfg
   
After updating your designer.tra file, we must restart Designer in order for the updates to take effect.

Simple Example tibco hawk rule

we can use one rule to stop multiple ADB instances and/or applications.

In 'TestEditor' we can add multiple actions. In 'Action Editor' choose 'Method' for action type and search for adapters' microagent where we can select 'stopApplicationInstance' method.

Here is one simple example.

<?xml version="1.0" encoding="UTF-8" ?>
<ruleBase>
    <version>4.6.0</version>
    <name><![CDATA[stopRule]]></name>
    <schedule></schedule>
    <author>XPMUser on host at 09:56 uto, lip 7, 2011</author>
    <lastModification>XPMUser on host  at 09:57 uto, lip 7, 2011</lastModification>
    <comment><![CDATA[]]></comment>
    <rule>
        <name><![CDATA[Self:getUptime():60]]></name>
       <schedule></schedule>
        <overRuling>0</overRuling>
        <dataSource>
            <microAgentName><![CDATA[COM.TIBCO.hawk.microagent.Self]]></microAgentName>
            <methodName>getUptime</methodName>
            <interval>60000</interval>
        </dataSource>
        <test>
            <name><![CDATA[(Total millisec > 0)]]></name>
            <schedule></schedule>
            <operator class="COM.TIBCO.hawk.config.rbengine.rulebase.operators.GreaterThan" >
                <operator class="COM.TIBCO.hawk.config.rbengine.rulebase.operators.RuleData" >
                   <dataObject class="java.lang.String" ><![CDATA[Total millisec]]></dataObject>
                </operator>
                <dataObject class="java.lang.Long"  value="0" />
            </operator>
            <consequenceAction>
                <name><![CDATA[stopApplicationInstance()]]></name>
                <schedule></schedule>
                <microAgentName>COM.TIBCO.ADAPTER.adr3.SomeAdapter</microAgentName>
                <methodName>stopApplicationInstance</methodName>
                <properties>
                </properties>
                <performOnceOnly/>
                <escalationTime>0</escalationTime>
            </consequenceAction>
            <consequenceAction>
               <name><![CDATA[stopApplicationInstance():1]]></name>
                <schedule></schedule>
                <microAgentName>COM.TIBCO.ADAPTER.adr3.SomeOtherAdapter</microAgentName>
                <methodName>stopApplicationInstance</methodName>
                <properties>
                </properties>
                <performOnceOnly/>
                <escalationTime>0</escalationTime>
            </consequenceAction>
            <trueConditionPolicy>
                <trueCountThreshold>1</trueCountThreshold>
            </trueConditionPolicy>
            <clearOn>
               <clearOnFirstFalse/>
            </clearOn>
        </test>
    </rule>
</ruleBase>

What are the monitoring best practices ?

Using TIBCO Hawk, we can have these practices to achieve an efficient and effective way of monitoring the system and  applications.

Process existence

Create a rulebase based on the process existence. This is to check or test whether there are not enough or too many process instances that are running or you can use this rulebase to automatically restart a failed process.

Monitors the process resource utilization

Creating a rulebase that monitors the process resource utilization. This is to make sure that the use of the resources such as the memory and the CPU don’t get too big overtime, which might affect the system’s performance.

LogFile or Event Log Monitoring

Create a rulebase for your LogFile or Event Log Monitoring. Through this, you can make or define another rule based on the entries of those activities logged on which you’ll be able to perform necessary actions.

Checking for the existence of an Application Microagent also needs to be monitored. This is useful in a situation where you want to detect if the application that you are monitoring is not responding or not communicating with the agents.

Advanced features of TIBCO EMS

The advanced features of TIBCO EMS include JMS capability to non-Java programs such as C and C# Client API.

It also provides an administrative API which is used for managing and monitoring the EMS server and its components.

It can be configured for connection to other EMS servers such as fault-tolerance and routing so that multiple EMS servers can participate in the server operations.

It can also be connected to other TIBCO components specifically TIBCO Rendezvous and TIBCO SmartSockets.

It offers destination bridging which makes it easier to send the same message from one destination to other destinations.

It has JNDI and SSL support to provide security over the EMS server and its components. Also, it can be integrated into third party application servers.

What is SOA ?

Service-Oriented Architecture (SOA) is an architectural style whose goal is to achieve loose coupling among diverse interacting software applications, enabling organizations to take advantage of existing investments in applications and systems.

SOA facilitates the development of modular business services that can be easily integrated and reused, thus creating a flexible and adaptable infrastructure.

Using a SOA approach, an organization can focus more resources and budget on innovation and on delivering new business services.

Systems that can successfully use SOA can minimize the disruption of planned or unplanned outages in an enterprise.

Some of the advantages of using SOA are:

1. Reduction in development time and cost: SOA services are easily reused and can be rapidly assembled into new, composite applications.
2. Lower maintenance cost: Reusable services reduce the number and internal complexity of enterprise services.
3. High-quality services: Increased service reuse creates high-quality services through multiple testing cycles from different service consumers.
4. Lower integration costs: Standardized services know how to work together, enabling disparate applications to quickly and easily connect.
5. Reduce risk:Fewer, reusable services provide greater control over corporate and IT governance policies, and reduce the overall compliance risk to an enterprise.

How does SOA achieve loose coupling among interacting software agents?

It does so by employing two architectural constraints:

• A small set of simple and widespread interfaces to all participating software agents. Only generic semantics are encoded at the interfaces. The interfaces are universally available for all providers and consumers. 
• Descriptive messages constrained by an extensible schema delivered through the interfaces. No, or only minimal, system activity is prescribed by messages. A schema limits the vocabulary and structure of messages. An extensible schema allows new versions of services to be introduced without breaking existing services.

 

 

 

TIBCO ADB Adapter Error - “Failed to load shared library”

When starting an Adapter Service from the TIBCO Designer Adapter Tester, you might encounter this error: “Failed to load shared library, library name: adb55.dll”.

Fix this error with the following steps:

1. Open up C:\WINDOWS\system32.
2. Open up the TIBCO RV home directory (it might be C:\tibco\tibrv\8.1\bin).
3. Copy the libeay32.dll and the ssleay32.dll from the TIBCO RV home directory to the system32 directory.
4. Say yes if you are prompted to replace the files.

Restart your TIBCO Designer and try again.

TIBCO BusinessWorks On TIBCO Silver Fabric

BusinessWorks can run on  Private Cloud

TIBCO BusinessWorks is powerful enterprise integration technology that enables organizations to expose existing systems as services, build new services, and orchestrate and assemble services into applications – with little or no coding. Deployed by over 1,000 companies world-wide, BusinessWorks provides vital infrastructure for some of the most mission-critical service-oriented business environments in production today.

TIBCO Silver Fabric is a platform for creating a private cloud in your own data center that allows you to automate the provisioning and deployment of critical business applications, such as BusinessWorks, increasing your organization’s efficiency and reducing the risk of critical systems failures.

Cloud computing is being touted as the “next generation of computing” and is seen as vital to many in the IT industry. The many benefits of migrating to a cloud computing model include: increased efficiencies through shared resources, greater flexibility,global accessibility, and reduced expenses compared to standard physical computing. These benefits are offset by a few shortcomings,including loss of control and increased security risk, to name a few.

Manage Your Business, Not Your Environment

TIBCO believes organizations shouldn’t have to choose between efficiency and security. TIBCO Silver® Fabric llows you to setup a private cloud within your existing environment. This private cloud offers the same benefits as the public cloud without the additional risk. TIBCO’s solution comes with additional benefits not seen in public cloud offerings, such as automated dynamic scaling and run-time optimization – intelligent features that enable  TIBCO’s software to manage your environment, so you can manage your business.

EMS vs RV

• EMS is server based while RV uses a communications daemon on every machine.
• Depending on the requirements for speed and bandwidth, flexibility, security,reliability, availability and message passing between sub-nets one might require EMS or RV or a hybrid of both. 
• Both TIBCO RV and TIBCO EMS provide authentication, authorization and secure communication. Tibco RV and TIBCO EMS both provide secure socket layer (SSL) support for all communication. Where Web access is used, HTTPS is supported as well.
• In conclusion the EMS (JMS) hub-and-spoke model provides a better choice for the majority of EAI problems and RV is best left to its niches: high volume, high performance, 1-to-very many. 
• Generally use RV for speed topic / broadcast messages and EMS / JMS the more you are worried about assuring messages get to a receiver.  
• You can tune both products to go faster and more reliable. RV is basically broadcast via UDP, so needs RV Remote Daemons to cross subnets. You can bridge EMS to RV if you have specialised areas which you want to speed up.

TIBCO Products

Below are the TIBCO Products.

Business Optimization
Analytics and Visualization	TIBCO Spotfire
Complex Event Processing	TIBCO BusinessEvents
Data Governance	TIBCO Collaborative Information Manager
	TIBCO Data Quality
	TIBCO Data Quality Service Engine
	TIBCO Patterns – Search
	TIBCO Patterns – Learn
	TIBCO Profiler
Business Process Management
BPM in the Enterprise	TIBCO ActiveMatrix BPM
	TIBCO ActiveMatrix BPM Spotfire
Process Modeling	TIBCO Business Studio
Simple Applications and Workflows	TIBCO Formvine
Intelligent Operations  Management Platform	Nimbus Control
Cloud
	TIBCO Silver Fabric
Service-Oriented Architecture
Business Integration	TIBCO Adapters
	TIBCO BusinessWorks SmartMapper
Composite Applications	TIBCO ActiveMatrix Adapters
	TIBCO ActiveMatrix BusinessWorks
	TIBCO ActiveMatrix Service Bus
	TIBCO ActiveMatrix Service Grid
	TIBCO ActiveSpaces
	TIBCO Ajax Message Service
	TIBCO General Interface
	TIBCO PortalBuilder
Grid Computing	TIBCO GridServer
	TIBCO Grid Federator
	TIBCO Grid Analytics
SOA Governance	TIBCO ActiveMatrix Lifecycle Governance Framework
	TIBCO ActiveMatrix Policy Manager
	TIBCO ActiveMatrix Service Performance Manager
	TIBCO Hawk
	TIBCO Enterprise RTView
Mainframe Integration	IBM i Integration
	Mainframe Adapters
	Mainframe Messaging
	TIBCO Substation ES
	TIBCO Object Service Broker
Messaging	TIBCO Enterprise Message Service
	TIBCO FTL
	TIBCO Messaging Appliance P-7500
	TIBCO Rendezvous
	TIBCO Web Messaging for TIBCO Enterprise Message Service
Multi-Enterprise Connectivity	TIBCO BusinessConnect
	TIBCO BusinessConnect Insight
	TIBCO BusinessConnect Remote
	TIBCO BusinessConnect Trading Community Management
	TIBCO Foresight
	TIBCO Managed File Transfer
	TIBCO Slingshot
Telecommunications
Fulfillment Orchestration	TIBCO ActiveCatalog
	TIBCO ActiveFulfillment
	TIBCO KPSA
Other Products
	TIBCO Loyalty Lab
	TIBCO Netrics
	TIBCO OpenSpirit

Creating a TIBCO Hawk Domain

Creating a TIBCO Hawk Domain -

Create a TIBCO Hawk domain if you wish to isolate groups of TIBCO Hawk agents into independent monitoring sets.

Remember that each TIBCO Hawk agent can belong to only one TIBCO Hawk domain and cannot communicate with the TIBCO Hawk Display program on other domains.

The default is to monitor an entire network as one group. To use the default, omit the hawk_domain option.

Testing a Domain -

To test the domain option, temporarily set the TIBCO Hawk domain on a single system to a name such as test, then start (or restart ) the TIBCO Hawk Display.

If you entered a unique domain name, you only one discovered host should appear in the TIBCO Hawk Display main window.

This shows that only the local agent and the local TIBCO Hawk Display are using the test domain.

If the domain is as desired, you can now install it on the network systems where you want to assign specific computers to that domain.

All systems assigned to that TIBCO Hawk domain appear on the Display as part of that domain.

When finished with testing, change back to the default domain and restart the TIBCO Hawk Display to verify that it was correctly restored.

Enterprise Integration

Enterprise Integration reduces the cost ,improves the responsiveness and innovative.

Problems in IT today, that can be addressed by EAI are High Order processing cost,No consolidated view of the customer,High maintenance cost.

Integration Tool TIBCO will transforms the business into Event Enabled Enterprises.

Connections are established between systems without a coherent strategy

Connectivity based,Connection technology is standardized SOA Oriented connectivity

Integration Infrastructure is Intelligent infrastructure with support for process automation, transformation etc.

Catch activity vs Error path

In TIBCO, DB Adapters are C based and BusinessWorks is java based applications.

Hence we can't catch all the exceptions from Adapter in BW using Error path.

So we need to use either "catch" or "Write to Log" to know the error in BW. 

Scenario on EMS Server start up

If we have config file and duplicate queue/topic names are present in that conf file, so can we start this EMS server with that conf file ?

Yes,when starting EMS Server, if the duplicate queue/topic names are present in respecitve conf files, while starting up, EMS Server displays the duplicate names and the server will get start.

If  the duplicate entries are present in bridges.conf then what will happen while starting up the EMS server ?

duplicate entries are present in bridges.conf then it  displays the line number where it got the duplicate entry and server will not get start.

How can add a machine to DOMAIN from command line ?

Below are the steps to add a machine to domain from command line.

Step 1: Go to C:/tibco/tra/5.6/template/domainutility/cmdline

Step 2: Create a new file by copying the AddMachine.xml to another file name.
Ex: cp AddMachine.xml AddMachine_XX.xml

Step 3: Edit AddMachine_XX.xml, enter domain related specific entries.

Step 4: Go to C:/tibco/tra/5.6/bin, execute the following command.

./domainutilitycmd -cmdFile C:/tibco/tra/5.6/template/domainutility/cmdline/AddMachine_XX.xml –verbose

Step 5: Finish.

How can change the Global Variables value at runtime ?

To change the Global Variables value at runtime, add a property like below in the application .tra file and restart the application.

tibco.clientVar.SharedResource/Connections/MaxSessions=16

xpath function "tib:render-xml" vs "Render XML" activity

• We can use tib:render-xml function instead of Render XML activity if we are not modifying XML structure.
• We can't set encoding with the function, where as with activity we can.
• By default, Render XML activity do the XML validation, not Schema validation.

  

BusinessWorks Process Design Methodology

Analysis

TIBCO BusinessWorks implicitly supports analysis and design by offering a set of objects representing services and activities as the basis for the project flow. The design team can use these objects during project design.

Domain Setup

A TIBCO administration domain is the set of software and hardware resources used by integration project.
Services Configuration

TIBCO BusinessWorks uses different types of services that can be accessed from within the process.

Process Design

Using the TIBCO Designer GUI, you create your business process using predefined activities and add

conditions and mapping as appropriate.

Deployment

The TIBCO administration domain supports a simple installation and deployment paradigm

Management and Monitoring

After the integration development team has configured and deployed the integration project, can use the TIBCO Administrator GUI for monitoring and management.