Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Tuesday, February 5, 2019

TIBCO BW (BusinessWorks) and BE (BusinessEvents) Code Review Automation

As part of setting up DevOps pipeline for Continuous Integration / Continuous Deployment (CI – CD) for TIBCO BW (BusinessWorks) and BE (BusinessEvents), we used the BW Code Scanner and BE Code Scanner products from https://bwcodescanner.com. They are very comprehensive from code review perspective with 100+ rules for BW Code review and 75+ rules for BE Code Review. Automating this as part of DevOps pipeline helped us to focus the manual code reviews on Design and functional gaps.

BW Code Scanner and BE Code Scanner perform the static code review analysis for hard coded configurations, adherence to best practices, optimization opportunities and known patterns of defects. One interesting part about them is that they are very in-depth. BW Code Scanner, for example, traverses all the paths from each activity node in a process instance and evaluates the transitional conditions in all paths from Start to End while evaluating for missed configurations and missing transitions. In our case, we had a JMS Queue Receiver in client acknowledgment mode and it highlighted all the success and error paths from Start to End where there was no message confirmation. This helps to ensure that the message doesn’t end up in an infinite loop. In addition to this, it was doing dynamic global variable mapping, so, in the above case, even if the acknowledgment mode was configured as Global variable, it was able to detect based on value and highlight the paths where message confirmation is missing. Pretty comprehensive indeed.

Another interesting feature is SonarQube plugins for TIBCO BW 5.x, 6.x and BE. You can copy these plugins in the extensions folder of SonarQube and the code review results are pushed into SonarQube so that you can assign, track, add comments – all the niceties associated with SonarQube platform. In addition to these, they provide an override platform to customize the output report. This helped us reduce the severity of some of the rules, and ignore the rules that we did not agree with.

How to automate TIBCO BW Code Review with Jenkins and SonarQube – The steps
  1. We setup the command-line version of BWCodeScanner and BECodeScanner in one of the Jenkins slave nodes. This setup is needed to break the build.
  2. Copy the BW SonarQube plugin into extensions directory on Sonarqube server
  3. Install Sonarqube Scanner plugin for Jenkins
  4. In the Jenkins job, under Execute Sonarqube Scanner, configure the Analysis properties to configure Sonarqube server, any BW Code Scanner specific properties like license, override file path etc.
  5. We created the build parameters for source location, reports location etc. and added Build step to execute BW Code Scanner command line. They showed us similar configuration for the Enterprise Cloud version too, but we went with command line version.
  6. As part of the build setup, we piped these steps, so that SonarQube execution happens, pushing the results to Sonarqube server, then based on the parameter that we pass, the build is broken in command line execution for a specific threshold. For example, we configured BW Code Scanner command line to break the build whenever it finds a code violation of severity – Critical or above.
  7. As part of the DevOps pipeline, this would run code review analysis on the BW and BE projects, push the results to SonarQube, break the build if the code has any critical violations (customized by us) from the developers, and emails the code review report as excel sheet. Pretty nifty right !
  8. One interesting thing to note is that BW Code Scanner and BE Code Scanner support multiple projects even with Sonarqube plugin. So, for certain SVN repositories, we setup one job that would do a complete code review analysis across multiple projects in the repository and send one consolidated report as email.
  9. Enterprise on-premise version came with web version in addition to command line and we could just drop the WAR file in Tomcat and have a functional website where someone could just review the Source zip file or EAR files manually to get a consolidated Excel or PDF report.
Sample Report - Excel Project Details






Sample Report – Sonarqube Issues Report



References

Friday, December 18, 2009

Creating a routed queue between two Tibco EMS Servers with Authorization enabled

Imagine a scenario of creating a routed queue between two Tibco EMS servers A and B with home queue on B. Pretty simple right,

On server A, just define

create queue Q1@B secure, failsafe, global

Not so simple with Authorization enabled on the two servers. For messages to flow on routed queue, ServerA has to have ServerB as an authorized user to receive and ServerB has to have ServerA as an authorized user to send. So to setup the route properly with Authorization enabled the following has to be worked upon. This assumes that the route is already in place and the user ServerB is created on ServerA and the user ServerA is created on ServerB and Q1 is setup as a global queue on Server B.

Routing between two servers A and B

Q1 is on B (home queue) and we need to create a routed queue on A and send to B

On server A

create queue Q1@B secure, failsafe, global
grant queue Q1 user=ServerB receive

On Server B

grant queue Q1 user=ServerA send

Tuesday, February 3, 2009

Configuring JDBC Datasource in Tomcat 5.0.28 in Eclipse WTP

I use Eclipse WTP for developing Web applications. In my previous post I mentioned that I changed the Database connection pool to use the Weblogic connection pool. The configuration in Weblogic was straightforward and simple, but it posed an issue for me during development because I was not able to configure the Datasource in the Installed Server Runtime in my Eclipse WTP. Either I had to use Reflection to supply the class and modify it between Environment deployments and local deployments or needed to fix the issue with Tomcat Datasource. The error that I was constantly getting was


ERROR DatabaseManager.getConnection - org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class '' for connect URL 'null'
org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class '' for connect URL 'null'
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:780)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
at com.pepsi.bdb.connections.DatabaseManager.getConnection(DatabaseManager.java:36)


Googling the error wasn't very fruitful because it mostly dealt with configuring Datasource in Tomcat directly, but not much information was available on configuring with Eclipse WTP. Also, there were several ways of configuring it mentioned in Serverside website and none seem to work for me. Finally I figured it out today and here is how I did

In Eclipse WTP, expand the configured Server under Servers, you will see a file server.xml. This is a copy of the server.xml file that Eclipse creates for you, but uses the libs from the Tomcat folder you configured. In this file add your Datasource under the tag <GlobalNamingResources> as shown below:


<GlobalNamingResources>
<Environment name="simpleValue" type="java.lang.Integer" value="30"/>
<Resource auth="Container" description="User database that can be updated and saved" name="UserDatabase" type="org.apache.catalina.UserDatabase"/>
<Resource name="myDbPool" type="javax.sql.DataSource"/>
<ResourceParams name="UserDatabase">
<parameter>
<name>factory</name>
<value>org.apache.catalina.users.MemoryUserDatabaseFactory</value>
</parameter>
<parameter>
<name>pathname</name>
<value>conf/tomcat-users.xml</value>
</parameter>
</ResourceParams>
<ResourceParams name="myDbPool">
<parameter>
<name>validationQuery</name>
<value>select * from dual</value>
</parameter>
<parameter>
<name>maxWait</name>
<value>5000</value>
</parameter>
<parameter>
<name>maxActive</name>
<value>4</value>
</parameter>
<parameter>
<name>password</name>
<value>(password)</value>
</parameter>
<parameter>
<name>url</name>
<value>jdbc:oracle:thin:@(servername):(portNum):(SID)</value>
</parameter>
<parameter>
<name>driverClassName</name>
<value>oracle.jdbc.driver.OracleDriver</value>
</parameter>
<parameter>
<name>maxIdle</name>
<value>2</value>
</parameter>
<parameter>
<name>username</name>
<value>(username)</value>
</parameter>
</ResourceParams>
</GlobalNamingResources>


Go to the folder where your server is actually located and place the jar file ojdbc14.jar (for Java 1.4, for lower versions, uses classes12.jar) in the folder %TOMCAT_HOME%\common\lib.

In the server.xml on the Eclipse, place an entry inside the <Context> element of your application that is inside the <Host> element as below:


<Context docBase="myApp" path="/myApp" reloadable="true" source="org.eclipse.jst.j2ee.server:myApp">
<ResourceLink name="myDbPool" global="myDbPool" type="javax.sql.DataSource"/>
</Context>


That is it. This did the trick for me. For the web.xml entry inside your web application and Java code, refer to my previous post.

Thursday, January 22, 2009

Configuring JDBC Datasource in Weblogic and using it in the code in a web application

I decided to leverage the Connection pooling features provided by Weblogic in one of the web applications that I wrote. This application uses plain JDBC since dome dynamic capabilities are required for sql statements and so we decided against Hibernate/Spring. Initially the code was written against Open source Snaq DB Connection pooling. Apparently, Weblogic has very good connection pooling features that an application can leverage. Since the code was object oriented and all DAO classes used a DBManager class to get their connection, all I needed to change was a method.

The Weblogic documentation shows how to configure a Data source on Weblogic but has little mention on how to get it working from your web application. The following details should help you set it up. To configure a Data source check the following Weblogic documentation.
http://edocs.bea.com/wls/docs90/jdbc_admin/jdbc_datasources.html.
In the Data source, I am assuming that you configured the JNDI name as "myAppDbPool"

In your web application, you will need a web.xml entry and an entry in a new file weblogic.xml.

The web.xml entry looks like below


<resource-ref>
<res-ref-name>myAppDbPool</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>


If you don't have a weblogic.xml file in your app create one. It goes in your WEB-INF beside web.xml file. The weblogic.xml file looks like below

<!DOCTYPE weblogic-web-app PUBLIC "-//BEA Systems, Inc.//DTD Web Application 8.1//EN"

"http://www.bea.com/servers/wls810/dtd/weblogic810-web-jar.dtd">

<weblogic-web-app>

<reference-descriptor>
<resource-description>
<res-ref-name>myAppDbPool</res-ref-name>
<jndi-name>myAppDbPool</jndi-name>
</resource-description>
</reference-descriptor>

</weblogic-web-app>


That's the configuration part. Now all you need to do is code your web application to retreive the Database connection from this Datasource. I made the DBManager class singleton to make sure only one instance runs at a given time. The DBManager class method looks like below:

public Connection getConnection() throws Exception
{
InitialContext ctx=new InitialContext();
DataSource ds=(DataSource)ctx.lookup("java:comp/env/myAppDbPool");
Connection connection=ds.getConnection();
return connection;
}


That's it. Use this connection in your DAO and you are all set. But make sure that your DAO object calls connection.close() when it is done accessing the data. According to Weblogic documentation, this returns the connection to the pool for other objects to access it.

If you have better approach or you see any flaws in this, let me know in the comments.

Friday, November 7, 2008

Oracle SQL group by on a portion of a clob field in the database

I work on an application which writes its internal errors in processing to a DB table. The input data that caused the failure due to a parsing error or missing configuration error is put in a clob column in the database. This input data is an XML message and one of the tags identifies each application that sends data to my application. We had one of the architects who requested us to give a split up on the number of errors that have occurred for each application.

This was significant from Oracle perspective because you cannot do a group by on a clob. In order to do a group by, you need to be able to sort and clob fields do not have a natural sort order. The workaround is to do a to_char function on the clob field but you can get a max of 4000 chars. For me, this suited the requirement because the substring I needed was a small string. But the issue is that I needed to sort a portion of an XML tag inside the clob field and group them based on that. Not only that I needed to find a specific type of error in the XML message and count them.

After a little food for thought, here is the final query that fetched the sweet results


SELECT k.application_name, COUNT (*)
FROM (SELECT TO_CHAR
(SUBSTR (clobfield,
( INSTR (clobfield,
'ns0:appName',
1,
1
)
+ 21
),
( INSTR (clobfield,
'lt;/ns0:appName',
1,
1
)
- INSTR (clobfield,
'ns0:appName',
1,
1
)
- 22
)
)
) AS application_name
FROM tableName
WHERE appID = 1
AND INSTR (clobfield, 'ns0:appName', 1, 1) <> 0
AND INSTR (clobfield, 'ErrorMsg', 1, 1) <> 0
AND INSTR (clobfield, 'Application Not Configured', 1, 1) <> 0) k
GROUP BY k.application_name


The INSTR function selects the index of the occurrence of a string inside another string, while substr gets the substring from a string. The numbers 21 and 22 account for the tag name lengths and the additional characters in the tag name like > etc.

If you need to count based a specific portion of a clob, just modify the above query and you are good to go..