Showing posts with label coding. Show all posts
Showing posts with label coding. Show all posts

Tuesday, February 5, 2019

TIBCO BW (BusinessWorks) and BE (BusinessEvents) Code Review Automation

As part of setting up DevOps pipeline for Continuous Integration / Continuous Deployment (CI – CD) for TIBCO BW (BusinessWorks) and BE (BusinessEvents), we used the BW Code Scanner and BE Code Scanner products from https://bwcodescanner.com. They are very comprehensive from code review perspective with 100+ rules for BW Code review and 75+ rules for BE Code Review. Automating this as part of DevOps pipeline helped us to focus the manual code reviews on Design and functional gaps.

BW Code Scanner and BE Code Scanner perform the static code review analysis for hard coded configurations, adherence to best practices, optimization opportunities and known patterns of defects. One interesting part about them is that they are very in-depth. BW Code Scanner, for example, traverses all the paths from each activity node in a process instance and evaluates the transitional conditions in all paths from Start to End while evaluating for missed configurations and missing transitions. In our case, we had a JMS Queue Receiver in client acknowledgment mode and it highlighted all the success and error paths from Start to End where there was no message confirmation. This helps to ensure that the message doesn’t end up in an infinite loop. In addition to this, it was doing dynamic global variable mapping, so, in the above case, even if the acknowledgment mode was configured as Global variable, it was able to detect based on value and highlight the paths where message confirmation is missing. Pretty comprehensive indeed.

Another interesting feature is SonarQube plugins for TIBCO BW 5.x, 6.x and BE. You can copy these plugins in the extensions folder of SonarQube and the code review results are pushed into SonarQube so that you can assign, track, add comments – all the niceties associated with SonarQube platform. In addition to these, they provide an override platform to customize the output report. This helped us reduce the severity of some of the rules, and ignore the rules that we did not agree with.

How to automate TIBCO BW Code Review with Jenkins and SonarQube – The steps
  1. We setup the command-line version of BWCodeScanner and BECodeScanner in one of the Jenkins slave nodes. This setup is needed to break the build.
  2. Copy the BW SonarQube plugin into extensions directory on Sonarqube server
  3. Install Sonarqube Scanner plugin for Jenkins
  4. In the Jenkins job, under Execute Sonarqube Scanner, configure the Analysis properties to configure Sonarqube server, any BW Code Scanner specific properties like license, override file path etc.
  5. We created the build parameters for source location, reports location etc. and added Build step to execute BW Code Scanner command line. They showed us similar configuration for the Enterprise Cloud version too, but we went with command line version.
  6. As part of the build setup, we piped these steps, so that SonarQube execution happens, pushing the results to Sonarqube server, then based on the parameter that we pass, the build is broken in command line execution for a specific threshold. For example, we configured BW Code Scanner command line to break the build whenever it finds a code violation of severity – Critical or above.
  7. As part of the DevOps pipeline, this would run code review analysis on the BW and BE projects, push the results to SonarQube, break the build if the code has any critical violations (customized by us) from the developers, and emails the code review report as excel sheet. Pretty nifty right !
  8. One interesting thing to note is that BW Code Scanner and BE Code Scanner support multiple projects even with Sonarqube plugin. So, for certain SVN repositories, we setup one job that would do a complete code review analysis across multiple projects in the repository and send one consolidated report as email.
  9. Enterprise on-premise version came with web version in addition to command line and we could just drop the WAR file in Tomcat and have a functional website where someone could just review the Source zip file or EAR files manually to get a consolidated Excel or PDF report.
Sample Report - Excel Project Details






Sample Report – Sonarqube Issues Report



References

Monday, May 2, 2016

Mapping Google App engine application to Namesilo domain

I just finished mapping a Google App engine application to a domain purchased from NameSilo.com. Needless to say, it was NOT an easy experience. Jotting down the steps in case, any one else plans to do the same and hopefully I can save you time.


  1. Go to AppEngine Dashboard and go to Settings page on left-hand side.
  2. Select Custom Domains page on the right hand page and click on Add Custom domain
  3. Under Add a New custom domain, in Step 1, give your domain name and click on Verify.
  4. This takes you to google webmasters page with link starting with https://www.google.com/webmasters/verification/verification?
  5. In Select your domain dropdown, select Other (last Option).
  6. Under Step 1 "Add the TXT record below to the DNS configuration for" copy the whole link starting with google-site-verification=xxxx. Keep this page open.
  7. Now, go to Namesilo domain manager and in the middle find "Add TXT/SPF" record link and paste the value in TEXT field. Leave HOSTNAME field blank. This is critical step and it took me a while to figure it out.
  8. Now go back to web master tools page and click Verify. Depending on your stars, it might take about 5 minutes for this to work and it might show error or unable to verify.
  9. Once this is done, go back to App Engine cloud console page. Sign out and sign back in. You should see that verification is done and the domain shows up under step 2.
  10. Under Step 2, select the domain in http://myDomainName.com. Under 3. it shows a bunch of A and AAAA records.
  11. Go back to Domain manager page on Namesilo and click on A and AAAA records and create these 8 entries. For each, even though google says host as @, leave it as blank as Namesilo rejects it. This also took me a while to figure out.
  12. Now you are all set. Type your domain name in new browser and it should work. If using chrome open incognito tab and type your domain name. It should now work since chrome caches and you probably tried in between when it was not working.
Forwarding www.myDomainName.com
  1. In App Engine settings page Under Step, there is sub-domain mapping link. Add www and click on Add. It shows CNAME record to be added. Now go to Namesilo and add this record there as is. CNAME as WWW and value as shown in the App Engine settings page. This did not work after creating this record for me, so I am not sure if this step is required.
  2. Now go back to Domain manager page, select the check box in front of domain, and click on Forward Domains above that gets activated on selecting check box.
  3. Now, scroll down and under forwarding address, fill as www.myDomainName.com and choose http in dropdown before.
  4. Select Forwarding method as permanent forwarding and Path Forwarding as Yes and click Submit.
Congratulations!! Now you are all set to access your app engine application at http://myDomainName.com and www.myDomainName.com

Update

It turns out the above forwarding for www broke the Entire thing, since I started getting Error 301, too many redirects. What happened was that NameSilo removed A entries and CNAME entry from above steps with their own copies. I had to delete those entries and recreate those entries for this work. I could not find a way to remove the forwarding to www.myDomainName.com but I see from NameSilo pages that when you edit DNS entries, forwarding is disabled automatically even though I see the Domain forwarding page with entry to www.myDomainName.com. I am getting mixed results with forwarding based on browser cache, but it is finally working both for naked domain - http://myDomainName.com and www.myDomainName.com, so I am all set.





Friday, June 6, 2014

Google App engine application with godaddy domain mapping issue

Recently, I had an issue with mapping a domain to the Google App engine application that I had built. 
I bought a domain - cinemahunter.com and setup forwarding on godaddy to redirect from http://cinemahunter.com to http://www.cinemahunter.com
In Google App Engine, I clicked on settings, and then Add Domain. It asked me for Google Apps, gave my domain name, it asked me to login to godaddy.com, I logged in successfully with the credentials and it was good. At this point, there are 2 steps outlined.
  1. Change how your naked domain (cinemahunter.com) is redirected. I have www in the textbox between http:// and .cinemahunter.com. It also said - You must change the A-record at your domain host for this change to take effect. I am assuming this is the forwarding that I setup in Godaddy. I clicked on continue
  2. Now it said to create a A record in DNS management page. I went to Godaddy and under A (host) I deleted the entry with IP - 50.63.202.22 and created 4 records with the following as specified in Google apps for Host @ points to 216.239.32.21, 216.239.34.21, 216.239.36.21, 216.239.38.21
I saved the changes twice in Godaddy. Also, I verified that under CName(Alias) www points to @.
At this point, I clicked on I completed the steps on Google Apps website. But when I try to access my website by typing www.cinemahunter.com or http://cinemahunter.com, I get the following error.
Google
  1. That’s an error.
The requested URL / was not found on this server. That’s all we know.
After hours of going round and round with several links from google searches, I figured out that most of the documentation was outdated for the domain mapping.
So for anyone else, having this issue, here is the resolution.
Google Apps is no longer free. You have to pay about $60 a year for the service. For Google App engine users, google started giving $50 upto maximum of 3 because of this issue. They stopped it after having a fix for directly mapping Google app engine to domain. Now you no longer need Google Apps and what you need to do is to go to https://console.developers.google.com and select your application. On the left hand side, click under App Engine --> settings. This will give you three steps
  1. If you haven't already, verify that you own the domain. It will then be added to the list in step 2. --> Give your domain name here and verify. Now it will take you to godaddy login page and you sign in with your credentials.
  2. Select the domain you want to point to. This will show nodomain even after domain is verified in step 1 below. This confused me. But I logged out and logged in and I saw the domain cinemahunter.com in the dropdown. There are two options here either the main domain or the sub-domain with www. I chose the sub-domain since I had forwarding setup in godaddy.
  3. Go to godaddy and under CName, under www change points to ghs.googlehosted.com.
Thats it!! It worked like a charm within a minute. Google makes it so difficult with all these changes. All the search results for mapping GAE to godaddy were old instructions which led to nowhere and the info on these changes were to be found only after digging deeply. Hopefully this will help someone without wasting some 8 hours like for me. Good luck !!

Friday, December 18, 2009

Creating a routed queue between two Tibco EMS Servers with Authorization enabled

Imagine a scenario of creating a routed queue between two Tibco EMS servers A and B with home queue on B. Pretty simple right,

On server A, just define

create queue Q1@B secure, failsafe, global

Not so simple with Authorization enabled on the two servers. For messages to flow on routed queue, ServerA has to have ServerB as an authorized user to receive and ServerB has to have ServerA as an authorized user to send. So to setup the route properly with Authorization enabled the following has to be worked upon. This assumes that the route is already in place and the user ServerB is created on ServerA and the user ServerA is created on ServerB and Q1 is setup as a global queue on Server B.

Routing between two servers A and B

Q1 is on B (home queue) and we need to create a routed queue on A and send to B

On server A

create queue Q1@B secure, failsafe, global
grant queue Q1 user=ServerB receive

On Server B

grant queue Q1 user=ServerA send

Tuesday, February 3, 2009

Configuring JDBC Datasource in Tomcat 5.0.28 in Eclipse WTP

I use Eclipse WTP for developing Web applications. In my previous post I mentioned that I changed the Database connection pool to use the Weblogic connection pool. The configuration in Weblogic was straightforward and simple, but it posed an issue for me during development because I was not able to configure the Datasource in the Installed Server Runtime in my Eclipse WTP. Either I had to use Reflection to supply the class and modify it between Environment deployments and local deployments or needed to fix the issue with Tomcat Datasource. The error that I was constantly getting was


ERROR DatabaseManager.getConnection - org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class '' for connect URL 'null'
org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class '' for connect URL 'null'
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:780)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
at com.pepsi.bdb.connections.DatabaseManager.getConnection(DatabaseManager.java:36)


Googling the error wasn't very fruitful because it mostly dealt with configuring Datasource in Tomcat directly, but not much information was available on configuring with Eclipse WTP. Also, there were several ways of configuring it mentioned in Serverside website and none seem to work for me. Finally I figured it out today and here is how I did

In Eclipse WTP, expand the configured Server under Servers, you will see a file server.xml. This is a copy of the server.xml file that Eclipse creates for you, but uses the libs from the Tomcat folder you configured. In this file add your Datasource under the tag <GlobalNamingResources> as shown below:


<GlobalNamingResources>
<Environment name="simpleValue" type="java.lang.Integer" value="30"/>
<Resource auth="Container" description="User database that can be updated and saved" name="UserDatabase" type="org.apache.catalina.UserDatabase"/>
<Resource name="myDbPool" type="javax.sql.DataSource"/>
<ResourceParams name="UserDatabase">
<parameter>
<name>factory</name>
<value>org.apache.catalina.users.MemoryUserDatabaseFactory</value>
</parameter>
<parameter>
<name>pathname</name>
<value>conf/tomcat-users.xml</value>
</parameter>
</ResourceParams>
<ResourceParams name="myDbPool">
<parameter>
<name>validationQuery</name>
<value>select * from dual</value>
</parameter>
<parameter>
<name>maxWait</name>
<value>5000</value>
</parameter>
<parameter>
<name>maxActive</name>
<value>4</value>
</parameter>
<parameter>
<name>password</name>
<value>(password)</value>
</parameter>
<parameter>
<name>url</name>
<value>jdbc:oracle:thin:@(servername):(portNum):(SID)</value>
</parameter>
<parameter>
<name>driverClassName</name>
<value>oracle.jdbc.driver.OracleDriver</value>
</parameter>
<parameter>
<name>maxIdle</name>
<value>2</value>
</parameter>
<parameter>
<name>username</name>
<value>(username)</value>
</parameter>
</ResourceParams>
</GlobalNamingResources>


Go to the folder where your server is actually located and place the jar file ojdbc14.jar (for Java 1.4, for lower versions, uses classes12.jar) in the folder %TOMCAT_HOME%\common\lib.

In the server.xml on the Eclipse, place an entry inside the <Context> element of your application that is inside the <Host> element as below:


<Context docBase="myApp" path="/myApp" reloadable="true" source="org.eclipse.jst.j2ee.server:myApp">
<ResourceLink name="myDbPool" global="myDbPool" type="javax.sql.DataSource"/>
</Context>


That is it. This did the trick for me. For the web.xml entry inside your web application and Java code, refer to my previous post.

Thursday, January 22, 2009

Configuring JDBC Datasource in Weblogic and using it in the code in a web application

I decided to leverage the Connection pooling features provided by Weblogic in one of the web applications that I wrote. This application uses plain JDBC since dome dynamic capabilities are required for sql statements and so we decided against Hibernate/Spring. Initially the code was written against Open source Snaq DB Connection pooling. Apparently, Weblogic has very good connection pooling features that an application can leverage. Since the code was object oriented and all DAO classes used a DBManager class to get their connection, all I needed to change was a method.

The Weblogic documentation shows how to configure a Data source on Weblogic but has little mention on how to get it working from your web application. The following details should help you set it up. To configure a Data source check the following Weblogic documentation.
http://edocs.bea.com/wls/docs90/jdbc_admin/jdbc_datasources.html.
In the Data source, I am assuming that you configured the JNDI name as "myAppDbPool"

In your web application, you will need a web.xml entry and an entry in a new file weblogic.xml.

The web.xml entry looks like below


<resource-ref>
<res-ref-name>myAppDbPool</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>


If you don't have a weblogic.xml file in your app create one. It goes in your WEB-INF beside web.xml file. The weblogic.xml file looks like below

<!DOCTYPE weblogic-web-app PUBLIC "-//BEA Systems, Inc.//DTD Web Application 8.1//EN"

"http://www.bea.com/servers/wls810/dtd/weblogic810-web-jar.dtd">

<weblogic-web-app>

<reference-descriptor>
<resource-description>
<res-ref-name>myAppDbPool</res-ref-name>
<jndi-name>myAppDbPool</jndi-name>
</resource-description>
</reference-descriptor>

</weblogic-web-app>


That's the configuration part. Now all you need to do is code your web application to retreive the Database connection from this Datasource. I made the DBManager class singleton to make sure only one instance runs at a given time. The DBManager class method looks like below:

public Connection getConnection() throws Exception
{
InitialContext ctx=new InitialContext();
DataSource ds=(DataSource)ctx.lookup("java:comp/env/myAppDbPool");
Connection connection=ds.getConnection();
return connection;
}


That's it. Use this connection in your DAO and you are all set. But make sure that your DAO object calls connection.close() when it is done accessing the data. According to Weblogic documentation, this returns the connection to the pool for other objects to access it.

If you have better approach or you see any flaws in this, let me know in the comments.

Friday, January 9, 2009

Errors when using Struts 1.3.x that are not intuitive javax.servlet.ServletException: org/apache/commons/lang/UnhandledException

Today, I decided to upgrade a new application that I was working on to Struts 1.3.10. After getting the necessary jars and stuff from the struts-blank.war, and upgrading my webapp library with those, everything was fine until I ran into this error. Apparently, Struts 1.3.x removed some jars from the distribution, and the dependecies had to be resolved externally. The error gave me no clue that it was unable to find the commons-lang.jar. After struggling with it for sometime, I finally figured out that the commons-lang.jar was the missing jar. Usually when a jar is missing we get a NoClassDefFoundError that points in the right way, but the error trace for this gave no clue at all. Anyways if you happen to run into this error the key is to have the commons-lang.jar in your classpath from this location Apache Commons


2009-01-09 17:30:34,001 ERROR InsertTag$InsertHandler.doEndTag - ServletException in '/pages-bdb/batchListDetail.jsp': org/apache/commons/lang/UnhandledException
javax.servlet.ServletException: org/apache/commons/lang/UnhandledException
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:244)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:704)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:590)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:510)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:966)
at org.apache.jasper.runtime.PageContextImpl.doInclude(PageContextImpl.java:610)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:604)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:129)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:152)
at org.apache.struts.tiles.taglib.InsertTag.doInclude(InsertTag.java:764)
at org.apache.struts.tiles.taglib.InsertTag$InsertHandler.doEndTag(InsertTag.java:896)
at org.apache.struts.tiles.taglib.InsertTag.doEndTag(InsertTag.java:465)
at org.apache.jsp.pages.shared.template_jsp._jspx_meth_tiles_insert_2(template_jsp.java:173)
at org.apache.jsp.pages.shared.template_jsp._jspService(template_jsp.java:92)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:324)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:704)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:474)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:409)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1083)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:295)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:271)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:332)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:232)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534)

Friday, November 7, 2008

Oracle SQL group by on a portion of a clob field in the database

I work on an application which writes its internal errors in processing to a DB table. The input data that caused the failure due to a parsing error or missing configuration error is put in a clob column in the database. This input data is an XML message and one of the tags identifies each application that sends data to my application. We had one of the architects who requested us to give a split up on the number of errors that have occurred for each application.

This was significant from Oracle perspective because you cannot do a group by on a clob. In order to do a group by, you need to be able to sort and clob fields do not have a natural sort order. The workaround is to do a to_char function on the clob field but you can get a max of 4000 chars. For me, this suited the requirement because the substring I needed was a small string. But the issue is that I needed to sort a portion of an XML tag inside the clob field and group them based on that. Not only that I needed to find a specific type of error in the XML message and count them.

After a little food for thought, here is the final query that fetched the sweet results


SELECT k.application_name, COUNT (*)
FROM (SELECT TO_CHAR
(SUBSTR (clobfield,
( INSTR (clobfield,
'ns0:appName',
1,
1
)
+ 21
),
( INSTR (clobfield,
'lt;/ns0:appName',
1,
1
)
- INSTR (clobfield,
'ns0:appName',
1,
1
)
- 22
)
)
) AS application_name
FROM tableName
WHERE appID = 1
AND INSTR (clobfield, 'ns0:appName', 1, 1) <> 0
AND INSTR (clobfield, 'ErrorMsg', 1, 1) <> 0
AND INSTR (clobfield, 'Application Not Configured', 1, 1) <> 0) k
GROUP BY k.application_name


The INSTR function selects the index of the occurrence of a string inside another string, while substr gets the substring from a string. The numbers 21 and 22 account for the tag name lengths and the additional characters in the tag name like > etc.

If you need to count based a specific portion of a clob, just modify the above query and you are good to go..

Thursday, August 7, 2008

Log4j writing to a dynamic log file for every run

In my recent project, I was using Log4J and I had a requirement where I had to write to a new dynamic log file every time and the name of the log file was determined at runtime. In specific terms each run would produce a Project and the log file had to reflect the project name. Since this project name could be repeated across runs, the timestamp needed to be added to the log file. I searched across Google and didn't find much help in this regard. So I decided to post the code I wrote.

I didn't want to lose setting the log levels from the log4j.xml file and wanted all the options I could configure for the FileAppender except the file name. The file name was configured too, but the code had to overwrite and create a new file at runtime. Here is the code..



Date projDate = new Date(Long.parseLong(project.getTimeStamp()));
StringBuffer dateStr = new StringBuffer();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy_MM_dd_HH_mm_ss");
dateStr = sdf.format(projDate, dateStr, new FieldPosition(0));
String logFileName = logsDir + File.separator + project.getName() + "_" + dateStr.toString() + ".log";
log.info("\n**************Log file for this run: " + logFileName + "\n**************\n");

Logger rootLogger = Logger.getRootLogger();
Enumeration appenders = rootLogger.getAllAppenders();
FileAppender fa = null;
while(appenders.hasMoreElements())
{
Appender currAppender = (Appender) appenders.nextElement();
if(currAppender instanceof FileAppender)
{
fa = (FileAppender) currAppender;
}
}
if(fa != null)
{
fa.setFile(logFileName);
fa.activateOptions();
}
else
{
log.info("No File Appender found");
}



That's it. The logs were created for each and parallel runs would write to different files...

Wednesday, July 16, 2008

Java: Loading an XML file from the CLASSPATH

Today, I faced this issue in Java coding. I needed to load an XML file which was in my classpath but not in the same directory as the classes. The deliverable was a jar and the properties and the configuration XML files were in a different folder and were appended to the classpath at the runtime. The issue was that the file could not be located through the the statement


String fileName = getClass.getSystemResource("config.xml").getFile;


Then I thought that it checks relative to the current class and so used the statement


String fileName = ClassLoader.getSystemResource("config.xml").getFile;


But then, it wouldn't still recognize the file. The reason is the same, it checks relative to the classes folder. I did not want to get the classpath from the system and browse through it for the config file since the class path could get larger. A couple of google searches and a little research later, I found the solution. The workaround is by using the following statement


String fileName = Thread.currentThread().getContextClassLoader().getResource("config.xml").getFile;


It only makes sense since in the above statement, you get hold of the context class loader and find the path in the entire classpath...

Wednesday, May 14, 2008

Power of Java, In deleting the files that Windows cannot

Today, in one of the programs, I messed up a little piece of code that creates a folder in another, which ended up creating folders in an infinite loop. The folder name was ae, but there were so many of them ae in ae in ae .... Then I just wanted to clear the folder and start afresh with the fixed code. But guess what. Windows would not delete the folder because the name is too long. This was the exact error..

"cannot delete the file name you specified is not valid or too long"

I tried from Command prompt and tried other solutions from Google search like assigning a name on shared and trying to delete from network. And nothing worked. That's when I decided, let's take Java's help. Wrote a simple piece of code to loop through until it reaches the final folder and deletes everything. Ran the code and 2 minutes later, everything was gone. One more reason to love Java and being a programmer. Here is the code in case you are interested..



import java.io.File;
public class FileDelete
{

private void deleteFiles(File file)
{
if(file.isDirectory())
{
File child[] = file.listFiles();
for(int i = 0; i < child.length; i++)
{
deleteFiles(child[i]);
}

}
if(!file.delete())
{
System.out.println("Cannot delete file: " + file.getAbsolutePath());
}
}
public static void main(String[] args)
{
System.out.println("Attempting to delete files");
File _work = new File("");
FileDelete fd = new FileDelete();
fd.deleteFiles(_work);
}
}


Monday, April 28, 2008

Too smart or too novice in Java: Relevance of for loops

During the course of my work, I have had to fix some code written by others and modify existing code for enhanced functionality. In one of such projects, I encountered this piece of code written by someone


for(boolean valid = false; !valid; valid = true)
{
try
{
sp = spf.newSAXParser();
parser = new ProjectParser();
sp.parse(projectFile, parser);
}
catch(Exception e)
{
e.printStackTrace();
log.throwable(e);
throw new RuntimeException((new StringBuilder("Unable to parse ")).append(projectFile.getAbsoluteFile()).toString());
}
}


Let's see how this for loop works. It initializes a boolean called valid with a value false. The condition is until valid is false. The increment is making valid true. And then inside the loop, the coder wants to accomplish something. Now what is the point of this for loop at all. Was the for loop used because the coder didn't know that loops are used for, you guessed right, looping. If he/she just wanted it to run only once, then what is the point of using a for loop at all. How different is the below code from the above code.


try
{
sp = spf.newSAXParser();
parser = new ProjectParser();
sp.parse(projectFile, parser);
}
catch(Exception e)
{
e.printStackTrace();
log.throwable(e);
throw new RuntimeException((new StringBuilder("Unable to parse ")).append(projectFile.getAbsoluteFile()).toString());
}


This would execute the code only once and that is the desired behavior that was accomplished by using the for loop for once. I think keeping your code clean and staying away from such unnecessary stuff helps you in keeping your code clean and reducing any unwanted bugs. Why turn the hand around the neck to eat something when you could directly eat it.. Is this trying to be too smart or being too much of a novice to know what loops are used for. Or is there something, that I am missing that is something great that I haven't realized in using this type of code. If someone could elaborate, I would be glad...

Wednesday, March 12, 2008

Overcoming cross domain issues through php proxy server in Flex for RSS Reader samples

Today, I took a second look at my home page and decided that the blog link should not directly point to this page and should have an in-built RSS Reader for getting these posts there. And then the troubles started. People who know me already know my website and know that it was built in Flex. I had this idea of an in-built RSS Reader in my home page for long, but whenever I started working on it, I would have weird exceptions on page load, which were not straight forward and I wouldn't have time to debug them. So it remained the same, a hyperlink to this blog. Today a post on DZone caught my attention and I again tried this. I found a simple example which I implemented in my home page and thankfully everything went cool. But when I put it on the server, another problem started. The cross-domain problem..

The thing about flex is that, while being cool and xml dominated, the biggest problem is with the fact that you cannot load the rss or xml from other websites, without a cross-domain.xml file on the serving domain that lists the calling domain. Now I cannot ask blogger.com to put my domain name in their cross-domain.xml file. Can I ?? So I preferred the alternative route, which is widely written on the net, the proxy server route. And though I do not have Java hosting by my space provider, they support php and went for a simple php script..

<?php
header('Content-Type: text/xml');
$url = 'http://cognitivecache.blogspot.com/rss.xml';
$content = file_get_contents($url);
echo $content;
?>

The script looks so simple like a no-brainer right. Nope, that wouldn't still solve my problem. Because, the blogger wouldn't let the php script open a stream and I kept ending up with this error

Warning: readfile [function.readfile]: failed to open stream: No route to host..

After changing the scripts from the net thrice hoping the other one would be different, I kept coming back to the same problem. Finally I realized that it could be an issue with Blogger not allowing an incoming connection. Boom, that was it. I changed the code to point to my feed in FeedBurner and that was it. My new code looks like this

<?php
header('Content-Type: text/xml');
$url = 'http://feeds.feedburner.com/CognitiveCache';
$content = file_get_contents($url);
echo $content;
?>

Now, The RSS Reader gets its content now and it looks like a nice addition. A Happy ending to lot of time spent indeed !!

If you are still wondering about the whole process, what you need to do, is to burn a feed in FeedBurner and put the above code in a file called proxy.php. Now upload this file to your domain and use the url for xml in your flash file as "http://mydomain.com/proxy.php" and thats it..

Wednesday, October 10, 2007

statement.setMaxrows in JDBC vs Oracle rownum in SQL

If you want to limit the number of rows retrieved from an Oracle database in a Java/Servlet application using JDBC, you can accomplish this in two ways. One is using statement.setMaxRows in the JDBC or using rownum in the Oracle SQL.

For example, lets say there are 1000 employee records in the EMPLOYEES table which has the columns: empid, first_name and last_name. In order to retreive the first 200 rows, in java you can say

....

Connection dbConn = db.getConnection();
Statement stmt = dbConn.createStatement();
stmt.setMaxRows(200);
String sql = "select empid, first_name, last_name from employees";
ResultSet rs = stmt.executeQuery(sql);

...............

This gets you a resultset with 200 rows and everybody is happy. You could have alternatively done that using rownum on your sql query itself without using setMaxRows() on statement like below:

....

Connection dbConn = db.getConnection();
Statement stmt = dbConn.createStatement();
String sql = "select empid, first_name, last_name from employees where rownum<201";
ResultSet rs = stmt.executeQuery(sql);
...............

Even this time, the resultset has 200 rows and everybody is happy. Now the question is, which one should you use and why. I may not be an expert in talking about this, but here are my observations...

If you are dealing with a table which has relatively less records like in thousands, then either way works just fine. But if you are dealing with potentially huge database which has like millions of records, then I guess using rownum in the sql makes more sense. This is because the impact on your Java heap space would be lesser.

According to JDBC, the implementation of setMaxRows is as given below:
-----------------------------------
The maxRows limit is set to limit the number of rows that any ResultSet can contain. If the limit is exceeded, the excess rows are silently dropped.
------------------------------------

For example, lets say there are 14 million records in a table, from which you want to retrieve the first 3000 rows. If you use statement.setMaxRows(), the JDBC call retreives the entire 14 million records from the database table and sets the first 3000 records on the resultset object. But the retrieval of such huge number of records puts a huge lump of data on your JVM, and you are dependent on the Garbage Collector for retrieval of this space. But if you use rownum in the SQL, then the number of rows retrieved from the DB by JDBC itself is limited and there won't be any heap space issues. I ran a trial session with our DBA and it was confirmed that the statement.setMaxRows() doesn't modify the sql using rownum automatically. We traced the session and it was retrieving all the million records. And I came to a conclusion that the heap space issues I was facing was due to the usage of statement.setMaxRows() instead of using rownum on the SQL. Also the speed of retrieval of rows when you limit by rownum could be more, though I don't have substantial evidence to state it as a fact.

If you are an expert in Java / Oracle Databases and feel that part/total of this post doesn't make any sense or if it looks like complete bullshit, then please feel to share your expertise on this matter / bash me in the Comments:)

Monday, September 3, 2007

Fundamental mistake in Digg web page coding or fooling Diggers??

Everyone knows how famous Digg is today, on the web. So much that Time magazine rated 'You' as the person of the year because of the user power created by Digg. And Digg has opened gates for Developers by providing Digg APIs and contests for the best mashup. All these made me think great of Digg and the way the website is coded, but this fundamental flaw in coding the Digg page leaves me stunned. Is it an oversight or an attmept to fool the users??

Here is what I mean. Find the news dugg by any user on Digg by clicking on their link. I am showing the example with the news dugg by me.



As you can see at the bottom, where I circled, there are links to more than 10 pages with the Next button after 10 ... But if I click on 9th page for example, here is what I see..



Forget about 9th page, if I click on 4th page, here is what it shows..



So, basically, I had dugg 3 pages worth of news, but even in the second page, it shows me links till 10 pages and more as if I had dugg news worth more than 10 pages. Any programmer who has done basic programming would know how easy it is, to control the number of links for pagination. Calculate the number of diggs a user has, divide it by the number of links you show on each page to figure how many links to show for pagination.

I agree that it may not be a show stopper and it might merely be a small bug, but with the Digg team implementing new comments system and opening up APIs, I would surely expect higher standards of coding. Not impressed much Digg team.. So as I said above, is this just an oversight or an attempt to show off a lot of posts to the Digg users. I would side with the former but still expect them to fix this..