Big + Far Math Challenge @ ICC

April 22, 2017


Recently I participated and won First prize in Big Far Math challenge hosted by ICC. The challenge description can be found here –

Participating in it was a quite exciting and learning experience for me. I could explore different technical areas while gathering data and preparing visualizations with it.

I have shared the source code and a static version of the visualization on GitHub. The dynamic version was hosted on Apache Solr running on my local desktop.

You can visit the project page @ from which you can navigate to the visualizations that I came up with.

I am also sharing the presentation given to the judges as part of assessment if you are looking for more details.

– Amit

Running Apache Spark on Windows

July 10, 2016

Running hadoop on windows is not trivial, however running Apache Spark on Windows proved not too difficult. I came across couple of blogs and stackoverflow discussion which made this possible. Putting down my notes below which are outcome of these reference material.

  1. Download ( )
  2. Download Hadoop distribution for Windows from
  3. Create hadoop_env.cmd  in {HADOOP_INSTALL_DIR}/conf directory.
    SET JAVA_HOME=C:\Progra~1\Java\jdk1.7.0_80
  4. In a new command window run hadoop-env.cmd followed by  {HADOOP_INSTALL_DIR}/bin/hadoop classpath
    The output of this command is used to initialize SPARK_DIST_CLASSPATH in spark-env.cmd (You may need to create this file.)
  5. Create spark-env.cmd in {SPARK_INSTALL_DIR}/conf
     #spark-env.cmd content
     SET HADOOP_HOME=C:\amit\hadoop\hadoop-2.6.0
     set SPARK_DIST_CLASSPATH=<Output of hadoop classpath>
     SET JAVA_HOME=C:\Progra~1\Java\jdk1.7.0_80
  6. Now run the examples or spark shell from {SPARK_INSTALL_DIR}/bin directory. Please note that you may have to run spark-env.cmd explicitly prior running the examples or spark-shell.

References :

Big O notation

May 14, 2016


Came across a very nice introductory article on Big O notation.

iBatis or Hibernate ?

December 16, 2011

I have been thinking to write this one since long. It almost now took 10 months to write the new blog entry.
In this one I will be talking about the iBatis and Hibernate. I purposely did not mention comparison as I myself do not think they are comparable to each other. Both of them have a different purpose and different applications in designing the solutions for Data access. While taking interviews I often come across candidates mentioning iBatis as a competitor of Hibernate. This exactly is the reason I felt I should share my views around the same.

iBatis is a very good and handy SQLMapper tool. I had experienced it’s usefulness while consuming the iBatis framework to build the solution that involved invoking Oracle Stored Procedures that returns REFERENCE cursors back. You simply configure the returned resultset to the Java objects and you are done. You get to work with the Java Objects in your application without bothering to write the usual and monotonous traversing logic and initializing the Java objects on your own. But this is where some colleagues starts thinking it as an ORM(Object Relational Mapping) tool. This is not the fact. iBatis  helps you map Java objects to the Parameters being passed to Stored PRocedure/Function/SQL without worrying about JDBC.

ORM has a altogether different domain to address. To understand this we have to first understand the challenges involved in mapping Object model to Relational model. This is usually called as Object Relational Impedance. We will see few of them hereafter.
First and foremost thing is Object Inheritance available in Object model which is not in Relational model. Hibernate as such addresses this by implementing Table per Class, Table per subclass and Table per Class hierarchy. We cannot go into more details of these and already a lot of information is available around this. The crux is when persisting/retrieving the Object hierarchy to/from database, either a discriminator column or table joins are implemented based on the strategy selection.
Another important feature that ORM tool provides is persisting the Object Graph when parent is being persisted. These are some of the things which are not addressed by iBatis. Well, you can do this with iBatis provided you write your own code for this. ORM tools also has a feature that helps identifying the dirty instance and persist it to the database once it is associated to Persistence context. Apart from this ORM has lot of other features which can be checked here.

In essence when we want to build a solution that involves just retrieving data from database and present it to the user, iBatis is simpler and effective solution. Whereas when one has the liberty to derive Relational model from the Object model and heavily retrieve/persist Object Graphs, Hibernate is appropriate solution. Using hibernate only for Data retrieval makes your solution over-engineered.

Hope this helps someone trying to compare iBatis and Hibernate.

Polymorphism without Inheritance

October 6, 2010

Can Polymorphism exist without Inheritance?

Abstraction“, “Encapsulation“, “Inheritance” and “Polymorphism” forms the basic building blocks of OOA&D.
We are so much overwhelmed by these facts that an immediate answer is “No“.

In Java there are two types of Polymorphism. One is Runtime(Dynamic) and other is compile time(Static).
In Runtime polymorphism the actual method to be invoked is decided based upon the actual object that the reference type refers at Runtime. This is achieved by OVERRIDING the method in sub classes. Of course this is where “Inheritance” is the KEY which is used to implement “Polymorphism”.

In Compile time or Static polymorphism, the method to be invoked is decided at compile time by diagnosing parameters of the method. This is implemented using Method OVERLOADING. Now this one does not depend on the Inheritance hierarchy because a method can be overloaded in different ways in the same class.

What does this mean?
Even without the Inheritance hierarchy we can have Polymorphism though of a Static kind.



Producer-Consumer using Java Threads – wait,notifyAll

April 27, 2010


In Java , threads communicate with each other using “wait,notify and notifyAll” APIs.  Producer-Consumer is the classic example to explain these APIs. The example below tries to show how we can use inter-thread communication to read shared data when it is available. I have also tried to demonstrate use of Thread Interrupt using which we can cancel the running task, in this case exiting the application.

 * @author Amit
 * This Producer-Consumer example tries to read/write one shared String variable.
 * This example also uses Thread interrupt concept to stop the application.
 * The variable in main() are defined final so that we can use them in
 * Anonymous inner class implementation of thread within main() method.
public class ConsumerProducerTest {

	static String data = new String();

	public static void main(String args[]) {

		//Declaring these variables as final so that
		//I can access them inside the Anonymous inner class
		//for user thread.
		final Object lock = new Object();

		final Thread readerThread = new Thread(new Reader(lock));
		final Thread writerThread = new Thread(new Writer(lock));

		//User Input Thread
		final BufferedReader in = new BufferedReader(new InputStreamReader(;
		new Thread(new Runnable() {
			public void run() {
				while(true) {
					try {
						String str = in.readLine();
						if("quit".equalsIgnoreCase(str)) {
					} catch (Exception e) {

class Reader implements Runnable {

	private Object lock;

	Reader(Object lock) {
		this.lock = lock;

	public void run() {
		while(true) {
			synchronized(lock) {
				if("".equals( {
					System.out.println("Reader is going to wait");
					try {
					} catch (InterruptedException e) {
						System.out.println("Reader thread interrupted !!");
					System.out.println("After call to Reader wait");
				} else {
					System.out.println("Reader got the data" +;
					System.out.println("Reader is notifying"); = ""; //Clear the data
					//Wait for 10 seconds before notifying other threads.
					try {
					} catch(InterruptedException ie) {
						System.out.println("Reader is interrupted while sleeping!!");
					System.out.println("Reader done notifying");

class Writer implements Runnable {

	private Object lock;

	Writer(Object lock) {
		this.lock = lock;

	public void run() {
		while(true) {
			synchronized(lock) {
				if("".equals( { = "data-"+System.currentTimeMillis();
						System.out.println("Writer wrote the data - " +;
						System.out.println("Writer is notifying reader");
						System.out.println("Writer done notifying");
				} else {
					try {
						System.out.println("Writer is going to wait");
						System.out.println("After call to Writer wait");
					} catch(InterruptedException ie) {
						System.out.println("Writer is interrupted !!");

Hope you find this useful.

Cheers !


Serialization – Compatible and Incompatible modifications to classes

April 15, 2010

Java Serialization allows us to write an Object into byte stream and again re-construct its state whenever needed.
In between Serialization and De-serialization the class definition might get changed. Some modifications in
the class definition would result in while De-serializing.
It is very important to understand these constraints. especially while dealing with Enterprise distributed Applications. As Distributed applications would always exchange the objects over the network stream using Serialization techniques.

Modifications those keep the Class compatible with Serialized version

  1. Add Fields
  2. Change Field from Static to Non-Static.
  3. Change Field from Transient to Non-Transient
  4. Add Class to the Object Tree

Modifications those make the Class incompatible with Serialized version

  1. Delete Field
  2. Change Field from Non-Static to Static
  3. Change Field from Non-Transient to Transient
  4. Changing the Class hierarchy
  5. Changing type of a Primitive Field

There are others changes as well those can be added to above list. They can be found here.
If you look at the above list the gist is “Addition of Field is permitted while Deletion cause Exception“.
Changing Field from Static to Non-static for transient to non-transient is kind of adding field to existing class.
While making non-static field to static and data-type of the field is like deleting the field which is incompatible.

This seems quite obvious from the Java Documentation but a beginner level would find this helpful, which is the reason
behind this post.

Cheers !


Intercepting HTTP Response using Servlet Filter

March 16, 2010

Servlet Filters are used to intercept the HTTP Requests/HTTP Responses.
Filters generally contain some logic that can be applied across a URL pattern.
I have used filters generally to verify if user is logged in or if user has appropriate privilege to access the resource.
While addressing such requirements we end up writing a Filter that is better known as Request Filter because it
intercepts the HTTP request. So most of the time when I ask someone if they know how to implement a HTTP Response Filter, they look puzzled. Of course,I too have gone through this situation in past.

Below is the sample code for Filter(Request Filter)  –

package com.amit.web.filter;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;

public class MyFilter implements Filter {

public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
// place your code here
 // pass the request along the filter chain
chain.doFilter(request, response);

public void init(FilterConfig arg0) throws ServletException {}
public void destroy() {}

You would know that the line “chain.doFilter(request,response)“, is responsible for executing the next Filter or web resource in line.
Hence if I map MyFilter for servlet named MyServlet then any request for MyServlet would be intercepted by MyFilter. When it encounters “chain.doFilter(request,response)“,it would call the MyServlet’s service method.
Once MyServlet has finished its task, the call to “doFilter” would return and if there is no code/logic written after this line, HTTP Response goes back to the client.
This precisely is the location where we can write the code that would alter the HTTP Response.
This means if we need to write a HTTP Response filter i.e. intercepting HTTP Response, then we need to write the business logic after “chain.doFilter(request,response)” and we are intercepting the HTTP response before it reaches the destination.  The method doFilter that would serve as HTTP Response filter is shown below –

public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
chain.doFilter(request, response);

Overall this is preety easy but usually goes unnoticed which is the reason I felt writing about this.

Another important point about Servlet Filters is –
Consider there are MyFilter1, MyFilter2 and MyFilter3 configured for any requests to Servlet MyServlet.
In this case MyFilter1, MyFilter2 and MyFilter3 will be executed in the order of configuration while intercepting the HTTP Requests.
While sending back the response they would be executed in the REVERSE order as shown in the diagram below.


Hope this helpes someone who is wondering how to implement HTTP Response Filters.Cheers !!

Decoupling Java applications and config/properties files

March 11, 2010

I have been developing a stand-alone Java application packaged as an executable JAR file.
The application had,

  1. Database interaction using iBatis APIs.
  2. Log the various activities in a log file using log4j API.

I had packaged the “” file that feeds the driver/username/password information to “SqlMapConfig.xml(iBatis configuration file)” within the JAR itself.Also the “” file was placed inside the JAR.

This introduced strong coupling/dependency between these files and JAR. Whenever I need to change database or log4j configuration, I had to re-package the JAR.

I was searching for a solution for this and based on google searches and interpretation of Java documentation for Class.getResourceAsStream() addressed this problem in the following way.


  1. Consider that we are generating this executable jar (myapp.jar) in  “myapp/dist” directory.
  2. To resolve dependencies on iBatis and log4j, put respective JARs in the same i.e “myapp/dist” directory.
  3. Add Class-Path attribute in “” file that lists these JAR files.
  4. This can be done by providing the custom manifest file while generating the JAR file. #2 and #3 are widely used in various enterprise as well as standalone applications so doing this is not an issue.

  5. Create a directory named “config” in “myapp/dist” where we will keep properties file. Of course this can be any name you like.
  6. Add the entry “config/” to the Class-Path attribute in “” file. This brings “config” directory in the classpath while executing the application.
  7. Put the “” and “” in “config” directory.
  8. Your application(may be constructor of your main class) can load the log4j configuration as shown below –
  9. 	/*Initialize log4j */
    	Properties props = new Properties();
  10. SqlMapConfig.xml” can point to the resource properties file as shown below. This does not change much as compared to log4j.
  11. <sqlMapConfig>
       <!-- This is where we specify the properties file where DB configuration is specified -->
       <properties resource=""/>
       <transactionManager type="JDBC" commitRequired="false">
       <dataSource type="SIMPLE">
         <property name="JDBC.Driver" value="${driver}"/>
         <property name="JDBC.ConnectionURL" value="${url}"/>
         <property name="JDBC.Username" value="${username}"/>
         <property name="JDBC.Password" value="${password}"/>
      <sqlMap resource="com/amit/test/xyz.xml"/>
  12. Now when you run your application (java -jar myapp.jar), the properties files will be picked up from the config directory.

This ensures you do not need to change the JAR file for making Database and log4j configuration changes.

The Directory structure for “dist” looks as shown below –

Directory Strcture for "dist": myapp |----dist | |----myapp.jar | |----ibatis- |----log4j-1.2.14.jar |----ojdbc14_g.jar |----config | | | classpath entry as below –

Class-Path: config/ log4j-1.2.14.jar ibatis- ojdbc14_g.jar

Why this works?
If you see the line that loads log4j configuration file, it has “/” at the start of filename.


As per the Java documentation(see below) for the method, we have to give “/” at the start and then only the resource will be located using absolute name calculated as mentioned.

I was facing issues earlier because,

  1. I was only giving “” while loading the log4j configuration in my program.
  2. In the, I was not appending “/” at the end of “config” directory. Without this the resource was not getting located.

Of course this is not just limited to iBatis and/or log4j, but any properties file that needs to be controlled outside the realm of the JAR file especially in stand-alone application can be configured this way.
Java API documentation to the rescue once again. Many thanks to them.

Cheers !!

Webapp generated Excel Spreadsheet opens in same browser window – IE6

March 8, 2010


While working on a web application that generates Excel spreadsheets , I faced the issue mentioned below.


Sometimes the spreadsheet is opened up in a new MS-Excel application window and often opened in the same browser window.  This is used to happen for Internet Explorer 6.0. On IE7 it was not reproducible i.e. used to open up in the MS-Excel application all the time leaving browser intact.

Additional Information:

I had implemented this spreadsheet generation as below –

  1. Generate the spreadsheet using Apache POI.
  2. Set header content-type to “application/“.
  3. In addition to content-type, I was also adding the header “Content-disposition” that assigns a filename to the content written on the response stream. Some excel files were named like “xyz/abc.xls“.
  4. Write the bytes on the Response stream.


We could figure out the behavior because when we ran the application and try to retrieve the report,

  1. For spreadsheets with filenames that did not contain “/” displayed the browser File Download dialog properly with “filename.xls”.
  2. Whereas if the filename contained “/” in it then the File Download dialog shows
  • EITHER the URL that is used to retrieve the excel report from server. e.g. if you have a ReportServlet that serves the excel report then it would show “ReportServlet” instead of filename.
  • OR some random filename as shown below


Once we fixed the filename generation logic to replace “/” with “-“(or any other valid character), reports started opening up in MS-Excel application than in same browser window.

Another thing to note here is that IE7 handles it internally by replacing “/” with “_” as shown below.

IE7-File Download Dialog


In addition to this while searching I found out the Microsoft knowledge-base which mentioned this issue can be resolved by doing some client side settings. This knowledge-base can be found at KB-162059.

Cheers !