Big + Far Math Challenge @ ICC

April 22, 2017

Hello,

Recently I participated and won First prize in Big Far Math challenge hosted by ICC. The challenge description can be found here – http://big-far.webflow.io/

Participating in it was a quite exciting and learning experience for me. I could explore different technical areas while gathering data and preparing visualizations with it.

I have shared the source code and a static version of the visualization on GitHub. The dynamic version was hosted on Apache Solr running on my local desktop.

You can visit the project page @ https://amitlondhe.github.io/bigfarmathchallenge/ from which you can navigate to the visualizations that I came up with.

I am also sharing the presentation given to the judges as part of assessment if you are looking for more details.

– Amit

Advertisements

Running Apache Spark on Windows

July 10, 2016

Running hadoop on windows is not trivial, however running Apache Spark on Windows proved not too difficult. I came across couple of blogs and stackoverflow discussion which made this possible. Putting down my notes below which are outcome of these reference material.

  1. Download http://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-without-hadoop.tgz ( http://spark.apache.org/downloads.html )
  2. Download Hadoop distribution for Windows from http://www.barik.net/archive/2015/01/19/172716/
  3. Create hadoop_env.cmd  in {HADOOP_INSTALL_DIR}/conf directory.
    SET JAVA_HOME=C:\Progra~1\Java\jdk1.7.0_80
    
  4. In a new command window run hadoop-env.cmd followed by  {HADOOP_INSTALL_DIR}/bin/hadoop classpath
    The output of this command is used to initialize SPARK_DIST_CLASSPATH in spark-env.cmd (You may need to create this file.)
  5. Create spark-env.cmd in {SPARK_INSTALL_DIR}/conf
     #spark-env.cmd content
     SET HADOOP_HOME=C:\amit\hadoop\hadoop-2.6.0
     SET HADOOP_CONF_DIR=%HADOOP_HOME%\conf
     set SPARK_DIST_CLASSPATH=<Output of hadoop classpath>
     SET JAVA_HOME=C:\Progra~1\Java\jdk1.7.0_80
    
  6. Now run the examples or spark shell from {SPARK_INSTALL_DIR}/bin directory. Please note that you may have to run spark-env.cmd explicitly prior running the examples or spark-shell.

References :


Big O notation

May 14, 2016

Hello,

Came across a very nice introductory article on Big O notation.

https://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/


iBatis or Hibernate ?

December 16, 2011

Hello,
I have been thinking to write this one since long. It almost now took 10 months to write the new blog entry.
In this one I will be talking about the iBatis and Hibernate. I purposely did not mention comparison as I myself do not think they are comparable to each other. Both of them have a different purpose and different applications in designing the solutions for Data access. While taking interviews I often come across candidates mentioning iBatis as a competitor of Hibernate. This exactly is the reason I felt I should share my views around the same.

iBatis is a very good and handy SQLMapper tool. I had experienced it’s usefulness while consuming the iBatis framework to build the solution that involved invoking Oracle Stored Procedures that returns REFERENCE cursors back. You simply configure the returned resultset to the Java objects and you are done. You get to work with the Java Objects in your application without bothering to write the usual and monotonous traversing logic and initializing the Java objects on your own. But this is where some colleagues starts thinking it as an ORM(Object Relational Mapping) tool. This is not the fact. iBatis  helps you map Java objects to the Parameters being passed to Stored PRocedure/Function/SQL without worrying about JDBC.

ORM has a altogether different domain to address. To understand this we have to first understand the challenges involved in mapping Object model to Relational model. This is usually called as Object Relational Impedance. We will see few of them hereafter.
First and foremost thing is Object Inheritance available in Object model which is not in Relational model. Hibernate as such addresses this by implementing Table per Class, Table per subclass and Table per Class hierarchy. We cannot go into more details of these and already a lot of information is available around this. The crux is when persisting/retrieving the Object hierarchy to/from database, either a discriminator column or table joins are implemented based on the strategy selection.
Another important feature that ORM tool provides is persisting the Object Graph when parent is being persisted. These are some of the things which are not addressed by iBatis. Well, you can do this with iBatis provided you write your own code for this. ORM tools also has a feature that helps identifying the dirty instance and persist it to the database once it is associated to Persistence context. Apart from this ORM has lot of other features which can be checked here.

In essence when we want to build a solution that involves just retrieving data from database and present it to the user, iBatis is simpler and effective solution. Whereas when one has the liberty to derive Relational model from the Object model and heavily retrieve/persist Object Graphs, Hibernate is appropriate solution. Using hibernate only for Data retrieval makes your solution over-engineered.

Hope this helps someone trying to compare iBatis and Hibernate.


Polymorphism without Inheritance

October 6, 2010

Can Polymorphism exist without Inheritance?

Abstraction“, “Encapsulation“, “Inheritance” and “Polymorphism” forms the basic building blocks of OOA&D.
We are so much overwhelmed by these facts that an immediate answer is “No“.

In Java there are two types of Polymorphism. One is Runtime(Dynamic) and other is compile time(Static).
In Runtime polymorphism the actual method to be invoked is decided based upon the actual object that the reference type refers at Runtime. This is achieved by OVERRIDING the method in sub classes. Of course this is where “Inheritance” is the KEY which is used to implement “Polymorphism”.

In Compile time or Static polymorphism, the method to be invoked is decided at compile time by diagnosing parameters of the method. This is implemented using Method OVERLOADING. Now this one does not depend on the Inheritance hierarchy because a method can be overloaded in different ways in the same class.

What does this mean?
Even without the Inheritance hierarchy we can have Polymorphism though of a Static kind.

Cheers,

Amit


Producer-Consumer using Java Threads – wait,notifyAll

April 27, 2010

Hello,

In Java , threads communicate with each other using “wait,notify and notifyAll” APIs.  Producer-Consumer is the classic example to explain these APIs. The example below tries to show how we can use inter-thread communication to read shared data when it is available. I have also tried to demonstrate use of Thread Interrupt using which we can cancel the running task, in this case exiting the application.

import java.io.BufferedReader;
import java.io.InputStreamReader;
/**
 * @author Amit
 * This Producer-Consumer example tries to read/write one shared String variable.
 * This example also uses Thread interrupt concept to stop the application.
 * The variable in main() are defined final so that we can use them in
 * Anonymous inner class implementation of thread within main() method.
 */
public class ConsumerProducerTest {

	static String data = new String();

	public static void main(String args[]) {

		//Declaring these variables as final so that
		//I can access them inside the Anonymous inner class
		//for user thread.
		final Object lock = new Object();

		final Thread readerThread = new Thread(new Reader(lock));
		final Thread writerThread = new Thread(new Writer(lock));
		writerThread.start();
		readerThread.start();

		//User Input Thread
		final BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
		new Thread(new Runnable() {
			public void run() {
				while(true) {
					try {
						String str = in.readLine();
						if("quit".equalsIgnoreCase(str)) {
								readerThread.interrupt();
						}
					} catch (Exception e) {
						e.printStackTrace();
					}
				}
			}

		}).start();
	}
}
class Reader implements Runnable {

	private Object lock;

	Reader(Object lock) {
		this.lock = lock;
	}

	public void run() {
		System.out.println("Reader::run");
		while(true) {
			synchronized(lock) {
				if("".equals(ConsumerProducerTest.data)) {
					System.out.println("Reader is going to wait");
					try {
						lock.wait();
					} catch (InterruptedException e) {
						System.out.println("Reader thread interrupted !!");
					}
					System.out.println("After call to Reader wait");
				} else {
					System.out.println("Reader got the data" + ConsumerProducerTest.data);
					System.out.println("Reader is notifying");
					ConsumerProducerTest.data = ""; //Clear the data
					//Wait for 10 seconds before notifying other threads.
					try {
						Thread.sleep(10000);
					} catch(InterruptedException ie) {
						System.out.println("Reader is interrupted while sleeping!!");
						System.exit(0);
					}
					lock.notifyAll();
					System.out.println("Reader done notifying");
				}
			}
		}
	}
}

class Writer implements Runnable {

	private Object lock;

	Writer(Object lock) {
		this.lock = lock;
	}

	public void run() {
		System.out.println("Writer::run");
		while(true) {
			synchronized(lock) {
				if("".equals(ConsumerProducerTest.data)) {
					ConsumerProducerTest.data = "data-"+System.currentTimeMillis();
						System.out.println("Writer wrote the data - " + ConsumerProducerTest.data);
						System.out.println("Writer is notifying reader");
						lock.notifyAll();
						System.out.println("Writer done notifying");
				} else {
					try {
						System.out.println("Writer is going to wait");
						lock.wait();
						System.out.println("After call to Writer wait");
					} catch(InterruptedException ie) {
						System.out.println("Writer is interrupted !!");
					}
				}
			}
		}
	}
}

Hope you find this useful.

Cheers !

Amit


Serialization – Compatible and Incompatible modifications to classes

April 15, 2010

Hello,
Java Serialization allows us to write an Object into byte stream and again re-construct its state whenever needed.
In between Serialization and De-serialization the class definition might get changed. Some modifications in
the class definition would result in java.io.InvalidClassException while De-serializing.
It is very important to understand these constraints. especially while dealing with Enterprise distributed Applications. As Distributed applications would always exchange the objects over the network stream using Serialization techniques.

Modifications those keep the Class compatible with Serialized version

  1. Add Fields
  2. Change Field from Static to Non-Static.
  3. Change Field from Transient to Non-Transient
  4. Add Class to the Object Tree


Modifications those make the Class incompatible with Serialized version

  1. Delete Field
  2. Change Field from Non-Static to Static
  3. Change Field from Non-Transient to Transient
  4. Changing the Class hierarchy
  5. Changing type of a Primitive Field

There are others changes as well those can be added to above list. They can be found here.
If you look at the above list the gist is “Addition of Field is permitted while Deletion cause Exception“.
Changing Field from Static to Non-static for transient to non-transient is kind of adding field to existing class.
While making non-static field to static and data-type of the field is like deleting the field which is incompatible.

This seems quite obvious from the Java Documentation but a beginner level would find this helpful, which is the reason
behind this post.

Cheers !

Amit