Dec 17, 2010

Let's ride the Camel - The simplest program to understand Camel

Camel Is and integration framework. It can pass data from one system to another. Here I will create a simple program to copy a file another place. Maybe this is simple but I think this easiest way to understand camel. I will use camel core and Java to do this. Then I will use Maven to do the same thing.

Think you need to copy a file call "File_A" in "Inbox" directory and you need to copy that file into "Outbox" directory. We can write a simple java I/O program to do that. It will be like this;

import java.io.*;

public class FileCopier {

   public static void main(String args[]) throws Exception {

File inboxDir= new File("Inbox");
File outboxDir= new File("Outbox");

outboxDir.mkdir();
File[] files=inboxDir.listFiles();
for (File source : files)
   if (source.isFile()){
File dest=new File(outboxDir.getPath() + File.separator + source.getName());
copyFile(source, dest);
}
   }

   public static void copyFile(File source, File dest) throws IOException {

  OutputStream out = new FileOutputStream(dest);
byte[] buffer=new byte[(int) source.length()];
FileInputStream in= new FileInputStream(source);
in.read(buffer);
try {
out.write(buffer);
}
finally {
out.close();
in.close();
}
   }
}

Note : You may need to create the Inbox directory in the same directory where FileCopier.java ecesis. Otherwise you must give the correct path. Next thing is, before you run this program make sure that the file Inbox/File_A ecesis.


Now we can try the same thing with camel.
To use camel we need to download camel library files (http://camel.apache.org/), and place them in the classpath. Then try this;

import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;

public class FileCopierWithCamel {

   public static void main(String args[]) throws Exception{

CamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder(){
  public void configure(){ from("file:Inbox?noop=true").to("file:Outbox");}
});
context.start();
Thread.sleep(10000);
context.stop();
   }

}


So, I think now you can see the difference. Thread.sleep() is to give some time to copy the file. In this program, it required to download camel. But If we use maven, maven will manage that part. No need to copying to class paths and so on.

To do the same thing with the help of maven you can try;
mvn archetype:generate

Then chose the default one - "maven-archetype-quickstart" and give relevant details. This will create a files and a directory. It will look like pom.xml, src. Source code will be placed in src directory and pom.xml will contain all dependencies and plugin details. We use camel as a dependency, so we need to edit the pom.xml. Add this code segment in between the <dependencies> </dependencies> tags.

   <dependency>
     <groupId>org.apache.camel</groupId>
     <artifactId>camel-core</artifactId>
     <version>2.5.0</version>
   </dependency>

For this program, it will require only the camel-core, but for some programs there are many dependencies. You can find these details from the camel website.

Now we need to replace the code part. You can simply replace the maven generated "App.java" from previously created FileCopierWithCamel.java file. You should insert the package details in the FileCopierWithCamel.java file and that can be grabbed from the "App.java" file.
As previous, we need to create "Inbox/File_A". It should create where the pom.xml ecesis. Now the directory structure will like pom.xml, src, Inbox.

Then you can run it by;
mvn compile java:exec -Dexec.mainClass=<package name>.FileCopierWithCamel

Now you can see, that the program had created the "Outbox" Directory and inside that, you can see the "File_A".

Dec 16, 2010

Database connection pooling

Databases are the most common storing technology in todays IT world. Most of IT related things such as softwares and websites store data in a database. Database integration in a system got many advantages but there are some disadvantages also.
The main disadvantage is it require some time to retrieve or to store data. The database is an external thing from the software or the website. So first software or the website must make a connection to the database. This connection establishment require 1 to 3 seconds of time. Then it must send the query to retrieve data or to store data in the database. And then it should close the connection properly. This time requirement is an over head to the system. If the system is used by a large number of people this will make a huge impact.
So to avoid this, we can use a pool of connections. The pool will keep some database connections alive without disconnecting it from the database. Then, if the system request a database connection it will give one of its live connection to the request. While it in use there may be another request also. Then again pool will release another live connection to that. Now the systems does not need to make connections to the database, its almost done by the database pool. So that time requirement goes down.
But there are some limitations with the database pools also. Pool contains a limited number of connections but that may reduce performance issues. The programmer or the system designer must do additional work to use a database pool and he/she should think how to manage it. Mainly how to handle the maximum number of connections, connection queues, priority based connections and so on. There are many library files which can be used like c3p0.
So now we can have a good integrated system with good performances.  

Dec 6, 2010

Implementing a solution by using Camel.

Here we try to implement an integration solution by using Camel. The problem is, to send an e-mail to a central administration. In an organization each client can send an incident to the central administration. Organization use a website to do this process. So the client can visit the website and fill an incident form and send.  Website should pass that details to the relevant place by converting that details in to an E-mail.

To build this system we need Maven, Apache-cxf, Apache-Camel and Spring web framework. So lets try this.
first you need to create a Maven project. To do that you can type,
mvn archetype:create -DgroupId=org.apache.camel -DartifactId=camel-example-reportincident -DarchetypeArtifactId=maven-archetype-webapp

Then you need to add Apach-cxf dependencies to the pom.xml file.

You may need "cxf-rt-core" , "cxf-rt-frontend-jaxws", "cxf-rt-transports-http". After that we need to create a .wsdl file. We should put it in src/main/webapp/WEB-INF/wsdl/report_incident.wsdl .
 (refer this link to get the coding :- http://camel.apache.org/tutorial-example-reportincident-part1.html)

Now we need to add some plugins to generate java classes from the wsdl. To do that we need to integrate the cxf wsdl2java converter plugins and maven-compiler-plugin. After inserting these plugins you should edit the web.xml file.

In that file we must add a listener. Now we will use spring to do that.
<listener>
    <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>

Then we should specify the cxf servlet, URI and cxf-config,xml path details in the web.xml file.

So the path for cxf-config.xml is src/main/resources and you should place that xml file in that place.

The basic configurations are done for now. Now you need to create a java file and it should like this.

package org.apache.camel.example.reportincident;

/**
 * The webservice we have implemented.
 */
public class ReportIncidentEndpointImpl implements ReportIncidentEndpoint {

    public OutputReportIncident reportIncident(InputReportIncident parameters) {
        System.out.println("Hello ReportIncidentEndpointImpl is called from " + parameters.getGivenName());

        OutputReportIncident out = new OutputReportIncident();
        out.setCode("OK");
        return out;
    }

}

Before that you may need to remove the .jsp page which created by maven initially. Then you can try to compile the project.
Try;
mvn clean compile

If the project compiles without errors you can try to run it on a web container. So you need to edit the pom file again and insert maven-jetty-plugin to the main pom file. using jetty is easy than using any other web container, but camel supports for many finds of web containers too.
Then try;
mvn jetty:run

If you try to visit http://localhost:8080 you will see some details to your web application. Still it is not complete, si you wil not see any UI. You can use SOAPUI to do that thing.

Cont...

Dec 2, 2010

Important things when integrating an enterprice systems.

There are some important and critical things developers must think before starting the integration. In an enterprise, it will use different kinds of software on different platforms. This is the biggest issue which all the developers will face.

For an enterprise a developer or a team of developers cannot create a whole system, because it is a time consuming task. So what the developers do is, they try to integrate the existing software products. But there are some software products which offers its source code, but not almost all the software products do that thing. This will make the process more difficult.

An integrated system, applications must communicate with each other. But if an application totally depended on another application it will be a mess. Think if "A" application request some thing from "B" but "B" fails to respond to that request. If this kind of a thing happens, what "A" application must do? So the developer must integrate the system with low coupling to avoid such a situation. As in this case, "A" an use a time-out and send a error message to continue its work.

As I mentioned, there are some software products which offer its source code. But if a developer try to change the coding as he/she wish, it will create a performance issue on that software. So the developer must try to integrate the system with a minimum amount of changes.

Other thing is the technology which use to integrate the system. There are various kinds of techniques and most of them need some special knowledge hardware and some other things. That thing also a problem we should think.

As an integrated system applications must understand what the other application send to it. So the communication media (maybe a message or a signal) should be in a common data format. Developer must convert the message into a common format before it leave that sub system.

Even if these systems can communicate with each other, still there are some issues. If a sub system is sharing its data among other sub systems, every sub system must use it within a short time frame. Because, if a single sub system is using a shared resource it will lock that resource and other sub systems will wait until that resource is available. so the developer must think about the time limitations.

If the enterprise system got a common functionality, it will create another issue. Sharing a functionality between sub systems is not a simple task. And also if the system need to communicate with a remote system like a banking system or a shipping system that thing also add more weight on developers. These areas will me more complex.

The most important thing is the reliability. The information which send from a sub system should be reliable. Developers most make those communication in a secure manner. This will be a huge issue when it comes to remote communication.

These are the most important thing a development team  must think before developing an enterprise system. If a single information goes wrong, it will be a huge damage to the company. So the developer must use the suitable solutions as it requires in an integration. 

Dec 1, 2010

Do we need "Integration Patterns" ?

Have you ever think how this Blog works? No, isn't it. In the top you may see something like "Thilina's Blog". Then the whole area is divided in to two separate columns. One containes this article and the other may contain some additional links. If you try to view the same article after few days, you may see some new details on the right side column (maybe new links). So how it happen?

Most modern web pages display a main content and with some related links and informations. Even if you request a single thing, it must offer that thing as well as other related stuff. Likewise  there are some systems which need other systems informations in order to display or to do its task. When a system depends on another system, these things need to communicate each other.

Think you need to change your address details in your bank account. What you will do is, just simply login to the bank website and change the address. But if you try to view your address through a ATM machine or from a bank statement, that thing also updated. But have you update those thing? The thing is, when you update the details through the website it automatically update all other things also. So how to do this kind of a thing? One system should send data to the aother system and the that system should be able to read data from it.

In a bank there are several different sections. Likewise in a bank network also there are different services provided by different systems. As an example, ATM will provide a different service than a bank website. But both things work in a single bank system. In here we can see some systems must responds according to the service it can offer. How to implement this kind of a service oriented thing. System must respond to according to the request and chose the correct system which is capabler of doing the requested task.

Think you had login to your  banking website. Now you can view your bank details. If you go to an ATM machine and try to login with your PIN number, there also you will see your details. In both case you will see only your details but not others. Both systems must identify the user and view the correct content. The identification of the user is a common process for these two system. If this is more common thing, coding it again and again in different sub systems is not needed. But there is a need to communicate with that process in order to  complete a process in a different sub system.

If you need to buy something from a website, what you will do? We need only to insert our bank details. But, the website must verify wither we got enough credit to buy that thing or not. So  to do that it need to communicate with our bank. Now a system in a business need to communicate with another system which is also in another business. So how to communicate between two enterprise systems?

These are the main areas that an enterprise system developers must think. Most of the time enterprise system developers use per-build sub systems, because developing a hole system is very time consuming thing. That's why we need "Integration Patterns". Here I have describe main types of integration;

* Information Portal
* Data Replication
* Shared Business Function
* Service Oriented Architecture
* Distributed Business Process
* Business to Business integration.

When a developer need to develop an enterprise system he/she can use these methods in order to do it in less effort and without wasting time.

Nov 26, 2010

Apache Camel

" Apache Camel is a powerful open source integration framework based on known Enterprise Integration Patterns with powerful Bean Integration ".

So what is Enterprise Integration Patterns? In an enterprise they may use different software to do there task easily. But those software should interact with others and will not be isolated. Therefore, software must communicate with each other. But as an enterprise software, its communication is not so easy. To overcome this problem people use a common set of instructions or patterns when they implementing an enterprise solution. Most people use "Enterprise Integration Patterns" book and it describe about 65 categorise of integration patterns which was collected since 2002. Most of these patterns use asynchronous messaging architectures with the help of EAI, SOA, or ESB platforms such as  IBM WebSphere MQ, TIBCO, Vitria, SeeBeyond, WebMethods, BizTalk, Sonic, Fiorano, Mule ESB, ActiveMQ.

Beans Integration is, it provides a general purpose mechanism for processing messages using arbitrary Java objects. By using a bean reference into a route, you can call an arbitrary method on a Java object. From that you can access and modify the incoming exchange. So this bean binding mechanism is used in Apache Camel.

Then why we need camel? When you are designing an enterprise solution you need to use integration patterns and to implement that thing you may need to  use messaging modules like HTTP, ActiveMQ, JMS, JBI, SCA, MINA, etc. Camel create an easy environment to use these things, such as a pluggable data format option which can interact with other modules. It is a small library with less dependencies for easy embedding in any Java application. Camel can be used as a routing mechanism for Apache ServiceMix which is a powerful ESB and JBI container. So Camel can be used in complex enterprise integration solutions to do the job with less effort.