jump to navigation

Apache Cassandra Quick tour March 22, 2010

Posted by Terry.Cho in Distributed System.
Tags: , , , , ,
20 comments

Cassandra is distributed database system. It is donated to Apache open source group by Facebook at 2008.The Cassandra is based on Google Big Table data model and Facebook Dynamo distributed architecture. It doesn’t use SQL and optimized to high scale size of data & transaction handling. Even though Cassandra is implemented with Java language, other language can use the Cassandra as a client. (It supports Ruby,Perl,Python,Scala,PHP etc).

It is used to High Scale Size SNS like Face book,Digg,Twitter etc. It doesn’t support complex relationship like Foreign Key. It just provides Key & Value relationship like Java Hashmap. It is very easy to install and use.

Let’s look at data model of Cassandra

Data Model

Cassandra is based on google big table data model. It is called “Column DB”. It is totally different from traditional RDBMS.

Column

Column data structure which consists of column name and column value.

{name: emailAddress, value:cassandra@apache.org}
{name:age , value:20}

Column Family

Column family is set of columns. It is similar to row in RDBMS table. I will explain more detail about difference between Column Family and row in RDBMS later. Column Family has a key which identify each row in data set. Each row has a number of Columns.

For example, one row is

Cassandra = { emailAddress:casandra@apache.org , age:20}

“Cassandra” is key for the row, and the row has two columns. Keys of the columns are “emailAddress” and “age”. Each column value is “casandra@apache.org” and “20”.

Let’s look at Column Family which has a number of rows.

UserProfile={
Cassandra={ emailAddress:”casandra@apache.org” , age:”20”}
TerryCho= { emailAddress:”terry.cho@apache.org” , gender:”male”}
Cath= { emailAddress:”cath@apache.org” , age:”20”,gender:”female”,address:”Seoul”}
}

One of interest thing is each row can have different scheme. Cassandra row has “emailAddress” ,”age” column. TerryCho row has “emailAddress”,”gender” column. This characteristic is called as “Schemeless” (Data structure of each row in column family can be different)

KeySpace

Keyspace is logical set of column family for management perspective. It doesn’t impact data structure.

Super Column & Super Column Family

As I mentioned earlier, column value can have a column itself. (Similar to Java Hashtable can have ValueObject class as a ‘Object’ type)

{name:”username”
value: firstname{name:”firstname”,value=”Terry”}
value: lastname{name:”lastname”,value=”Cho”}
}

As a same way column family also can have column family like this

UserList={
Cath:{
username:{firstname:”Cath”,lastname:”Yoon”}
address:{city:”Seoul”,postcode:”1234”}
}
Terry:{
username:{firstname:”Terry”,lastname:”Cho”}
account:{bank:”hana”,accounted:”1234”}
}
}

UserList column family has two rows with key “Cath” and “Terry”. Each of the “Carry” and “Terry” row  has two column families – “Cath” row has “username” and “address’ column family, “Terry” row has “username” and “account” column family.

Cassandra Quick Test

Download Cassandra from http://incubator.apache.org/cassandra/ Extract zip file and run bin/cassandra.bat

We will connect Cassandra node with CLI interface. It is located in /bin/cassandra-cli.bat

The default TCP port number is 9160. You can change the port number in “conf/storage-conf.xml”

In “/conf/storage-conf.xml” file, default key space with name “Keyspace1” is defined. Column family type of the Keyspace is like this

Let’s put a new row with key name “Terry” which has Column (key=”gender”, value=”Male”)

Advertisements

Replication architecture in Cassandra and HBase March 19, 2010

Posted by Terry.Cho in Distributed System.
Tags: , , , ,
4 comments
Now i’m research distributed database architecture.
I found a very interesting article.
Apache Cassandra and Hadoop Hbase are most popular distributed database.
Twitter and Facebook are using Cassandra.
These solution is started from Google Big Table. So the data model is very similar.
The data model is called “Column database”. I will introduce the model later.
However my concern is how to replicate data across region (data centers in different region)
Here is very interesting information.
In case of Cassandra, it replicates data in every transaction. A coordinator captures changes and propagate it to other nodes.
But fiber based low latency network is required and there are no reference yet.
HBase data replication architecture looks very practical.
It captures change log and put it into replication queue. The replication message is passed to other nodes.
This mechanism is very similar to CDC (Change Data Capture).
Oracle Goden Gate, Quest Share Flex, MySQL geo replication are using this mechanism.

HBase replication looks more reasonable. It has common architecture and they have a reference.

===

After i had written this article, i got a feed back. Followed by the comment the article which i referenced is written by  fan of Hbase. Cassandra supports geo replication and has reference in face book. And Digg will deploy Cassandra in different data center.

But as i know even if facebook has two data center, they have fiber-link between the center. It is not a real geo replication. I will more research about cassandra data replication feature and re-post about this issue later.

SDP(Service Delivery Platform) Intro-from SOA perspective September 15, 2009

Posted by Terry.Cho in Architecture, SDP.
Tags: , , , ,
2 comments

Initiative

Common Telecom trend is Service Delivery Platform (SDP).

Telecom system has two challenges. Their business and technical platforms are changing very quickly. During few years they have been have a lot of platforms like Google Android, Microsoft Windows Mobile, Apple IPhone,Black berry,Symbian platforms. And service is extended from simple voice call & SMS to Multimedia service, application, Location based service etc. Following these business change they have to develop new application and integrate with legacy applications. Because telco has a lot of protocols and network layers (SIP,Paray-X,VoIP etc). Integration is getting more harder and complex. So they now need integrated and unified service platform to adopt business change quickly. This is the reason why SDP is required.

SDP is “Telco specific version of SOA”.

Concept of SDP is very simple. It exposes capability of system with unified interface. It reduces integration costs and let telco company to adopt new requirement quickly.

In addition by providing common services like billing,charging,logging, it removes duplication in implementation. This concept is exactly matched with SOA. So we can call SDP to “Telco specific version of SOA”. Only difference of SDP between traditional SOA is SDP requires network abstraction layer. In traditional SOA we had service enablement layer. It changes legacy protocol (SAP,MQ,Sieble) to unified SOA platform(Webservice etc.). It has been implemented with Adapters (JCA) or Other service enablement layers. In same way SDP has same abstraction layer. If traditional SOA communicated with legacy APPLICATION, SDP communicated with legacy NETWORK APPLICATION (SIP application, SMSC application, Paray-X based Application etc.). So in SDP there is network abstraction layer with network adapters.

Another difference between SDP and traditional SOA is SDP is B2C system (Service system). Traditional SOA is commonly designed for internal enterprise. SDP provides service to end user device (Mobile device, IP TV, Voip phone etc). So SDP covers more number of users than enterprise system and requires more faster response time. And message flow is more simpler than enterprise. As a result SDP has to have simple architecture to support fast response time, huge scalability.

Common architecture of SDP

Let’s think about common SDP architecture. SDP has 3 layers in the internal.

sdp

Ÿ   Service enablement layer

Network based application like SMS,VOIP has own protocol. Service enablement layer abstracts this network application and expose the capabilities with unified message protocol. By doing this user easily can use network based application without telco specific technology.

Service enablement layer also covers legacy integration. SDP can be integrated with legacy business system. For example Billing and Charging system in SDP can be integrated with Financial modules in ERP system. So Service enablement layer expose capability of legacy business system.

As a result the service enablement layer abstracts all of technical differences and let user can see the legacy components with same way.

Ÿ   Service component

Service component is software system that provides capabilities. Like SOA services. In SDP we have common services for telco system.

For example Contents Management, Billing, Charging & Rating, User Profile Management, Location Based Management. These components are implemented over specific vendor solution.

Commonly customer already has the service components. In that case abstract legacy service component and delegate the service in this service component layer.

Ÿ   Service exposure

Service components are exposed by this layer. This layer provides infrastructure of service.

For example it orchestrates existing services and make a new function. Mediate existing service. I will not describe this layer detail because this layer is same as traditional SOA concept (Orchestration with BPM, Mediation & single entry point with ESB etc)

Oracle solution mapping

Oracle has products that implements SDP.

SOA suite

For service exposure and service component development SOA suite is right for the requirements.

Oracle Service Bus is positioned for single entry point of SDP. And it enables service mediation. Complex service orchestration and long running process is implemented with Oracle BPEL/PM.

Oracle BPEL/PM also used for legacy system integration with various legacy adapters.

Oracle Coherence is used for storing & caching data. It is useful to manage & caching user profile data.

ODI is used for data provisioning between service components. And also can be used in contents provisioning.

Oracle WebLogic Server is used for component implementation with middleware features. It hosts component.

OCSG

It is used as network adapter in SDP. It supports specific telco protocol and expose the service to SOA platform. It also supports partner management and billing etc. Some capability can be duplicated with SOA platform or legacy business application. In that case based on business scenario, design should be considered.

OAAS

Like OCSG it enables network abstraction. But it just support SIP based application.

Conclusion

Common problem in SDP project is nobody can define SDP. SDP is de-facto standard. There is no standard reference architecture. So definition of SDP is different depends on industry. (Mobile, IPTV, VoIP etc.)

Think about SDP simple. SDP is telco specific SOA. It needs network adapter in addition to traditional SOA. And it is B2C system. It means it requires more concurrent user , faster response time and more simple architecture (message exchange format) than enterprise system. After understanding these characteristics of SDP, SDP is not hard to understand anymore.

ESB Design Pattern – Generic Proxy Pattern August 19, 2009

Posted by Terry.Cho in Architecture.
Tags: , , , , , , ,
3 comments

Generic Proxy Pattern

The Generic proxy pattern is a design pattern for ESB to reduce factory job and provide more flexibility to ESB design. Oracle ESB has 2 service types.

  • Business Service : This service provides a delegate for the real web service. It is used as a interface for real webservice.
  • Proxy Service : This service has mediation pipeline to provide flexibility for business service access. It covers transformation, routing, adding of new functionality and sometimes includes service orchestration.

One issue in OSB design proxy service is that normally we have cross cutting concerns like logging,authentication,authorization etc. Typically in OSB, we have to implement these in each proxy service. This introduces a lot of factory work and reduces system flexibility as changing cross cutting concerns becomes more difficult.

For example, we can implement logging with a Java call out in OSB. We can put the call out  into each proxy service. If we change interface of the java-call out class, we have to change all of proxy service in the system. The Generic proxy pattern helps us to remove this factory job.

Today with introduction of WEB 2.0, there are a lot of defecto standard protocol like XML/HTTP RPC, REST,JSON etc. Sometimes SOA systems have to support these kinds of protocols.  The generic proxy pattern provides a protocol converter.

Generic Proxy Pattern is consists of four layer.

genericproxy

  1. Edge Proxy (Inbound) :  This is entry point of our ESB. It covers protocol conversion to internal SOA protocol. (Web service)
  2. Common Proxy : This is point to handle cross cutting concerns for system – Authorization,Authentication,Logging,Billing etc. And route the service request to appropriate local proxy.
  3. Local Proxy : It is one-to-one mapping to Business service. It can just forward request to business service. OR it contains mediation logics (Intermediary logic – Transformation, Routing, Function adding etc)
  4. Edge Proxy (Outbound/Optional) : This proxy is another edge proxy for outbound protocol conversion. This is optional. In our architecture internal protocols are integrated with webservice. This proxy is used for legacy integration.
  5. Business Service :  There is some design options.

In type of service, we can provide internal service and external service. Internal service is for internal system and it doesn’t need any security checking. The security checking is covered by firewall etc. In that case we can separate Edge proxy&Common proxy to external and internal.

In the edge proxy we can separate internal and external service with URL prefix. Http://myplatform/internal/serviceURI Http://myplatform/external/serviceURI.

And each edge proxy uses internal common proxy and external common proxy. In internal common proxy we remove security related logic & billing related logic.

Logical proxy and business service are reused same way for external and internal exposing.

EAI project strategies-4. EAI basic architecture August 3, 2009

Posted by Terry.Cho in Architecture, EAI.
add a comment

Lots of EAI solution provides EAI components and frameworks but when we integrate system we have to make framework for the target systems. In this article I will explain about formal reference EAI architecture.

eai architecture

Interface

Interface Module is most basic module that integrates sender and receiver system.

Inbound

This module gets a request from sender system. It has two modules

Adapter

Adapter is entry point from Legacy system. It has native interface for the legacy system. It transfer the data into internal format like XML.

Message Transformation

Sometimes this module is combined into Adapter. This module translates native format of input message into EAI internal data type and format. Data format means data structure including header etc. Each enterprise has internal standard data structure with transaction header, sender id, timestamp etc. This module translated unformatted message to enterprise internal standard message.

Mediation

This module is very important in EAI.It processes incoming message to fit out bound message.

Routing

This module route incoming message to target system depends on contents or header. Routing can be occurred single source system to single target system (1:1 multiplicity) or N:1, 1:N multiplicity.

Mapping

This module mapping structure of data from source system to destination system. It changes message structure by using mapping between message fields.

MEP (Message Exchange Pattern)

Mediation also handling MEP. MEP means message communication methods like Sync, Async, Deferred. It uses message queue like JMS to covert MEP.

Outbound

Outbound is same as Inbound module. It translate EAI internal message into native message of destination system.

Monitoring and Trouble management.

EAI also has to have transaction monitoring and audit trouble & recovery feature.

Transaction Logging

Transaction Log is used to recovery/audit transaction when EAI or connected system has been failed. In transaction log, the transactions are identified by unique transaction id. The id is usually described in message header.

Concern of transaction log in EAI is where we store the log. Writing log always need physical disk IO. It directly impacts EAI interface performance.

We can choice log storage between file system or database.

If we use file log, the performance is faster than database. But problem is the log files are stored across a lot of hardware. So when problem is occurred and to find the transaction log, we have to search all of log files across all EAI machines.

In contrast if we use database for storing log. It provides very convenient log analysis. If transaction error occurred we can find it by simple querying from database. Compare to file  based logging, it provides single logging point. But if the database has been down, the problem impacts production EAI system. (EAI system has a dependency to database )

To solve these problem, I suggest async log writing system with JMS.

JMS based logging separates logging system and EAI system. Even if database behind EAI is failed, the impacts are minimized. (Log datas are stored in JMS queue until the database is recovered)

In JMS based logging system, we have to consider where the jms messages are stored. (memory/file or database). Memory based JMS store is fastest but can be lost message if JMS server is down suddenly. File based JMS store doesn’t lost message but performance is slower than Memory based JMS. (But I recommend to use file store based JMS )

Finally JMS based logging system can minimize performance degradation in EAI. But has a lot of factors to be considered in architecture design.

Error Handling Logic (Error Hospital)

This module covers error in EAI system. Error handling in EAI system has 3-steps.

Aware the error, logging cause of the error, and recover the error.

I don’t know who make the word named “Error Hospital”. Doesn’t it look great?

If an error has been occurred in interface, the error is handled gathered into error hospital. And the error is handled by error handling policy (Ignore, automatically recovery, notify to system engineer etc.)

Error Handling Policy

The error handling policy that I mentioned are like this.

Policy Description Note
Ignore Simply ignore the fault and purge message.
Report Report the fault. Optionally send notification message (Email,IM,SMS etc)
Retry Automatically retry the transaction with specified time after specified sleep time After failing all of retry, next error handling policy definition is required.
Manual

handling

Report the fault and let user select policy described above

Infrastructure

Transaction handling

Commonly EAI requirements has 10~30% XA based distributed transaction. It is easy way to guarantee data integrity between integrated system. Even if XA is easy to apply internal architecture of XA is complex. So it needs a sophisticated testing.

Dynamic deployment

This is one of most important features in EAI. If one interface has a problem, the interface should be deactivated to prevent problem propagation or impact to other interface.

EAI project strategies-3. EAI Implementation Process July 27, 2009

Posted by Terry.Cho in Architecture, EAI.
Tags: , , ,
add a comment
Second is delivery process of EAI project.
In persepctive of tranditional water-fall model, to do item is like below.
Analysis Phase
In this phase we define “What to do EAI system ?”
1) Define interface type
It is about, what business systems are integrated? What is techical interface type of the system (EJB,IIOP,MQ,Web Service etc)
In this phase, decide to what interface use XA.
2) Define MEP (Message Exchange Pattern)
Gather requirement of MEP. Sync,Async,Real time, Near real time and multiplicities (1:1,1:N etc)
3) Define Message structure
Define message format. What header fields are required? What format will be used? (MFL?Text?XML? etc)
After finishing Anlaysis phase, we understand the EAI system what will do.
Design Phase
1) Designe architecture
Based on requirement from analysis phase. Design architecture and system.
2) Implement prototype
With the design, priotize most critical functionalities and design and implement prototype.
3) Test with prototype
Valdating prototype and enhance architecture
4) Gather interface list
Gather interface list what business system and functional will be integrated.
EAI project has a dependency to schedule of other business systems. Based on interface list and integration schedule, EAI system arrange their interface implementation schedule.
Important thing in design phase is validate architecture. After validating the architecture, EAI team implement interface with validated architecture by factory job.
Removing risk in design phase is important because architecture related changes are requires a lot of resource and time. So removing risk in beginning phase enables reduce cost.
Implementation Phase
1) Implement interface
Implement interface based on schedule. EAI interface implementation impacts business team implementation. Because business system communicates by using EAI system.
2) Step by step transition
Transition (Deployment) plan in EAI is very important. As i mentioned lot of time EAI system has dependencies to other system and it also has dedicated network line (X.25 etc)/
So during implementation interfaces deployed in development system. After validating the interface, it is deployed to staging system. Other business system connected the staging system.
Finally all of implementation has been finished and ready to move production, EAI system is deployed into production environment.
I will describe more about deployment environment later.
3) Monitoring and fix
After starting deploying interface to staging system. (Open interface to business team). EAI project team monitors the interfaces. The interfaces always has errors like “Table comlumn mismatch”,”Message type missmatch” etc.
The error let the interface down and it brings complain from business development team.
In implementation phase, support business team to follow up their schedule by matching EAI interface implementation schedule. If delay is occurred, business team try to responsibilities to EAI team for the schedule delay.
And there are many requirement change like number of interfaces, type etc. So in the implementation phase, interface schedule management is very important.

Second is delivery process of EAI project.

In perspective of traditional water-fall model, to do item is like below.

eaiprocess

Analysis Phase

In this phase we define “What to do EAI system ?”

1) Define interface type

It is about, what business systems are integrated? What is techical interface type of the system (EJB,IIOP,MQ,Web Service etc).  In this phase, decide to what interface use XA.

2) Define MEP (Message Exchange Pattern)

Gather requirement of MEP. Sync,Async,Real time, Near real time and multiplicities (1:1,1:N etc)

3) Define Message structure

Define message format. What header fields are required? What format will be used? (MFL?Text?XML? etc)

After finishing Analysis phase, we understand the EAI system what will do.

Design Phase

1) Design architecture

Based on requirement from analysis phase. Design architecture and system.

2) Implement prototype

With the design, prioritize most critical functionalities and design and implement prototype.

3) Test with prototype

Valdating prototype and enhance architecture

4) Gather interface list

Gather interface list what business system and functional will be integrated.

EAI project has a dependency to schedule of other business systems. Based on interface list and integration schedule, EAI system arrange their interface implementation schedule.

Important thing in design phase is validate architecture. After validating the architecture, EAI team implement interface with validated architecture by factory job.

Removing risk in design phase is important because architecture related changes are requires a lot of resource and time. So removing risk in beginning phase enables reduce cost.

Implementation Phase

1) Implement interface

Implement interface based on schedule. EAI interface implementation impacts business team implementation. Because business system communicates by using EAI system.

2) Step by step transition

Transition (Deployment) plan in EAI is very important. As i mentioned lot of time EAI system has dependencies to other system and it also has dedicated network line (X.25 etc)/

So during implementation interfaces deployed in development system. After validating the interface, it is deployed to staging system. Other business system connected the staging system.

Finally all of implementation has been finished and ready to move production, EAI system is deployed into production environment.

I will describe more about deployment environment later.

3) Monitoring and fix

After starting deploying interface to staging system. (Open interface to business team). EAI project team monitors the interfaces. The interfaces always has errors like “Table comlumn mismatch”,”Message type missmatch” etc.

The error let the interface down and it brings complain from business development team.

In implementation phase, support business team to follow up their schedule by matching EAI interface implementation schedule. If delay is occurred, business team try to responsibilities to EAI team for the schedule delay.

And there are many requirement change like number of interfaces, type etc. So in the implementation phase, interface schedule management is very important.

Conclusion.

As i mentioned EAI project has a lot of dependencies to other system or project and their are many requirement changes about interfaces. In EAI project how to manage schedule and environment is very important to success EAI project

EAI project strategies-2. Requirements July 27, 2009

Posted by Terry.Cho in Architecture, EAI.
Tags: ,
6 comments
First pesepctive, EAI system requirements.
Waht is EAI system requirement. Commonly EAI system requirements are defined based on business like below.
Internal integration
This is integrating internal enterprise system. A lot of these types of integrations are “on-line”(real time).
Sometimes it uses XA based distributed transaction to support data integrities.
External integration
This type is B2B integration. System is located on external of the enterprise.
One of interest thing of this type is using dedicated network like dedicated TCP/IP or X.25 network
It impacts EAI architecture style. For example we have 2 X.25 line for some company. But we have 10 EAI system instances. What shall we do?
We have to consume request by all of the EAI instances and route the request to EAI instances that has been connected to X.25 line.
And we have to think about fail over when the X.25 connected EAI instance down.
In B2B style integration, it is very hard to use XA based distributed transaction. So we need another way to support data integraities like by using transaction trace logs etc.
Batch
In integration world there are two requirement about Batch. (Bulk data integration.)
One is for online business and the other one is for data anlaysis.
First thing is handling transaction data in online system. Apply data changes to the other system.
It happened about a few seconds later after original change has been made. We call it as “Defered or Near-Realtime”.
In this case, because it is online transaction data, integrity of the data is very important. So someimes XA is used.
Sencond case is for data analysis. Aggreate data into data warehouse and generate report. EAI has a role to gather data to warehouse.
This job is usally happened after office hour and amount of data is very huge. Commonly EAI doesn’t not handle this type of data trasfer.
This is ETL (Extract Transform Loading) area.
Simply i summarized EAI system business boundary based on business type.
But when we design EAI architecture, we have to think about MEP (message exchange patterns)-Sync,Async,Fire & Forget,Master Detail and Multiplicities (1:N,1:1 etc)
In addition OAM (Operation, Administration and Monitoring) requirements are defined in this phase.

First pesepctive, EAI system requirements.

Waht is EAI system requirement. Commonly EAI system requirements are defined based on business like below.

eaitype

Internal integration

This is integrating internal enterprise system. A lot of these types of integrations are “on-line”(real time). Sometimes it uses XA based distributed transaction to support data integrities.

External integration

This type is B2B integration. System is located on external of the enterprise. One of interest thing of this type is using dedicated network like dedicated TCP/IP or X.25 network.  It impacts EAI architecture style. For example we have 2 X.25 line for some company. But we have 10 EAI system instances. What shall we do? We have to consume request by all of the EAI instances and route the request to EAI instances that has been connected to X.25 line. And we have to think about fail over when the X.25 connected EAI instance down.

In B2B style integration, it is very hard to use XA based distributed transaction. So we need another way to support data integraities like by using transaction trace logs etc.

Batch

In integration world there are two requirement about Batch. (Bulk data integration.)

One is for online business and the other one is for data anlaysis. First thing is handling transaction data in online system. Apply data changes to the other system. It happened about a few seconds later after original change has been made. We call it as “Defered or Near-Realtime”.In this case, because it is online transaction data, integrity of the data is very important. So someimes XA is used.

Sencond case is for data analysis. Aggreate data into data warehouse and generate report. EAI has a role to gather data to warehouse. This job is usally happened after office hour and amount of data is very huge. Commonly EAI doesn’t not handle this type of data trasfer. This is ETL (Extract Transform Loading) area.

Simply i summarized EAI system business boundary based on business type. But when we design EAI architecture, we have to think about MEP (message exchange patterns)-Sync,Async,Fire & Forget,Master Detail and Multiplicities (1:N,1:1 etc) In addition OAM (Operation, Administration and Monitoring) requirements are defined in this phase.

EAI project strategies-1. Approach July 24, 2009

Posted by Terry.Cho in Architecture, EAI.
add a comment

Even if EAI (Enterprise Application Integration) has been introduced a few years ago. Enterprise Application Integration is still important in Enterprise Architecture. From this post i will explain a strategies to success EAI project.

How to approach EAI project ?

There is 4 perspectives to sucess EAI project.
eai1

Business Requirement

First, requirement for EAI.  Define target integrated business system (ERP,CRM,SCM etc). What kinds of interface will be used? (Socket? WebService? EJB? IIOP? MQ etc). How can we trace transaction and fault. What is target performance. What is message exchange patterns (Sync/Async/Batch etc).

In common EAI system can have a lot of capabilities, but no one can implement all of things. We have to extract EAI requirement based on customer’s requirement & environment.

Architecture

After defining requirement, we can design conceptual architecture.

Process

One of differences EAI is, EAI integrated other systems. It means it has a lot of  “Communication” in the project. We have to sync EAI interface between sender side of business unit and receiver side of business unit. In Enterprise , number of interface usually over  300~400 number. So managing communication and collaboration for implementing interface between different business units.

Development & Production Enviroment (Deploy)

As i mentioned in “Process”, EAI project has a dependency to other business system implementation project. In their implementation phase, EAI system has to provide integration between development environment of other business system. In addition EAI project team also has to have their own development environment.

In case of EAI system, it can have dedicated network line for B2B for example X.25 etc. So we have to consider deployment plan with the network line migration plan.

From next post i will explain about these 4 perspects with more detail.

New J2EE Architecture with Oracle Coherence. July 24, 2009

Posted by Terry.Cho in Architecture.
Tags: , , ,
add a comment

Intro

In case of Korea success rate of JEE based SI projects are getting lower. A one of the reason we can think about is framework. Most JEE projects are using framework like spring,hibernate,JAX-WS,XML framework etc. It helps us to leverage efficiency of implementation and provide more higher quality of architecture. In contrast it makes our system to more complicate. It causes a trouble in production system. Makes system hang, bring high cpu usage, memory usage etc
In this document, I will introduce WAS and coherence based architecture to raise up quality of JEE architecture.

Pain point of traditional JEE architecture
Traditional JEE architecture is like below.

1All of business logics are packaged into one application (1 WAR or 1 EAR file) including user interface. BIZ logic is implemented with EJB or POJO (Spring etc. POJO is current trend). UI is implemented Servlet/JSP of MVC framework.

This traditional JEE architecture has weak points.

Requires lots of redundant memory.
Current requirement for IT is being increased and complex is increased. So size of application and memory usage of application is being increased. The problem is JVM has a limitation in memory size. In 32bit JVM, Java can use usually maximum 1~1.5 GB Heap memory. Even if we can use 64bit JVM. We still have the problem in Garbage collection (GC) time. In full GC time, JVM suddenly stopped. If we increased the java heap size, the full GC time also increased. From JVM 1.4 there are many GC tuning options like Concurrent GC (CMS), Parallel GC. We are not free from the GC time.
In the traditional architecture all of biz logics are packaged into one application. Memory usage is getting increased. Especially it makes a burden in JVM permanent memory area. (The area that classes are loaded.)

Trouble affects other business Logic
Another problem of traditional JEE architecture is trouble propagation. Because all of business logics are running over same WAS instance, if some business logic makes a problem like high CPU usage, high memory usage affects the other business logic. It means if only one business logic has a problem, whole system can be stopped. Even if our weblogic server provides workmanager and clustering feature which prevent problem propagation, the problem from business logic can not be prevented

How to solve this pain point?
So how can we solve this problem? Answer is easy. Separate the business logics to separate WAS instance.

 2
It helps us by
Effective memory usage
Because only one business logic is deployed to one instance. Memory utilization is became low.

Prevent affection of problem between different business logic
Business logic is separated, so if one business logic has a problem, only the business WAS instance (or cluster) has a trouble. It doesn’t provide any effect to other business.

Efficient resource allocation
In addition separating business by WAS instance can make us to use hardware resource efficiently .
In Hard ware platform which supports CPU partitioning, system administrator can assign CPU to specific business logic. In traditional JEE architecture, it is hard to assign the resource to business. Because all business logics are running over same WAS instance, so cannot select process which CPU is assigned to.

So, even if the separation of business logic by WAS instance has a lot of architectural, why the traditional JEE architecture doesn’t implement it?
Problem comes from shared data & state information management. 3
In JEE architecture needs to share business data across the business logics or keep user state data by using Http Session. But if we separate business logic by WAS instance, it cannot share the session data and shared information across the WAS instances.
 It is a reason why our suggested architecture cannot be realized.

Separate business logic across WAS instance with Oracle coherence.
Finally with Oracle Coherence solution we can realize the architecture.
Coherence is Data Grid Solution which is not just Caching solution but is extendible, high performance, high availability large shared memory with multiple interface.
 Basically Coherence support HTTP session replication. It gives us more Http session capacity and solves the problem to sharing user state data. 4
By using coherence we can solve all of the pain point in traditional JEE architecture.
We get additional advantage from this architecture. We can plan a capacity based on business logic. (Assign WAS instance to specific biz logic based on capacity.)
 And it also give us freedom of redeployment. We don’t have to redeploy whole of WAS instance and don’t have to restart whole instances.

Conclusion
It is hard to win a deal with only WebLogic server in JEE application market. We need a approach to suggest integrated package of our product and especially with ARCHITECTURE which can solve pain point of our customers.
WebLogic and Coherence are very excellent product from before acquired by Oracle. Acquisition should provide advantage with chemical interactions. This is the case that can make the chemical interactions and will provide a higher position in biding with other vendors.