jump to navigation

EAI project strategies-4. EAI basic architecture August 3, 2009

Posted by Terry.Cho in Architecture, EAI.
add a comment

Lots of EAI solution provides EAI components and frameworks but when we integrate system we have to make framework for the target systems. In this article I will explain about formal reference EAI architecture.

eai architecture

Interface

Interface Module is most basic module that integrates sender and receiver system.

Inbound

This module gets a request from sender system. It has two modules

Adapter

Adapter is entry point from Legacy system. It has native interface for the legacy system. It transfer the data into internal format like XML.

Message Transformation

Sometimes this module is combined into Adapter. This module translates native format of input message into EAI internal data type and format. Data format means data structure including header etc. Each enterprise has internal standard data structure with transaction header, sender id, timestamp etc. This module translated unformatted message to enterprise internal standard message.

Mediation

This module is very important in EAI.It processes incoming message to fit out bound message.

Routing

This module route incoming message to target system depends on contents or header. Routing can be occurred single source system to single target system (1:1 multiplicity) or N:1, 1:N multiplicity.

Mapping

This module mapping structure of data from source system to destination system. It changes message structure by using mapping between message fields.

MEP (Message Exchange Pattern)

Mediation also handling MEP. MEP means message communication methods like Sync, Async, Deferred. It uses message queue like JMS to covert MEP.

Outbound

Outbound is same as Inbound module. It translate EAI internal message into native message of destination system.

Monitoring and Trouble management.

EAI also has to have transaction monitoring and audit trouble & recovery feature.

Transaction Logging

Transaction Log is used to recovery/audit transaction when EAI or connected system has been failed. In transaction log, the transactions are identified by unique transaction id. The id is usually described in message header.

Concern of transaction log in EAI is where we store the log. Writing log always need physical disk IO. It directly impacts EAI interface performance.

We can choice log storage between file system or database.

If we use file log, the performance is faster than database. But problem is the log files are stored across a lot of hardware. So when problem is occurred and to find the transaction log, we have to search all of log files across all EAI machines.

In contrast if we use database for storing log. It provides very convenient log analysis. If transaction error occurred we can find it by simple querying from database. Compare to file  based logging, it provides single logging point. But if the database has been down, the problem impacts production EAI system. (EAI system has a dependency to database )

To solve these problem, I suggest async log writing system with JMS.

JMS based logging separates logging system and EAI system. Even if database behind EAI is failed, the impacts are minimized. (Log datas are stored in JMS queue until the database is recovered)

In JMS based logging system, we have to consider where the jms messages are stored. (memory/file or database). Memory based JMS store is fastest but can be lost message if JMS server is down suddenly. File based JMS store doesn’t lost message but performance is slower than Memory based JMS. (But I recommend to use file store based JMS )

Finally JMS based logging system can minimize performance degradation in EAI. But has a lot of factors to be considered in architecture design.

Error Handling Logic (Error Hospital)

This module covers error in EAI system. Error handling in EAI system has 3-steps.

Aware the error, logging cause of the error, and recover the error.

I don’t know who make the word named “Error Hospital”. Doesn’t it look great?

If an error has been occurred in interface, the error is handled gathered into error hospital. And the error is handled by error handling policy (Ignore, automatically recovery, notify to system engineer etc.)

Error Handling Policy

The error handling policy that I mentioned are like this.

Policy Description Note
Ignore Simply ignore the fault and purge message.
Report Report the fault. Optionally send notification message (Email,IM,SMS etc)
Retry Automatically retry the transaction with specified time after specified sleep time After failing all of retry, next error handling policy definition is required.
Manual

handling

Report the fault and let user select policy described above

Infrastructure

Transaction handling

Commonly EAI requirements has 10~30% XA based distributed transaction. It is easy way to guarantee data integrity between integrated system. Even if XA is easy to apply internal architecture of XA is complex. So it needs a sophisticated testing.

Dynamic deployment

This is one of most important features in EAI. If one interface has a problem, the interface should be deactivated to prevent problem propagation or impact to other interface.

Advertisements

EAI project strategies-3. EAI Implementation Process July 27, 2009

Posted by Terry.Cho in Architecture, EAI.
Tags: , , ,
add a comment
Second is delivery process of EAI project.
In persepctive of tranditional water-fall model, to do item is like below.
Analysis Phase
In this phase we define “What to do EAI system ?”
1) Define interface type
It is about, what business systems are integrated? What is techical interface type of the system (EJB,IIOP,MQ,Web Service etc)
In this phase, decide to what interface use XA.
2) Define MEP (Message Exchange Pattern)
Gather requirement of MEP. Sync,Async,Real time, Near real time and multiplicities (1:1,1:N etc)
3) Define Message structure
Define message format. What header fields are required? What format will be used? (MFL?Text?XML? etc)
After finishing Anlaysis phase, we understand the EAI system what will do.
Design Phase
1) Designe architecture
Based on requirement from analysis phase. Design architecture and system.
2) Implement prototype
With the design, priotize most critical functionalities and design and implement prototype.
3) Test with prototype
Valdating prototype and enhance architecture
4) Gather interface list
Gather interface list what business system and functional will be integrated.
EAI project has a dependency to schedule of other business systems. Based on interface list and integration schedule, EAI system arrange their interface implementation schedule.
Important thing in design phase is validate architecture. After validating the architecture, EAI team implement interface with validated architecture by factory job.
Removing risk in design phase is important because architecture related changes are requires a lot of resource and time. So removing risk in beginning phase enables reduce cost.
Implementation Phase
1) Implement interface
Implement interface based on schedule. EAI interface implementation impacts business team implementation. Because business system communicates by using EAI system.
2) Step by step transition
Transition (Deployment) plan in EAI is very important. As i mentioned lot of time EAI system has dependencies to other system and it also has dedicated network line (X.25 etc)/
So during implementation interfaces deployed in development system. After validating the interface, it is deployed to staging system. Other business system connected the staging system.
Finally all of implementation has been finished and ready to move production, EAI system is deployed into production environment.
I will describe more about deployment environment later.
3) Monitoring and fix
After starting deploying interface to staging system. (Open interface to business team). EAI project team monitors the interfaces. The interfaces always has errors like “Table comlumn mismatch”,”Message type missmatch” etc.
The error let the interface down and it brings complain from business development team.
In implementation phase, support business team to follow up their schedule by matching EAI interface implementation schedule. If delay is occurred, business team try to responsibilities to EAI team for the schedule delay.
And there are many requirement change like number of interfaces, type etc. So in the implementation phase, interface schedule management is very important.

Second is delivery process of EAI project.

In perspective of traditional water-fall model, to do item is like below.

eaiprocess

Analysis Phase

In this phase we define “What to do EAI system ?”

1) Define interface type

It is about, what business systems are integrated? What is techical interface type of the system (EJB,IIOP,MQ,Web Service etc).  In this phase, decide to what interface use XA.

2) Define MEP (Message Exchange Pattern)

Gather requirement of MEP. Sync,Async,Real time, Near real time and multiplicities (1:1,1:N etc)

3) Define Message structure

Define message format. What header fields are required? What format will be used? (MFL?Text?XML? etc)

After finishing Analysis phase, we understand the EAI system what will do.

Design Phase

1) Design architecture

Based on requirement from analysis phase. Design architecture and system.

2) Implement prototype

With the design, prioritize most critical functionalities and design and implement prototype.

3) Test with prototype

Valdating prototype and enhance architecture

4) Gather interface list

Gather interface list what business system and functional will be integrated.

EAI project has a dependency to schedule of other business systems. Based on interface list and integration schedule, EAI system arrange their interface implementation schedule.

Important thing in design phase is validate architecture. After validating the architecture, EAI team implement interface with validated architecture by factory job.

Removing risk in design phase is important because architecture related changes are requires a lot of resource and time. So removing risk in beginning phase enables reduce cost.

Implementation Phase

1) Implement interface

Implement interface based on schedule. EAI interface implementation impacts business team implementation. Because business system communicates by using EAI system.

2) Step by step transition

Transition (Deployment) plan in EAI is very important. As i mentioned lot of time EAI system has dependencies to other system and it also has dedicated network line (X.25 etc)/

So during implementation interfaces deployed in development system. After validating the interface, it is deployed to staging system. Other business system connected the staging system.

Finally all of implementation has been finished and ready to move production, EAI system is deployed into production environment.

I will describe more about deployment environment later.

3) Monitoring and fix

After starting deploying interface to staging system. (Open interface to business team). EAI project team monitors the interfaces. The interfaces always has errors like “Table comlumn mismatch”,”Message type missmatch” etc.

The error let the interface down and it brings complain from business development team.

In implementation phase, support business team to follow up their schedule by matching EAI interface implementation schedule. If delay is occurred, business team try to responsibilities to EAI team for the schedule delay.

And there are many requirement change like number of interfaces, type etc. So in the implementation phase, interface schedule management is very important.

Conclusion.

As i mentioned EAI project has a lot of dependencies to other system or project and their are many requirement changes about interfaces. In EAI project how to manage schedule and environment is very important to success EAI project

EAI project strategies-2. Requirements July 27, 2009

Posted by Terry.Cho in Architecture, EAI.
Tags: ,
6 comments
First pesepctive, EAI system requirements.
Waht is EAI system requirement. Commonly EAI system requirements are defined based on business like below.
Internal integration
This is integrating internal enterprise system. A lot of these types of integrations are “on-line”(real time).
Sometimes it uses XA based distributed transaction to support data integrities.
External integration
This type is B2B integration. System is located on external of the enterprise.
One of interest thing of this type is using dedicated network like dedicated TCP/IP or X.25 network
It impacts EAI architecture style. For example we have 2 X.25 line for some company. But we have 10 EAI system instances. What shall we do?
We have to consume request by all of the EAI instances and route the request to EAI instances that has been connected to X.25 line.
And we have to think about fail over when the X.25 connected EAI instance down.
In B2B style integration, it is very hard to use XA based distributed transaction. So we need another way to support data integraities like by using transaction trace logs etc.
Batch
In integration world there are two requirement about Batch. (Bulk data integration.)
One is for online business and the other one is for data anlaysis.
First thing is handling transaction data in online system. Apply data changes to the other system.
It happened about a few seconds later after original change has been made. We call it as “Defered or Near-Realtime”.
In this case, because it is online transaction data, integrity of the data is very important. So someimes XA is used.
Sencond case is for data analysis. Aggreate data into data warehouse and generate report. EAI has a role to gather data to warehouse.
This job is usally happened after office hour and amount of data is very huge. Commonly EAI doesn’t not handle this type of data trasfer.
This is ETL (Extract Transform Loading) area.
Simply i summarized EAI system business boundary based on business type.
But when we design EAI architecture, we have to think about MEP (message exchange patterns)-Sync,Async,Fire & Forget,Master Detail and Multiplicities (1:N,1:1 etc)
In addition OAM (Operation, Administration and Monitoring) requirements are defined in this phase.

First pesepctive, EAI system requirements.

Waht is EAI system requirement. Commonly EAI system requirements are defined based on business like below.

eaitype

Internal integration

This is integrating internal enterprise system. A lot of these types of integrations are “on-line”(real time). Sometimes it uses XA based distributed transaction to support data integrities.

External integration

This type is B2B integration. System is located on external of the enterprise. One of interest thing of this type is using dedicated network like dedicated TCP/IP or X.25 network.  It impacts EAI architecture style. For example we have 2 X.25 line for some company. But we have 10 EAI system instances. What shall we do? We have to consume request by all of the EAI instances and route the request to EAI instances that has been connected to X.25 line. And we have to think about fail over when the X.25 connected EAI instance down.

In B2B style integration, it is very hard to use XA based distributed transaction. So we need another way to support data integraities like by using transaction trace logs etc.

Batch

In integration world there are two requirement about Batch. (Bulk data integration.)

One is for online business and the other one is for data anlaysis. First thing is handling transaction data in online system. Apply data changes to the other system. It happened about a few seconds later after original change has been made. We call it as “Defered or Near-Realtime”.In this case, because it is online transaction data, integrity of the data is very important. So someimes XA is used.

Sencond case is for data analysis. Aggreate data into data warehouse and generate report. EAI has a role to gather data to warehouse. This job is usally happened after office hour and amount of data is very huge. Commonly EAI doesn’t not handle this type of data trasfer. This is ETL (Extract Transform Loading) area.

Simply i summarized EAI system business boundary based on business type. But when we design EAI architecture, we have to think about MEP (message exchange patterns)-Sync,Async,Fire & Forget,Master Detail and Multiplicities (1:N,1:1 etc) In addition OAM (Operation, Administration and Monitoring) requirements are defined in this phase.

EAI project strategies-1. Approach July 24, 2009

Posted by Terry.Cho in Architecture, EAI.
add a comment

Even if EAI (Enterprise Application Integration) has been introduced a few years ago. Enterprise Application Integration is still important in Enterprise Architecture. From this post i will explain a strategies to success EAI project.

How to approach EAI project ?

There is 4 perspectives to sucess EAI project.
eai1

Business Requirement

First, requirement for EAI.  Define target integrated business system (ERP,CRM,SCM etc). What kinds of interface will be used? (Socket? WebService? EJB? IIOP? MQ etc). How can we trace transaction and fault. What is target performance. What is message exchange patterns (Sync/Async/Batch etc).

In common EAI system can have a lot of capabilities, but no one can implement all of things. We have to extract EAI requirement based on customer’s requirement & environment.

Architecture

After defining requirement, we can design conceptual architecture.

Process

One of differences EAI is, EAI integrated other systems. It means it has a lot of  “Communication” in the project. We have to sync EAI interface between sender side of business unit and receiver side of business unit. In Enterprise , number of interface usually over  300~400 number. So managing communication and collaboration for implementing interface between different business units.

Development & Production Enviroment (Deploy)

As i mentioned in “Process”, EAI project has a dependency to other business system implementation project. In their implementation phase, EAI system has to provide integration between development environment of other business system. In addition EAI project team also has to have their own development environment.

In case of EAI system, it can have dedicated network line for B2B for example X.25 etc. So we have to consider deployment plan with the network line migration plan.

From next post i will explain about these 4 perspects with more detail.