jump to navigation

$ (dollar) prefix for angularJS January 17, 2014

Posted by Terry.Cho in Uncategorized.
Tags: , ,
add a comment

$ prefix in angularJS
It is for angularJS predefined service only.
It is just a naming convention from the below snippet http://docs.angularjs.org/tutorial/step_05

‘$’ Prefix Naming Convention
You can create your own services, and in fact we will do exactly that in step 11. As a naming convention, angular’s built-in services, Scope methods and a few other angular APIs have a ‘$’ prefix in front of the name. Don’t use a ‘$’ prefix when naming your services and models, in order to avoid any possible naming collisions.

Concept of AngularJS Service & Controller (Note) January 17, 2014

Posted by Terry.Cho in Uncategorized.
Tags: , , ,
add a comment

The service is some kinds of library which can be injected by using DI.
The service is singleton and binded to module
Angular injects dependencies using “constructor” injection.
and when it tried to use the service.

The service is injected as a parameter in constructior or it can be injected by using injection annotation ($inject)
The service can be injected to other services or controller

* Service dependency injection to Controller
angular.module(‘invoice2’, [‘finance2’])
.controller(‘InvoiceController’, [‘currencyConverter’, function(currencyConverter) {
The controller import module named ‘finance2’.
In Invoice Controller it tried to use ‘currencyConverter’, so the service is refered by the controller like “controller(‘InvoiceController’, [‘currencyConverter’, ”
and the try to use currencyConverter the parameter also has been passed to constructor function like this “function(currencyConverter)”

* Controller Parameter
[‘servicename’,’servicename’….,function(servicename,servicename)]
the ‘servicename’ is the service name will be injected to the controller
You can also alias service name in function like
[‘$scope’, ‘angularFire’, function(skop, af) {}]
the service $scope will be mapped to skop and angularFire will be mapped to af.
But is is not recommended to change the service name.
※ reference http://stackoverflow.com/questions/19238191/understanding-angular-js-controller-parameters

Service development process December 4, 2013

Posted by Terry.Cho in ALM (Application Lifecycle Management).
add a comment

1. Feature define

This is first step. It gathers user requirement and define features. The feature set can have about 2 depth. Big Feature group and detail feature.

For example if I want to develop “blogging service”

Feature (Lv1)

Feature Lv2

Detail

Post Article

Write text

Write article with text

Attach image

User can upload the image and attach the image in the article

Save as draft

User can temporary store the draft version of posting. The draft version is not posted to other user, only blog owner can see the contents of draft version……

Edit or delete article

After article has been posted, it can be modified later

To write the feature set. Feature should be listed based on user’s service usage flow (sequence)

For example. To use blog service, user has to register the blog service and log in. After log in, user can post article into the blog. The posted article will be displayed in blog. As u can see, It has a sequence.

Feature Lv 1.

Register

Log in

Post Article

Display Article

And the feature description should be enough. It has to be

  • Testable : the description should be enough that test engineer make a test case with the description.
  • UX design : the description should be enough that UX designer can design UX with the description

To define a feature there are some tips.

Competitor analysis

Prototyping

2. UX wireframe (Optional)

After feature define  has been done, it need to draw UX design. It called as “UX wire frame” which is skeleton of UX. For each feature UX flow needs to be defined and each page for the flow should be described. During the UX wireframe design, some missing feature or information can be defined like user profile field etc.


3. Review/rewrite feature and feature lock down

Whey we have to lock down feature?

In Agile methodology, the principal is “requirement is being changed” So in agile, requirement change is always acceptable. And team is defining requirement during the project. Basically I agree and during the project team members are being smarter. But in business world “forecast” is very important. Without feature, marketing team cannot make a plan for advertisement etc. without feature, team cannot estimate cost (for Software/Hardware purchasing). Such a SNS service agile is right. But most of service and product engineering to let other stakeholders plan for them (Marketing plan, recruitment plan, budget plan). Minimum feature should be fixed even if it can be changed later. Even in the big team with out fixed version of feature, making a new feature during the project brings a lot of communication burden.

However, I call this activity as a “Feature Lock down”. After feature has been defined and revised many ways we need to lock down feature.

What is feature lock down?

During the feature lock down phase, we revise the feature with UX wireframe, it will let us to find out missing feature. In addition with restricted resource and timeline, feature will be removed and prioritized to meet a schedule.

Initial version of feature is may be very business oriented term. It is not enough for development. So it need to be changed to more implementation oriented granularity.


4. Architecture design (Conceptual)

After feature lock down has been done, conceptual system architecture should be defined.

Business architecture

The business architecture contains business scenario and restriction etc. It has number of topics like below

1) Service model

Service model describes how the service works and how it makes a business (generate revenue). It should be described in 1~2 page with simple diagram.

  • What major feature is provided ?
  • Who use the feature?
  • How profit is generated?
  • What is external system to interact

2) Business strategy

Business strategy, which contains key value of the service. and strategy like time to market, target user, feature differences compare to other competitor’s service etc.

3) major feature

It describes major feature (level 1. feature) to explain about the service.

4) capacity planning

It has capacity like

  • total number of user
  • concurrent user
  • expected response time
  • TPS (throughput per second)

This capacity planning is very important to estimate system size and budgeting etc.

5) Business road map.

Not just for this phase of project, it also combines big service road map. Feature upgrade plan. or global deployment plan ( ‘14.1Q US, 14’2Q Europe etc)

And also need to define external system interaction like ERP,CRM system in company or 3’rd party solution interaction etc.

Technical architecture

1) Application Architecture

It describes how to the system works in application perspective. To explain it, technical architecture combines 4 things like below.

  • Component : It describes, which component compose the service. The best way to describing component is to user layer approach.

     Like depth 1,2,3, define high level component first and break down. and for the each level, component description should attached.

  • Component relationship : It describes relationship (& dependency) between components. 
  • Component flow per feature : To provide each feature, it explains how component interact together. If Component & Component relationship describes static view. It describes dynamic view per user feature. In the logic flow per feature, component is described and how they interact in time sequence and description of each interaction. 
  • Component interface : It describes interface spec and technology how the components interact each other. It combines interface technology like REST, Protocol buffer, RabbitMQ etc. and interface style (Message Exchange Pattern) Detail interface spec is not combined here. It will be described another document. 

MEP sample diagram from :  https://access.redhat.com/site/documentation/en-US/Fuse_ESB/4.3.1/html/Implementing_Enterprise_Integration_Patterns/files/MsgRout-Multicast.html

After this stage has been done, people can understand what we will make in technical perspective.

2) Solution Architecture

To realize the application architecture, we always need a solutions like web server, web development framework (spring, django etc), rdbms. In solution architecture, technology stack is described. In architecture concept define phase it doesn’t need to be described so detail

3) Technical Architecture (Deployment Architecture)

Technical architecture describes physical hardware or infrastructure architecture like server,disk and network device etc. Or nowdays, in this section Cloud deployment architecture can be described if you will use cloud. In concept phase, detail is not required. But at least what kind of infrastructure is required should be described. (Like Shared file system etc)

4) Data Architecture

data architecture describes, how data is stored, and where the data will be stored. In concept phase, domain modeling is very important. What kind of entity is involved. What is relationship between the entities. Even if i described full frame of Architecture design concept, it doesn’t need to be such detail. It is enough to explain big concept of architecture.

5. Technology selection & prototyping (Optional)

After big architecture has been defined, it needs to select solution. Now days software development is not “build from scratch” but “stitching solutions and software”. So selecting appropriate solution is very important. Actually this is not mandatory, but I highly recommend to have technology selection phase before start development. There are number of decision items to choosing the technology

  • Popular : Popular technology has a lot of information in web. (it means it can get a help or information from logs of blog etc).There are number of tips to measure technology popularity. Google trends (http://www.google.com/trends) shows interest of such keyword. For example if u want to compare two different solutions like mongodb vs couchdb. Google trends shows comparison of the search keyword. By using that u can recognize which solution is more popular
    another way is using “job search site”. If u find job position with technology keyword. You can find out which technology is more popular. By using same way u can get a same information from Amazon Book. “Stackoverflow.com” is also very good site to know the technology trend.
  • Supportable: Solution sometimes needs a support like a training, consulting sometimes maintenance like bug fix, enhancement etc. In case of commercial solution, it can be supported usally. But if there is no subsidiary in your country or don’t have skilled counsultant or engineer in you country, it is hard to get support. Open source case, subscription or commercial supported vendor is very useful in initial phase. If there is no commercial supported open source, the open source community should be strong to get a support from the community.
  • Easy to learn : Learning curve is very important
  • Cost
  • Reference
  • Market share
  • Meet in requirement

After you filter number of candidate technology, there is a way to measure the technology

  • PoC
  • BMT
  • Prototyping

6. Planning

Resource Planning

Budget Planning

7. Detail Architecture Planning

8. Interface design (On going)

9. Development environment set up

10. Development with Scrum (Iterative)

11. Retrospective and enhance. (Iterative)

12. QA

13. Production Release

14. Operation

 

Role & Responsibility

Thinking about JIRA Epic & User Story hierachy December 4, 2013

Posted by Terry.Cho in ALM (Application Lifecycle Management).
Tags: , , , ,
add a comment

The original problem that I have in JIRA.

JIRA & GreenHopper plug in are very great. But unfortunately. It just support 3 level depth like below.

  • Epic_1234: User Administration
    • User_Story_0100: Users can self-register
      • Task_8001: Create new page with registration form
      • Task_8002: Create user class to support all required data
      • Task_8003: Create methods to collect user data and insert into database
      • Task_8004: Create database schema to support user details

Epic > Story > SubTask.

In my experience. major feature for service are around 20~40. And in big project it can be more bigger. But JIRA green hopper Epic space is not enough to display whole things.

Idle requirement management way is

Here is very interesting article – http://q1systems.com/?p=418

They are using hierarchy Feature Category > Feature > User Story


But because of JIRA product restriction, these depth can not be supported in product.

Another workaround, I can imagine is use Excel together. Here is another plug in that I can use unlimited hierarchy in JIRA – http://almworks.com/structure/overview.html but in that case I cannot use GreenHopper.

Apache Tomcat Tuning Guide March 13, 2013

Posted by Terry.Cho in Java Programming, Performance & Capacity Planing.
Tags: ,
3 comments

Tomcat Tuning Guide

 

Tomcat configuration $Tomcat/conf/server.xml

Assumption

This configuration is optimized for REST/HTTP API call. And it doesn’t use any reverse proxy like Apache, NginX etc. We will reside simple L4 switch infront of tomcat groups.

In addition we will not use Tomcat Clustering, Session etc. So the clustering configuration is omitted.

Listener Setting

<Listener className=”org.apache.catalina.security.SecurityListener” checkedOsUsers=”root” /> 

checkedOsUser setting means Unix system user “root” cannot start Tomcat. If user starts tomcat as a root user it makes log file as a root user permission. In that case tomcat user cannot delete the log file.

<Listener className=”org.apache.catalina.core.JreMemoryLeakPreventionListener” /> 

This makes detect memory leak.

Connector Setting

protocol=”org.apache.coyote.http11.Http11Protocol” 

It makes tomcat use BIO. Tomcat has options for IO (BIO,NIO,APR). APR is fastest IO setting. It uses Apache web server IO module, so it is fastest. But it uses C code (JNI call), it can have a risk to kill tomcat instance. (with core dump). APR is more faster about 10% than BIO. But BIO is more stable. Use BIO. (Default is BIO)

acceptCount=”10″

It specifies server request queue length. If message is queued in the request queue, it means server cannot handle incoming message (it is overloaded). It will wait for idle thead and the request message will be pending. This setting reduce total size of request queue to 10. If the queue has been overflowed, client will get a error. It can protect server from high overload and let system manager to know the server has been overloaded.

enableLookups=”false”

In Java Servlet Code, user can look up request message origin (IP or URL). For example user in yahoo.com send request to server, and Tomcat try to resolve incoming request IP address. “enableLooksups” option enables return DNS name not a IP address. During this processing Tomcat look up DNS. It brings performance degradation. This option removes DNS look up stage and increase performance.

compression=”off” 

We are using REST protocol not a normal web contents like HTML,Image etc. This options allows to compress HTTP message. It consumes computing power but it can reduce network payload. In our environment compression is not required. It is better to save computing power. And in some particular Telco network, compression is not supported.

 connectionTimeout=”10000″

It is HTTP Connection time out (client to server). It is milliseconds. (10,000 = 10 sec).

If server cannot make a connection from client til 10 sec. It will throw HTTP time out error. In normal situation, our API response time is under 5 sec. So 10 sec means, server has been overloaded. The reason why I increased the time up to 10 sec is, depends on network condition, connection time will be deferred.

maxConnections=”8192″

The maximum number of connection, tomcat can handle. It means tomcat can handle maximum 8192 socket connection in a time. This value is restricted by Unix system parameter “ulimit –f” (You can check up in unix console)

maxKeepAliveRequests=”1″

As I mentioned above, this configuration is optimized to REST API request not a common web system. It means client will send REST API call only. It sends the request and get a response. Client will not send request in a short time. It means we cannot reuse the connection from the client. So this setting turn of HTTP Keep Alive. (After response the request from client, tomcat disconnect the connection immediately)

maxThreads=”100″

This defines total number of thread in Tomcat. It represents max number of active user at that time. Usually 50~500 is good for performance. And 100~200 is best (it is different depends on use case scenario).

Please test with 100 and 200 values and find value for performance. This parameter also get a impact from DB connection pool setting, even if we have a lot of thread , and the total number of db connection is not enough, the thread will wait to acquire the connection.

tcpNoDelay=”true”

This allows us to use TCP_NO_DELAY in tcp/ip layer. It makes send small packet without delay. In TCP, to reduce small package congestion, it gathers small packet to tcp buffer until it has been filled and send the packet. TCP_NO_DELAY option makes send small packet immediately even though TCP buffer is not full.

 

JVM Tuning

Java Virtual Machine tuning is also very important factor to run Tomcat

The focus of JVM tuning is reducing Full GC time.

-server

This option makes JVM to optimize server application. It tunes HotSpot compiler etc internally. This option is very important and mandatory in server side application

-Xmx1024m –Xms1024m -XX:MaxNewSize=384m -XX:MaxPermSize=128m

This memory tuning options, our infrastructure is using c1.mediuem amazon instance, so the available memory is about 1.7 gb total. Heap size is 1G and let them to have fixed size. It defines max 1Gb, min 1Gb heap size. The NewSize is 384mb (1/3 size of total heap size). 1/3 New Size is best performance usually. Perm size is defines area of memory to load class. 64mb is enough. But we will use 128m first time and tune based on gc log analysis later.

Total physical memory consumption is 1G heap + 128mb perm = 1.128 GB and JVM internally uses memory to run JVM itself. It consumes about 350~500mb. So total estimated required memory is about 1.128GB+500m = 1.5 GB.

As I mentioned, c1.mediuem size has only 1.7GB physical memory. If consumed memory exceeds actual physical memory, it makes disk swapping. If JVM memory is swapped out to disk, the performance is significantly degraded. Please take care swapping is not occurred.

-XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./java_pid<pid>.hprof

These options are for trouble shooting “OOM (Java Out Of Memory Error”. If out of memory error has been occurred. The memory layout will be dumped to disk. The location of dumpfile is specified by “-XX:HeapDumpPath” option

 -XX:ParallelGCThreads=2 -XX:-UseConcMarkSweepGC

These options specify GC strategy. It uses ParallelGC for Minor collection and 2 threads will be used for the Minor GC. And for Old area, concurrent gc will be used. It will reduce Full gc time

-XX:-PrintGC -XX:-PrintGCDetails -XX:-PrintGCTimeStamps -XX:-TraceClassUnloading -XX:-TraceClassLoading

These option specifies GC logging. It logs the GC log detail to stderr (console output). It shows usage trend os Java Heap memory, time stamp etc. (it contains old,new & perm area usage).

Especially, ClassLoading & UnLoading option show what class is loaded and unloaded to memory. It helps us to trace Perm Out of memory error.

 

Reference : http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

Logging

  1. 모든 log는 catalina.out 하나의 파일에 모두 쌓이게 할것 à LogAppender를 Console Appender로 변경하면 된다.

①     Tomcat 자체가 쓰는 로그 (별도로 지정 안하면 원래 Console로 나옴)

②     Application에서 LogBack을 이용해서 로깅 되는 로그 (별도로 ConsoleAppender를 개발단에서 정의해줘야 함)

③     GC 로그 (별도로 지정 안하면 원래 Console로 나옴)

DB Connection Pool Setting

Please use tomcat dbpcp connection pool. Apache-common connection pool is not updated frequently. Tomcat dbcp connection pool is updated well.

The basic setting guide is “Let number of connection in the pool to keep exact number of connection”. It can be done by set min conn and max conn to same number.

Total number of connections (including read + write) should be around. 70~80. We are using 100 threads in one tomcat instance. The 70~80% will use db connection at the same time.

 

Please attach. Current connection pool settings here

 

Library Setting

Developer packaged java lib inside war file (WEB-INF/lib) it can increase usage of perm memory. And sometime can bring confusion about “which lib is actually used”. It means if same lib(jar file) resides in $TOMCAT_HOME/lib and WEB-INF/lib. The lib in $TOMCAT_HOME/lib wil be used and lib is WEB-INF/lib will be ignored.

To solve this problem. Remove common library like my-sql-jdbc driver and dbcp lib jar file from WEB-INF/lib and move it into $TOMCAT_HOME/lib

Apache Cassandra Quick tour March 22, 2010

Posted by Terry.Cho in Distributed System.
Tags: , , , , ,
20 comments

Cassandra is distributed database system. It is donated to Apache open source group by Facebook at 2008.The Cassandra is based on Google Big Table data model and Facebook Dynamo distributed architecture. It doesn’t use SQL and optimized to high scale size of data & transaction handling. Even though Cassandra is implemented with Java language, other language can use the Cassandra as a client. (It supports Ruby,Perl,Python,Scala,PHP etc).

It is used to High Scale Size SNS like Face book,Digg,Twitter etc. It doesn’t support complex relationship like Foreign Key. It just provides Key & Value relationship like Java Hashmap. It is very easy to install and use.

Let’s look at data model of Cassandra

Data Model

Cassandra is based on google big table data model. It is called “Column DB”. It is totally different from traditional RDBMS.

Column

Column data structure which consists of column name and column value.

{name: emailAddress, value:cassandra@apache.org}
{name:age , value:20}

Column Family

Column family is set of columns. It is similar to row in RDBMS table. I will explain more detail about difference between Column Family and row in RDBMS later. Column Family has a key which identify each row in data set. Each row has a number of Columns.

For example, one row is

Cassandra = { emailAddress:casandra@apache.org , age:20}

“Cassandra” is key for the row, and the row has two columns. Keys of the columns are “emailAddress” and “age”. Each column value is “casandra@apache.org” and “20”.

Let’s look at Column Family which has a number of rows.

UserProfile={
Cassandra={ emailAddress:”casandra@apache.org” , age:”20”}
TerryCho= { emailAddress:”terry.cho@apache.org” , gender:”male”}
Cath= { emailAddress:”cath@apache.org” , age:”20”,gender:”female”,address:”Seoul”}
}

One of interest thing is each row can have different scheme. Cassandra row has “emailAddress” ,”age” column. TerryCho row has “emailAddress”,”gender” column. This characteristic is called as “Schemeless” (Data structure of each row in column family can be different)

KeySpace

Keyspace is logical set of column family for management perspective. It doesn’t impact data structure.

Super Column & Super Column Family

As I mentioned earlier, column value can have a column itself. (Similar to Java Hashtable can have ValueObject class as a ‘Object’ type)

{name:”username”
value: firstname{name:”firstname”,value=”Terry”}
value: lastname{name:”lastname”,value=”Cho”}
}

As a same way column family also can have column family like this

UserList={
Cath:{
username:{firstname:”Cath”,lastname:”Yoon”}
address:{city:”Seoul”,postcode:”1234”}
}
Terry:{
username:{firstname:”Terry”,lastname:”Cho”}
account:{bank:”hana”,accounted:”1234”}
}
}

UserList column family has two rows with key “Cath” and “Terry”. Each of the “Carry” and “Terry” row  has two column families – “Cath” row has “username” and “address’ column family, “Terry” row has “username” and “account” column family.

Cassandra Quick Test

Download Cassandra from http://incubator.apache.org/cassandra/ Extract zip file and run bin/cassandra.bat

We will connect Cassandra node with CLI interface. It is located in /bin/cassandra-cli.bat

The default TCP port number is 9160. You can change the port number in “conf/storage-conf.xml”

In “/conf/storage-conf.xml” file, default key space with name “Keyspace1” is defined. Column family type of the Keyspace is like this

Let’s put a new row with key name “Terry” which has Column (key=”gender”, value=”Male”)

Replication architecture in Cassandra and HBase March 19, 2010

Posted by Terry.Cho in Distributed System.
Tags: , , , ,
4 comments
Now i’m research distributed database architecture.
I found a very interesting article.
Apache Cassandra and Hadoop Hbase are most popular distributed database.
Twitter and Facebook are using Cassandra.
These solution is started from Google Big Table. So the data model is very similar.
The data model is called “Column database”. I will introduce the model later.
However my concern is how to replicate data across region (data centers in different region)
Here is very interesting information.
In case of Cassandra, it replicates data in every transaction. A coordinator captures changes and propagate it to other nodes.
But fiber based low latency network is required and there are no reference yet.
HBase data replication architecture looks very practical.
It captures change log and put it into replication queue. The replication message is passed to other nodes.
This mechanism is very similar to CDC (Change Data Capture).
Oracle Goden Gate, Quest Share Flex, MySQL geo replication are using this mechanism.

HBase replication looks more reasonable. It has common architecture and they have a reference.

===

After i had written this article, i got a feed back. Followed by the comment the article which i referenced is written by  fan of Hbase. Cassandra supports geo replication and has reference in face book. And Digg will deploy Cassandra in different data center.

But as i know even if facebook has two data center, they have fiber-link between the center. It is not a real geo replication. I will more research about cassandra data replication feature and re-post about this issue later.

ENTERPRISE 2.0 BY ANDREW McAFEE December 2, 2009

Posted by Terry.Cho in Enterprise 2.0.
Tags: , ,
add a comment

I’m reading the book “ENTERPRISE 2.0 BY ANDREW McAFEE”. I am a previewer in some Korean publishing company. They sometimes ask me about what book deserved to translated to Korean.

Yesterday i got the book for a preview. Actually i was waiting for the book. Andrew Mcafee is professor in Havard business school who is one of most famous people in E2.0 world.

As i thought the book looks great, it introduced case study and reason why we need E2.0 and introduction of E2.0 , success strategy etc.

I’m reading this book with excitement.

Advanced REST November 13, 2009

Posted by Terry.Cho in Uncategorized.
Tags: ,
add a comment

The basic concepts of REST were covered in Chapter 1. Still REST can be implemented in a more advanced version if utilizing the merits of HTTP.I would like to explain about the advanced REST in this Chapter with an example by using http://www.infoq.com/articles/subbu-allamaraju-rest .Let’s assume that the business scenario is making an account transfer from a bank.

1. Implementing an account transfer scenario through internet banking

STEP 1. Log-in to the internet banking system.

STEP 2. View the list of accounts of the relevant user through user ID.

http://bank.org/accounts?findby=7t676323a

200 OK
Content-Type: application/xml;charset=UTF-8
<accounts xmlns=”urn:org:bank:accounts”>
<account>
<id>AZA12093</id>
<customer-id>7t676323a</customer-id>
<balance currency=”USD”>993.95</balance>
</account>
<account>
<id>ADK31242</id>
<customer-id>7t676323a</customer-id>
<balance currency=”USD”>534.62</balance>
</account>
</accounts>

Regarding user ID 7t76323a, the two accounts of AZA12093 and ADK31242 will be returned along with the balance in those accounts. As the user made inquiry on each account number, the details of each account will be http://bank.org/account/AZA12093

http://bank.org/account/ADK31242

STEP 3. To see the business scenario of making account transfer,

POST /transfers
Host: bank.org
Content-Type: application/xml;charset=UTF-8

<transfer xmlns=”urn:org:bank:accounts”>
<from>account:AZA12093</from>
<to>account:ADK31242</to>
<amount currency=”USD”>100</amount>
<note>RESTing</note>
</transfer>

100 USD is transferred from AZA12093 to ADK31242 through HTTP post. Upon a successful account transfer, the following result value will be returned.

201 Created
Content-Type: application/xml;charset=UTF-8
<transfer xmlns=”urn:org:bank:accounts”>
<from>account:AZA12093</from>
<to>account:ADK31242</to>
<id>transfer:XTA8763</id>
<amount currency=”USD”>100</amount>
<note>RESTing</note>
</transfer>

In this case, take careful note that the return value is not HTTP 200 but HTTP 201. As the account transfer takes place not immediately but afterwards through async, HTTP 201 Created (meaning that the account transfer request has been received) as well as Transfer Application No XTA8763 are returned.

STEP 4. To check the account transfer details after one day,

http://bank.org/check/XTA8763
GET /check/XTA8763
Host: bank.org
200 OK
Content-Type: application/xml;charset=UTF-8
<status xmlns=”urn:org:bank:accounts”>
<code>01</code>
<message xml:lang=”en”>Pending</message>
</status>

The relevant account transfer request is indicated in a pending status.

2. Implementing an account transfer scenario through internet banking using the advanced REST

At a glance, it seems like an advanced and well-developed REST, but actually cannot be called as a successful architecture with the real merits of REST incorporated.

REST should have the following features:

First, REST should be able to identify the resource by using URL.

Second, REST should be able to represent various approaches toward the resource by using many functions of the HTTP protocol – especially the HTTP Header. For example, the Message Exchange Pattern of Sync/Async type (Call Back using the correlation ID) as well as the Meta Data like ‘Last Update Time’ to use web cache should be defined in the HTTP Header. Also the input and output data format should be defined based on the contents type.

Third, REST should be able to represent the relation between resources or the status information of the current resource through a link.

Based on such features of REST, let’s develop the previously-created account transfer scenario again.

STEP 1. Log-in to the internet banking system.

STEP 2. View the list of accounts of the relevant user through user ID.

200 OK
Content-Type: application/vnd.bank.org.account+xml;charset=UTF-8
<accounts xmlns=”urn:org:bank:accounts”>
<account>
<id>AZA12093</id>
<link href=”http://bank.org/account/AZA12093&#8243; rel=”self”/>
<link rel=”http://bank.org/rel/transfer edit”  type=”application/vnd.bank.org.transfer+xml”  href=”http://bank.org/transfers”/&gt;
<link rel=”http://bank.org/rel/customer&#8221;
type=”application/vnd.bank.org.customer+xml”
href=”http://bank.org/customer/7t676323a”/&gt;
<balance currency=”USD”>993.95</balance>
</account>
<account>
<id>ADK31242</id>
<link href=”http://bank.org/account/ADK31242&#8243; rel=”self”/>
<link rel=”http://bank.org/rel/transfer&#8221;
type=”application/vnd.bank.org.transfer+xml”
href=”http://bank.org/transfers”/&gt;
<link rel=”http://bank.org/rel/customer&#8221;
type=”application/vnd.bank.org.customer+xml”
href=”http://bank.org/customer/7t676323a”/&gt;
<balance currency=”USD”>534.62</balance>
</account>
</accounts>

As explained above, a slightly different type of return value will appear. What to check first is the content-type. : application/vnd.bank.org.account+xml; the return type of content-type will be returned. First, +xml mean that the format of this document is xml. vnd.bank.org.account defines the structure of returned data. (One of the weak points of REST – undefined data type – can be resolved by assigning an ID to the unique name of XML schedule used as an input or output)

Another change is that the LINK part has been added. This LINK indicates how the status of the current resource can change and what kind of URL can be used for status change. Also, it can indicate the relations with another relevant resource and define the data type, which is returned upon calling, as content-type. If the data type and the relation with another resource are defined in the returned message, user will be able to understand how to use and the relation of the resource without referring to a separately defined service. Therefore such feature of REST is called a ‘self-descriptive message’.

<account>
<id>ADK31242</id>
<link href=”http://bank.org/account/ADK31242&#8243; rel=”self” type=”application/vnd.bank.org.account+xml”/>
<link rel=”http://bank.org/rel/transfer&#8221;
type=”application/vnd.bank.org.transfer+xml”
href=”http://bank.org/transfers”/&gt;
<link rel=”http://bank.org/rel/customer&#8221;
type=”application/vnd.bank.org.customer+xml”
href=”http://bank.org/customer/7t676323a”/&gt;
<balance currency=”USD”>534.62</balance>

In this case, there are 3 types of changes: Self, http://bank.org/rel/transfer and http://bank.org/rel/customer

Firstly http://bank.org/rel/transfer, which is an account, defines the relation with a “transferable account”. To make account transfer, send through http://bank.org/transfers URL and the return value will become application/vnd.bank.org.transfer+xml.

Secondly self indicates a more detailed information of the account itself. It can be inquired through http://bank.org/account/ADK31242 and the returned data type will be application/vnd.bank.org.account+xml.

Thirdly http://bank.org/rel/customer indicates the client information. It can be inquired through http://bank.org/customer/7t676323a and the returned data type will be application/vnd.bank.org.customer+xml.

STEP 3. Execute account transfer.

Send the following data to the URL that has been returned from STEP 2.

POST /transfers
Host: bank.org
Content-Type: application/vnd.bank.org.transfer+xml;charset=UTF-8
<transfer xmlns=”urn:org:bank:accounts”>
<from>account:AZA12093</from>
<to>account:ADK31242</to>
<amount currency=”USD”>100</amount>
<note>RESTing</note>
</transfer>

The return value is as follows.

201 Created
Content-Type: application/vnd.bank.org.transfer+xml;charset=UTF-8
<transfer xmlns=”urn:org:bank:accounts”>
<link rel=”self” href=”http://bank.org/transfer/XTA8763″/&gt;
<link rel=”http://bank.org/rel/transfer/from”type=”application/vnd.bank.org.account+xml”href=”http://bank.org/account/AZA12093″/&gt;
<link rel=”http://bank.org/rel/transfer/to” type=”application/vnd.bank.org.account+xml” href=”http://bank.org/account/ADK31242″/&gt;
<link rel=”http://bank.org/rel/transfer/status” type=”application/vnd.bank.org.status+xml” href=”http://bank.org/check/XTA8763″/&gt;
<id>transfer:XTA8763</id>
<amount currency=”USD”>100</amount>
<note>RESTing</note>
</transfer>

STEP 4. Check the status of the account transfer progress.

The status of the account transfer progress can be checked by using http://bank.org/check/XTA8763 which has been returned from STEP 3.

Conclusion

From this article, I tried to focus on defining the data types by using the content-type of HTTP as well as the relations between resources by using a link. Yet I haven’t found the “perfect” standardized design even though there are many REST related articles that embrace similar philosophies.

Of course, as shown in the aforementioned examples, the in/out data format can be defined based on the content-type. Although the out (return) data format can be defined based on the defined types in the link, still the input aren’t defined. Also according to other designs, there are ways to define the data type by defining the URL of the real XML scheme in the URL of XML namespace.

From a protocol point of view, a link can be used by defining the relations between resources. This approach sounds preferable but actually can create various constraints during the actual implementation.

This article only presents a mere example of REST design in a more advanced version by utilizing HTTP more efficiently. However, more considerations are required for a more practical REST design.

=======

Comment

Please kindly consider that this article covering the advanced style of REST design is yet an uncompleted version and needs to be supplemented with more feedback. I will try to introduce the advanced style – especially regarding LINK and data type – with more details next time. In my following article, you will be able to learn about how REST is actually implemented in JAVA.

REST Architecture overview November 12, 2009

Posted by Terry.Cho in Uncategorized.
Tags: , , ,
1 comment so far

REST Architecture

REST was introduced in 2000 from a thesis written by Roy Fielding, one of the founders of web (HTTP). As the current architecture wasn’t able to make full use of the superiority of the original web design, the Representational Safe Transfer (REST) was introduced as a network-based architecture that could best utilize the merits of web.

Basics of REST

Simply put, REST is a HTTP URI + HTTP Method, which clearly states the target resource through URL and defines the operations of such relevant resource through Method.

Resource

One of the key features of REST is representing all resources as ‘Resource’. This Resource is expressed through HTTP URL. For example, a user named ‘bcho’ from a javastudy website can be expressed as http://www.javastudy.co.kr/users/bcho while a HP printing machine located at the 9th floor of a Gangnam office can be expressed as http://printers/localtion/seoul/kangnamgu/9f/hp. In this way, all resources can be expressed through HTTP URL.

Action

Then how are the operations of the relevant resource represented? In this case, HTTP Method is utilized.

*  In order to bring the member information of bcho from a javastudy website,
URI : http://www.javastudy.co.kr/users/bcho
Method : GET

* Also in order to create the relevant member,
URI : http://www.javastudy.co.kr/users/bcho
Method : POST
PayLoad
<payload>
<name>Cho Dae Hyup</name>
:
</payload>

*  In order to delete the relevant member,
URI : http://www.javastudy.co.kr/users/bcho
Method : DELETE

*  In order to change the information of the relevant member,
URI : http://www.javastudy.co.kr/users/bcho
Method : PUT
PayLoad
<payload>
<name>Cho Dae Hyup</name>
:
</payload>

* In other words, the 4 Methods from the HTTP Protocol define CRUD of the Resource.

HTTP Method Meaning
POST Create
GET Select
PUT Create or Update
DELETE Delete

Shortcomings of REST

However, there are several shortcomings such as the number of available Methods is only 4. For example, Methods like ‘send email’ or ‘log write’ cannot clearly be expressed through HTTP Method.

As the existing programming style has taken an operation-oriented approach centered on functions or Methods, those methods conflict with the resource-based approaches that REST embraces. The reason for REST being called as architecture rather than a simple protocol is that it can be appropriately applied to resources that have CRUD (ex. DBMS).

Then how can these shortcomings be resolved?

As a matter of fact, CRUD is not enough to express all operations. To express operations with control or functional features, HTTP/PUT or POST Method is used or a functional approach is required. For example, a ‘send mail’ operation can be changed into a meaning of ‘create a mail to send it to someone’ through HTTP/POSThttp://www.xxx./sendmail/to/{emailaddress}.

Still, there are cases that cannot change the meaning of the context.

In these cases, HTTP/PUT or URL is used to grant the meaning of control. The grade of user ID bcho can be changed through the following.

For example, http://www.xxx/users/bcho/upgrade

In fact, the most difficult challenge in designing a REST-based architecture is how to define this URL. One of the merits of REST is that it is quite easy to grasp the meanings through this URL or HTTP Method. Therefore, much effort is required in defining the URL.

Pros & Cons

Pros

The existing web infrastructure can be utilized in tact.

This is one of the largest advantages. As the existing HTTP is used as it is, there is no need to break a firewall when making a remote calling and the load balancer equipment like L4 can be used in tact.

The exciting thing is that the web cache can be used as it is. Because every resource is expressed uniquely through URL, it can be stored within the web cache and especially operations that are selective can be returned by this cache without going through actual business transactions. Such features are strongly beneficial from performance and resource utilization perspectives.

Easy

Compared with the web service, which encompasses so many cumbersome SPECs like WS*-I, WS Reliable Messaging, WS Transaction, REST doesn’t need a separate SPEC. Generally REST is called as a ‘Defactor standard’, which only requires proper use of HTTP URL and Method.

Cons

No standard and, therefore, hard to manage

The reason that REST is drawing much public attention these days is because non enterprise companies like Google, Yahoo and Amazon are eyeing away from the complexity and difficult standards of web service. As the meanings of data don’t sound like mission critical business requirements, a standard that is easy enough to exchange data transactions would be satisfactory and doesn’t need to be at an enterprise-level. Also there is no company like vendors that wants to take the lead in creating a standard.

There are only standard-looking ones that are used frequently and being created tacitly. (These are called as ‘Defactor standard’.)

As the standards are ambiguous, it becomes a problem in managing them during development. Considering that a clear-cut standard paves the way for a development process or pattern to be created in accordance with several SPECs, a proprietary standard of REST should first be established and used to design a REST-based system. However, in some cases, a misunderstanding on the REST concept can place a REST flag on the wrong architecture. In fact, Flickr.com – a leading runner of WEB 2.0 – once evoked a controversy around REST architecture by putting the name, ‘hybrid REST’ on the API that was designed in a RPC style without internalizing the REST features.

Note. When Flickr’s Hybrid Rest processes a certain operation,

Methods are handed over to query string in the form of http://URL/operation?name=operationname. At a glance, this looks like a RESTful design. But actually the URL for every resource is the same and the operations are simply divided by query string. This goes against the original design principle of REST, which grants a unique URL to every resource.

However, it is worrisome that many local people misunderstand such kind of design as REST and some portals actually offer this kind of services.

As mentioned-above shortly, REST is not a protocol like web service. It is architecture. Because it is a resource-based architecture, system should be designed suitable to the REST concept.

Use of Alternative Key

For example, resource is usually represented by one row from the DB. In the case of DB, primary keys exist in the format of complex keys. (Multiple columns are combined to become one PK, in this case) Although this can be a valid design for the DB, HTTP URL has a hierarchical structure according to / and, therefore, such representation becomes unnatural.

For example, if the PK of DB is defined as “residence registration # of the householder” + “region” + “name,” there is no problem expressing like such in the DB. But this way of definition becomes unnatural (having a strange meaning) when being expressed in REST because it would look like userinfo/{ residence registration # of the householder }/{ region }/{ name}.

In addition, there are many problems in assigning a unique key to a resource and one way to resolve this problem is to use an alternative key (AK). In this case, a unique key with no meaning acts like a key and be used in a field called ‘AK’ within the DB table. Already Google’s REST adopted an architecture using such kind of AK.

However, adding an AK field to the DB means the overall DB design needs to be changed. If this is the case, an architectural approach is required when using REST because the architecture of the overall system must be changed for REST.

Other options to make a RESTful design can include 1) techniques that express the relations between resources through href (link), 2) versioning method, 3) naming rule, 4) cross cutting concern transaction by using ESB, 5) routing, etc. The architecture design method and highly-advanced REST will be covered in the next contribution, titled ‘Highly-Advanced REST Architecture’.

Misunderstanding on REST

REST = HTTP + XML Protocol?

I once proposed to have a discussion on REST at one local community web site on the grounds of designing a highly-advanced REST system for the project. Then, I became to realize that most people simply understood REST as something sending an XML by using HTTP.

Never

REST is an architecture that expresses the resource by best-utilizing the web features. (It is not a protocol) And of course, it must use HTTP. However, XML is not mandatory. It is still ok to use other languages like JSON or YAML. Depending on how well the resource is represented and the web features are utilized determine a correct understanding on the REST architecture concept.

The reason why I am writing this paper is because there is no document that explains the REST concept or principle well enough.

Is REST easier than WebService?

Not exactly. Of course, REST would be much easier than Web Service if being developed from scratch. Because creating a proprietary standard, designing a message with a simple XML, and simply sending that message through HTTP would be required only. However this is looking from a service provider prospective. For those who actually use this service, any XML or JSON data being returned upon request to the HTTP client need to be parsed one by one.

What about web service? Thanks to its clear-cut standard, a web service will be created automatically once the coding takes place based on POJO or JAX-WS. Especially the client stub can be created automatically through WSDL according to the given service contract. For this reason, even though users don’t know the protocol spec at all, web service can be easily utilized as if calling and using a java library.

From my opinion, it looks easier to develop a web service rather than REST. The simplest and the most productive option is a WS-I based web service that is simple enough and well arranged.

Prospect of REST

Not much demand for REST exists in Korea yet. Maybe this is because of users trying to avoid designing highly complicated system or maybe because not much demand has been created for an open-style system.

However already prestigious overseas web sites are offering services under a REST-based architecture. Amazon, one of the representative open API companies is planning to do a REST-based migration on the open APIs previously developed with web service. Although REST is a defector standard, it is known as a difficult standard, which is unavoidable in the web world.

In addition, REST rising from the open side is about to be included in the next JEE6 version as being added as a JAX-RS (JSR311), one of J2EE Specs. (For an open source, REST is included in Apache CXF and a framework called ‘Jersey’ of Sun. Even though REST is translated into a specification, it would be a specification about ‘how’ to implement and still its flexibility will stay unchanged.) A specification called, ‘WADL’ which is similar to ‘WSDL’ of web service has been released as a standard for REST. However, ‘WADL’ only indicates URLs for REST service and yet representing the schema of every message within the actual operations. For this reason, it is right to say that there is no standardized service contract for REST yet, and it is difficult to judge that whether this would serve as an advance or disadvantage in the future.

Conclusion

For its advantage of simplicity and maximum utilization of web features, REST is being spread out surprisingly in the overseas region. As a rare technology developed from the open side, REST has even been adopted as a standard technology (JSR-311).

Still, due to its strong flexibility, REST cannot be controlled or managed easily and, for this reason, hasn’t yet been widely used in the Enterprise System but instead being mostly used for service system. Due to a lack of understanding on the REST concept in Korea as well as inactivated open API, REST is not a popular option yet.

However, it is time for local developers to make preparation for using REST as this concept is becoming main-stream among prestigious international service companies.

In the next contribution, a more sophisticated REST architecture will be covered.