Did You Know There is a "C" in SOA?

Don't Forget the Consumer in Service-Oriented Architecture

by Gus Björklund, Vice President, Technology, Progress Software Corporation

This article originally appeared in SOA Web Services Journal, September 2006.

When designing your SOA and services, keeping service consumers in mind will make the job easier. Consumers must conform to the interfaces of each service they use and invoke them with the right data in the right format. The more similarity there is among the services, the less coding and translation your consumers will have to do. Using the techniques of transformation, semantic data modeling, and a conceptual data model can make your job much easier both during initial design and testing and when making changes later.

The interface specifications, protocols, and data formats used in SOA and Web-services are designed to create services with "loose coupling" between a service consumer and the service provider. The service provides a consumer with an abstraction representing some business function so that the consumer doesn't have to be concerned with the details of how the function is implemented or how and where its data are stored. The only thing the consumer has to know is how to call the service using standards like WSDL, SOAP, HTTP.

Once you have implemented some services, you can then compose higher-level business functions and so-called "composite applications" by invoking multiple services and orchestrating interactions among them. If you build all your services from scratch for a project, you can design interfaces and data models that interact smoothly and consistently. Opportunities for such "green field" designs are rare and today's new system is tomorrow's legacy system anyway. More than likely, you will be using existing as well as new services and applications that were bought or built long ago and have already, or soon will have, service interfaces added to them.

The Fly in the Ointment

An application's service interface may be new, but it probably won't be clean and consistent with your other services since an application's functionality and existing interfaces usually dictate what can be provided as a service and the data it can accept and provide. Interacting with multiple services can become complicated -- even though they use standard protocols and formats for interface definitions and invocation, these protocols and formats only deal with the syntax of the data and not with their vocabulary or meaning. Each application or service is likely to use different names for similar things and the same name for different things. Such disagreements and mismatches are commonplace and not limited to service-oriented architectures. Here are a few examples:

• Many of us have an address book on our computers and another on our cell phones and software that synchronizes them. Unfortunately the definition of an entry in each is different, with variations field names and the number of fields. One device may have fields for home, work, cell, fax, and modem telephone numbers and an email address and the other the other device home, work, mobile, pager, assistant, and main numbers and multiple email addresses. The synchronization software knows how to map the differing names but it can't map all the fields one-to-one. Still, it makes "good enough" compromises because most people don't use all the fields.
  
• Many companies provide access to an external CRM system like SalesForce.com for sales people and use an internal ERP system for processing orders. Those two systems keep different information about customers so using the Salesforce.com web service interface to get customer information and then correlating it with data from ERP system may not be straightforward. Like the address book, the data formats are different, but so is the intended use and content of the data. One keeps track of things like prospects and their interactions with the sales rep, and the other keeps track of things like what someone has bought, how much they paid, and when.
  
• The two systems both have a notion of "customer" but one may call the customer identifier "cust_num" and the other something like "csidno", and one may contain a number and the other letters and numbers. One may have "orders" represented as an XML document with elements for "orderlines" and the other an "orders" XML document with the elements "orderheaders" and "orderdetail". Other data elements will present similar issues. Reconciling or mediating the differences in data models and service interfaces for applications like these is more difficult than reconciling differences in address books.
  

The consumer bears responsibility of providing the correct data elements to the service in accordance with its interface definition. In many situations, the service designer gets to decide what it wants to use. When you design service interfaces, you should try to make it as easy as possible for the consumers to use. That isn't always possible, especially if you use services provided by someone else, like another department or an external organization like SalesForce.com, UPS, or Google.

Transformation to the Rescue!

To mediate the differences among services' data formats, you can introduce a data transformation or mediation layer into the architecture. This layer becomes responsible for mediating data formats between service invocations, performing conversions and translations for the consumer. The consumer then has the data in the right format when it wants to invoke a service using data supplied to it by another service.

You can construct the transformation operators using XQuery, XSLT, or even hand coded Java functions to translate one data layout and vocabulary to another with relative ease. If you are using an Enterprise Service Bus (ESB) as a backbone for reliable communication and service management, you can put your transformation services on the bus. The ESB can then invoke the transformations automatically as needed.

When you build transformations one by one as you need them, the result is a set of transformations for converting data received from service A into the right form for sending to service B, service A to service C, service B to service C, and so on. Each transformation does one specific job in isolation from the others. Now everything will run smoothly, right?

Transformation Is Not Enough

While transformation is necessary and helps to make service-integration easier, by itself it isn’t enough. Even when you use good tools like DataDirect's Stylus Studio or Altova's XMLSpy to create them, having many such point-to-point transformations can make things complicated, with the following disadvantages:

• It reintroduces the tight coupling SOA was supposed to eliminate in the first place. Instead of providing agility, your SOA will become brittle and difficult to change.
  
• We've spent more than 50 years inventing new applications and their data models so there are likely to be many and complicated mismatches between applications. Every organization has many mismatches in their systems. Dealing with them can keep you busy for along time.
  
• It’s pretty likely that you'll end up with duplicate sections and duplicate operations in your transformations, perhaps with unintentional variations, especially if they are made by different people.
  
  
• Maintaining hundreds or thousands of transformations takes a lot of work and keeping them all correct, consistent, and current will be next to impossible.
  
• The individual transformations encapsulate knowledge of the data semantics in many different places. That knowledge can't be easily found or reused effectively.

Transformations certainly help but they aren't enough.

Conceptual Data Models to the Rescue!

Instead of many individually designed point-to-point translations, if you define mappings from each service to a common conceptual data model then it will be possible to generate the transformations you need automatically and use them where needed. This can greatly reduce the amount of work you have to do because instead of defining transformations for pairs of services, you define one mapping for each service. Then the necessary point-to-point transformations are generated from the mappings from service to conceptual model. Note that there is no need to actually transform to and from the conceptual model (that's why it is conceptual, instead of real). With the right tool for defining and extending the conceptual data model and service mappings and storing them, this approach solves a number of the problems we mentioned previously.

As you define and extend the conceptual model, you can use clear and understandable names for all the data elements no matter how cryptic they may be in the service interfaces. When the same data element is used by multiple services, the modeling tool can show them all together enabling you to see where an element is being used even when the names are different.

Data Validation

The service interface standards we have now don't include much ability to express rules or constraints that must be applied to the data presented to the service. For example,

• a retail customer may not be allowed to use purchase orders, while commercial customers in good standing can,
• there may be several related fields that must be validated together, such as street address, city, state, and zip code, so all must be present,
• a shipper can't be assigned to an order until a ship-to address has been specified,
• are postal codes a string of digits, or letters and digits? How many? It depends on the country.
• two addresses, current and future, are required in a move order for relocating broadband cable service.

A service always has to validate its inputs, but in loosely coupled systems, it will often be necessary to do data validation in consumers as well, especially in service consumers that interact with users who want early feedback in data entry, for example, before an order is submitted.

The conceptual model can be used to capture such constraints so they can be applied everywhere they are needed in a uniform manner. When the validation logic is generated from these rules you can be sure they are the same everywhere.

Calculations

Often a data element used in one service may have a different data format than in another, and the values expressed in different units (inches versus millimeters, Fahrenheit versus Celsius, pounds versus kilograms, and so on). One service may define a value with 3 decimal places (e.g. 3.602) and another as a whole number (e.g. 3). Should the decimal value be rounded or truncated when converting?.

The conceptual data modeling tools should be able to define such commonly used conversions as well as application specific conversion or calculations so that they can be defined once and used where needed. This will make the conversions easier to change and improve overall system reliability.

Industry-standard Data Models

Standard data models have been defined by many different industry-specific groups, such as insurance (ACORD), healthcare (HL7), and telecommunications (SID). Most of these data models are large and complex, with 1,000 or more classes. If you can use one of these data models, it can both save you considerable effort and ease conformance with your industry’s standards.

To use these standards, you will have to add them to your conceptual data model. Since they are usually large and complex, the modeling tool has to have the means for importing them. Once imported, you will also want to be able to add layers on top of the industry model without changing it. For example, since these data models are usually abstract, attribute names tend to be generic, such as a set of “ContactMediums” instead of “main telephone number” and home telephone number”. You will want to add mappings to the names used by your own data and applications.

The Metadata Repository

The metadata (the mappings, transformation rules, data format conversions, validation rules and all the other things we;ve discussed already) have to be stored somewhere where they can be easily used and where the tools can get and save their artifacts. This is the role of the metadata repository. The repository is not just a big shoe-box you can throw all the metadata and related artifacts into. It should give you a way to organize and keep track of everything.

When you use repository-based modeling tools, in addition to easier implementation, ongoing maintenance costs will be lower as well. You can achieve the following benefits:

• Automated impact analysis -- As you plan changes to your SOA, the tools can provide reports about the interdependencies between services and consumers and the impact of any proposed changes.
  
• Reusable components -- When all the data definitions, transformations, validation rules, conversion formals, and so on, as well as the generated code are stored in the repository, you’ll be able to find and reuse elements that have already been developed.
  
• Correctness -- By using the elements defined in the repository where they are needed, overall system and application correctness, consistency and reliability will be enhanced. You have the assurance that a change can be propagated to all the places it needs to go. Your SOA will have fewer bugs.
  

Conclusion

Take care of the needs of your service consumers to be sure you can achieve the SOA benefits of simpler and faster application and data integration, agility and ability to respond to the needs of your business more effectively. Modelling tools, a common conceptual data model, and a metadata repository will promote reuse, reduce errors, and control the effects of change. Then you can bask in the glorious sunshine of your SOA.

References

"Semantic Interoperability of Web Services – Challenges and Experiences" lsdis.cs.uga.edu/library/download/techRep2-15-06.pdf

Jim Gabriel, "Best Practices in Integrating Data Models for SOA," Web Services Journal, Vol. 5, Issue 2, February 2005 (http://www.sys-con.com/story/?storyid=48031)

"Scaling SOA Through EAI Enhancement and Use of Model-based Standards" - a whitepaper from Progress Software

XSL Transformations (XSLT) Version 1.0 World-Wide Web Consortium Recommendation http://www.w3.org/TR/xslt

Stylus Studio XSLT Editor http://www.stylusstudio.com/xslt_editor.html