ESB Architecture and Lifecycle Definition
A Brief Introduction
Progress Software developed a comprehensive definition of the Progress® Sonic ESB®, including its development and deployment lifecycle at an architectural level, using industry-standard modeling techniques. The following is an overview of the ESB definition, which is fully defined in the white paper The Sonic ESB: An Architecture and Lifecycle Definition.
Service-oriented architecture (SOA) is fast becoming the systems and software approach of choice for the majority of medium and large enterprises.(1) According to the principles of SOA, IT professionals design, implement and deploy information systems from components that implement discrete business functions. These components, called "services", can be distributed across geographic and organizational boundaries, can be independently scaled and can be reconfigured into new business processes as needed. This flexibility provides a range of benefits for both the IT and business organizations.
Core to the IT infrastructure supporting SOA is the enterprise service bus (ESB) which connects and mediates all communications and interactions between services. Given the importance of the enterprise service bus (ESB) to SOA implementation - and its affects on costs, development and deployment processes - it is important to understand the functions and structure of the ESB and how its design permits rapid change in, easy connections to, and clear visibility and control of services and processes in a SOA-based application.
Figure 1 shows an example of SOA used to integrate and mediate a multi-channel financial trade process. This represents a combination of existing systems "wrapped" with a services interface and new services created to enhance the business process. This example illustrates how an equity trade order would originate from an Order Management Service, undergo a transformation of data elements to support the requirements of the trading system and the Compliance Service, flow through to trigger a fund transfer with the Funds Transfer Service, and ultimately get logged as a completed transaction in the Logging Service. Rather than an integrated application, in this case the trade process is executed as a sequence of heterogeneous process steps, coordinated by the SOA infrastructure.
Figure 1: Example of SOA Application
In this example the ESB is providing the connectivity with routing between services, the transformation of XML data to allow services with varying interfaces to communicate, and the business process sequencing of services.
Progress Software introduced the industry's first ESB and has continued to be the market leader in ESB functionality. As such, the Progress® Sonic® ESB (hereafter referred to simply as "ESB") sets a reference mark for the SOA marketplace. The ESB is complemented by additional products in the SOA infrastructure portfolio including Sonic ESB Workbench, Sonic BPEL Server, Apama ESP, and DataXtend Semantic Integration (SI).
The approach being taken in this full ESB definition effort is to describe each of the key ESB functional areas through exposition of its function and benefits accompanied by structure diagrams based on an industry-standard approach, UML (Unified Modeling Language). UML Class diagrams provide a definitive architectural definition for the ESB that is useful for learning and discussion. Figure 1 employs an ESB diagramming notation introduced by David Chappell in his book Enterprise Service Bus.
Services, Communication and Mediation
Core to the support of a service-oriented architecture (SOA) is the support of services and their communications. Figure 2 shows a UML Class diagram with an overview of the ESB structures supporting services, communications and mediation.
Figure 2: Overview Architecture for Services, Communication and Mediation
ESB Services (left of Figure 2)
ESB services may be any of a number of types. Typical ESB services which implement mediation are: XML Transformation, Content-Based Routing and User Defined Mediation. An XML Transformation Service provides data conversion from one XML document to another according to an XSLT specification. A Content-Based Routing Service directs messages to various other service endpoints according to configured routing rules. A User-Defined Mediation Service implements a function such as logging or custom validation which complements the core business logic. A User-Defined ESB Service connects to business services (see below) to implement core business logic.
ESB services execute within the ESB Container. The ESB container instantiates ESB services and provides the systems management interface for its services, the management of communication protocols, the capability to have multiple threads executing services for scalability, and the standard capability to execute distributed processes.
Connected Business Services (right of Figure 2)
Connected services implement the business logic for ESB-integrated applications. Types of business services include: Web Services, Legacy Systems, JMS (Java Message Service) Applications and all varieties of Application Server Applications. Each of these types of business services can communicate with each other through the ESB, and with ESB services to perform mediation. In Figure 1 the Order Management Service is a connected service which connects in through an HTTP Connection. The Compliance Service, Trading System and Funds Transfer Service are business services that have been interfaced directly to the ESB as user-defined ESB services.
Communications and ESB Processes (center of Figure 2)
This section of Figure 2 shows the core capability for services (both ESB and business) to communicate and to execute an ESB Process which is a sequence of steps invoking services or other ESB processes. Figure 1 shows an ESB process called Trading Process that has five steps.
An ESB service sends and receives messages through references to ESB endpoints and ESB addresses. For example, an ESB service such as the compliance service in Figure 1 can configure its Exit to refer to the ESB address of another service to receive output messages - in this example that would be the Trading Service. Alternatively, the ESB address can refer to an ESB process.
An ESB process is defined by an ESB Itinerary which contains a sequence of Process Steps. Each process step can invoke one (or multiple) ESB services, other ESB processes, or an external Web service. The ESB itinerary is carried with its associated ESB message providing a completely distributed, and hence reliable, execution environment for the ESB process. There is no central processing engine for the itinerary as each ESB container can process and route the next step in the ESB itinerary. This provides both efficient execution without the need to communicate to a central site and reliability as processing can continue even with some losses of connectivity.
Each ESB service can have its entry and exit endpoints changed without requiring changes to their implementations.
The ESB Endpoint provides the capability for ESB services to send and receive messages with a specified Quality-of-Service, for example "best-effort", "reliable" or "guaranteed, once-and-only-once". The connection tells what supporting messaging channel is used by an ESB endpoint to connect with an ESB service or a business service. This messaging channel can use an ESB native connection, or for external systems can use Web Service Connection, JMS Connection or JCA (J2EE Connector Architecture) Adapter.
The ESB Connection employs user configurable asynchronous and synchronous delivery semantics including "publish-and-subscribe", "send-and forget" and "request-reply". In the example in Figure 1 the ESB connection between the Order Management Service and the Trading Process is Web Services request-reply. If a business or ESB service needs to change the manner in which it connects, this can be affected without requiring changes to its implementation. When services require differing types of connections, the ESB provides the mediation between the differing protocols.
The ESB Message class and the sends and receives associations between ESB endpoint and ESB message create the basic structure for one ESB service to send a message to another ESB service through an ESB service's Dispatcher. The dispatcher manages the sending and receiving of ESB messages between ESB services and external endpoints, in effect implementing the arrows between process steps shown in Figure 1. This dispatcher also manages ESB processes.
Industry-standard WSDL (Web Services Definition Language) Definitions can be created for ESB services and ESB processes. This allows the use of standard tools and integration with other SOA infrastructure components such as a UDDI Registry. For example, in Figure 1 the entire Trading Process ESB process is exposed as a Web service.
Key benefits of this approach to implementing ESB services, mediation and connection to business services include:
- Business agility through rich support for mediation services including XML transformation and built-in support for distributed ESB processes.
- Broad reach across many kinds of business services and communications protocols; integration of heterogeneous legacy and new Web service elements.
- Distributed process execution, performance and reliability even in highly-distributed deployments.
ESB Development, Deployment and Production Management
Intrinsic to the ESB is rich support for the entire services life-cycle from development to deployment to management. Figure 3 shows a UML Class diagram with an overview of some of the key structures for this support. The figure is divided to reflect the migration of ESB services from development systems to staging/test systems to a distributed production environment. A critical requirement for the Sonic SOA Suite is to allow such a migration to occur in the easiest error-free manner possible.
Figure 3: Overview Architecture for ESB Development, Deployment and Production Management
Development Systems (left of Figure 3)
The top of this diagram section shows Development Tools while the bottom, labeled "Development ESB" defines the core components of the ESB infrastructure which are used in each stage of the ESB Lifecycle from development to staging/test to production environments.
The Sonic Workbench provides each developer with a set of tools for ESB development and deployment including creation of custom ESB Service Type implementations. For example, in Figure 1, the Logging Service would be a custom ESB service type. With a set of Resource Editors, in concert with third party and industry standard development environments, developers create Resource Files which are stored in the ESB Repository (a configuration repository). The Sonic Workbench provides for the creation of Development Projects containing all of the configuration files for a specific service development project in support of a composite or ESB application. In the trading example this would include files describing the ESB process and it's WSDL file, used to expose the process as a Web service. It would also include the Logging Service implementation, among other resource files.
Editors include: the ESB Process Editor for visually creating and editing business process definitions tying together multiple ESB services, the XLST Editor for creating the XLST Stylesheet defining an XML transformation and the Routing Rules Editor to create the configuration file used by content-based routing services.
The Management Console provides the management framework for configuring the Sonic SOA infrastructure for ESB services, ESB containers and the supporting SonicMQ multi-protocol communication server elements. The Management Console allows for the configuration of new instances of ESB services and ESB containers to allow an easy "configure-and-go" approach to managing the SOA infrastructure. Using this console, for example, the ESB Connection between all services in the Figure 1 trade process would be set to "once-and-only-once" to prevent the chance of any message losses.
Staging/Test Systems and Deployment Tools (center of Figure 3)
Moving configurations from Development to Staging/Test to Production environments is done with the Deployment Tool. Selected resource files and configurations are moved through the creation of intermediate archive files and the use of the Deployment Map File to map service and messaging parameters to operate properly in the targeted environment where quality of service settings, security policies and ESB addresses may differ. The diagram shows the promotion and deployment of configurations from the development directory service to the staging/test directory service.
Production Systems (right of Figure 3)
The deployment tools are also used to promote and deploy ESB configurations from the staging/test directory service to the Production ESB Repository.
The management console is used to place ESB containers and their configured services onto multiple distributed systems where interaction policies, transport bindings and communication layer security policies are established.
The ESB provides highly reliable support for distributed systems. As shown in the diagram section labeled "ESB Distributed", an ESB Repository Cache is used to provide local copies of the configuration files used by ESB containers and ESB services executing on distributed systems. This provides the ability for distributed systems to be both high-performance and reliable, and allows continued independent operation even when the production directory service is unavailable or unreachable. This same execution environment is available, but not illustrated, in both the staging/test and development environments.
Having multiple distributed systems in an ESB provides scaling of ESB service capacity across machines and locations. Each distributed system shares the same configuration for reporting faults and sending status back to the management console. In the financial trade process example, it would be possible to replicate the transform service on multiple execution machines such that if it were needed between each step in the process (a likely situation) that there would be multiple instances running across a network of machines that could be recruited to address peak.
The administrator uses ESB services and ESB processes to automate the handling of these errors and notification events in a service-oriented fashion, leveraging the same ESB infrastructure. For example, if errors occurred in the Compliance Service step of the trade process depicted in Figure 1, the normal fault endpoint of that service could be configured to route the error back to the requestor using a reply-to semantic or a separate exception process could be defined to receive the fault on it's entry endpoint.
Key benefits of this approach to implementing ESB development, deployment and management include:
- Rich set of development and management tools allow for easy "configure-and-go" deployment.
- Deployment tools provide support for the ESB lifecycle and allow for the automated mapping of configurations and movement of implementation resources from development to staging/test to production environments.
- Distributed execution environment including directory caching allows for high-performance, scalable and continuously-available SOA systems.
For a more thorough definition of the Sonic ESB, including a complete UML class diagram, download the white paper>The Sonic ESB: An Architecture and Lifecycle Definition.
(1) Heffner, Randy, "Delivering Real World SOA", Forrester Research, 2005.