JBoss SOA Governance: Project Overlord

SOA Governance on ebizq

2010-04-29T14:44:00.000-07:00

It's been a while since we posted something, but this cross post should be interesting.

Using Process Governance in support of Process Versioning

2009-06-12T03:22:00.000-07:00

Seeing the recent blog by Van Alst reminded me how low tech process versioning is. As he points out, it requires careful implementation of the processes, to avoid having long running process instances that may cause difficulties when trying to upgrade either the business process, or the infrastructure on which that process is running.

His suggested approach is to implement those BPEL processes, that would normally be long running, as event/message driven processes that can optionally utilise short running BPEL processes. However the disadvantage is the complexity that may be introduced to accommodate the potentially large number of business process versions that need to be supported. Each new version of the business process needs to accommodate existing running process instances that may be at various stages in the event driven implementation. However suitable tooling could be provided to help manage this.

The typical process versioning techniques currently used are:

1) Active processes should continue to use the same process definition (e.g. BPEL) version, while new process instances would use the most recent version.

2) Terminate all currently active processes – this seems a bit extreme, and can only really work if all systems involved have a way to undo or back out the work already done, and then replay the transaction on the new business process.

3) Migrating the process – although some simple changes can be handled using this approach, it can be very difficult to achieve without introducing errors. This is because the control structure of a process may change, or new variables may be added that cannot be initialised 'after the fact'.

So options (2) and (3) may be suitable in rare cases, but they are not a general solution.

The problem with option (1) is that a business process may be changed while a business process instance (or transaction) has not 'touched' all of the BPEL process definitions involved. For simplicity, if we assume a business process is comprised of two BPEL process definitions, A and B – if a business process instance X starts with process A, and waits for some duration (i.e. may be awaiting user input), and in the meantime the business process is updated in such a way that the interface between process definitions A and B are changed. When process instance X restarts in process definition A, and attempts to interact with process definition B, it would fail. This is because process definition B has not previously been involved in business transaction X, and therefore the BPEL engine managing process definition B would not be aware that it needs to actually direct the interaction to an older version of process definition B.

Could process governance provide another solution to this problem?

Understanding what behaviour a service provides, and what behaviour a service requires from other services, may provide the basis for ensuring the appropriate versions of each service work together correctly to deliver a particular version of the overall business process. This means that when one service wants to interact with another service, it should invoke the process definition version that meets its behavioural requirements, and not necessarily the one that is the most recent version.

The other way in which behaviour could be of use is to help determine whether a new version of a particular process definition (i.e. BPEL) is backward compatible with an active process instance associated with an older version of the process definition. If this is the case, then it may be easy to migrate the process instance to a newer process definition, and thus enable the business process instance as a whole to evolve – however this would be subject to the backward compatibility constraint.

So in summary:

First approach is to make each component of the process as simple and short lived as possible, to avoid having to worry about issues related to versioning those components. The system runs using the most up-to-date version of each component, and the complexity in dealing with changes in a business process are managed by the glue between the tasks – the rules that govern how the results of one task are used to trigger one or more other tasks.

The second approach doesn't place restrictions on the long running nature of the individual services, but instead provides a way in which multiple versions of a business process can run simultaneously, and evolve within certain constraints to newer versions of the service components.

The main point is that a business process and the resulting service designs should not be defined with versioning in mind. As far as possible, this needs to be handled as part of a SOI. In the case of the second approach, this can be handled as part of the governance capabilities, by controlling the specific versions of services that are used based on the required behaviour.

Choreography comes of age

2009-05-19T05:45:00.000-07:00

A choreography, as defined in the WS-CDL spec, "describes peer-to-peer collaborations of parties by defining, from a global viewpoint, their common and complementary observable behavior; where ordered message exchanges result in accomplishing a common business goal."

WS-CDL was ground breaking as a standard for two reasons:

1) Normally a standardisation process occurs when a number of proprietary de-facto standards, supported by different groups within industry, need to be merged to provide a single true standard that all parties are happy with.

No such proprietary standard existed for choreography, and therefore the working group were in the rare position of having to create something from scratch.

2) For introducing the concept of a 'global' model - a global model being a description of the interactions between parties from a neutral or global (i.e. service independent) perspective.

Much respect to Nickolas Kavantzas of ORACLE for dreaming up the idea, as this has spawned a lot of new research, and therefore if WS-CDL has achieved nothing else, it has made a significant contribution to computer science. Pity ORACLE didn't follow up on Nick's excellent contribution by providing an implementation of the standard, especially as they were also the co-chair of the working group.

However WS-CDL has its downsides.

In an attempt to provide a language with a strong formal type system, it has become complex to use. It is perceived by many to be web service specific, due to its affiliation with W3C. However the most damaging aspect is the long running misconception that it is competitive to BPEL.

As mentioned above, WS-CDL is an example of a 'global' model, and therefore provides a service independent view of interactions between participants.

Abstract BPEL is an example of a 'local' model, i.e. it is a service specific view of interactions between that service and its directly associated participants. Therefore the standards are actually complementary, they should be able to work together harmoniously in their separate roles.

The problem is that one of the main disadvantages with WS-CDL is that it does not define its own 'local' model representation that would be fully compatible with the 'global' model language. Although Abstract BPEL can be used to fulfill this role, it is not ideal for the job.

Now choreography has come of age. We have a new champion emerging on the scene in the form of BPMN2.

The popular graphical notation has taken a number of significant steps forward with version 2, including the introduction of a choreography 'global' model to accompany the existing process 'local' model notation.

The benefit of this approach is that it is (1) not perceived to be web service specific, (2) there is a compatible notation between the 'global' and 'local' model perspectives, and (3) it is being incorporated into a standard that is already very popular, and therefore should make it easier for users to adopt the choreography concept.

BPM console blog moved

2009-05-18T05:46:00.000-07:00

I moved the BPM console blog to a new location:
http://relative-order.blogspot.com/

Testable architectures

2009-02-15T16:30:00.000-08:00

Steve has posted an interesting article on his recent use of the testable architectures approach he's been discussing for several years. It's obviously closely related to what we're trying to achieve together on Process Governance, so take a look.

Developing BPM console plugins

2009-01-15T12:24:00.000-08:00

I committed a first draft of a plugin mechanism that allows composition of BPM console features more easily. Initially we'd been looking for a convenient way to deal with the differences in jbpm3 and jbpm4, but now it has proven to be good solution for console mash-ups in general. For instance future integration or composition with Guvnor.

The idea is simple: While providing common elements like authentication, configuration and look and feel, the console needs to be assembled out of common and proprietary features. Not everything we build into it, works across BPM runtimes even though it's our foremost goal.

Fortunately GWT already ships with an extension mechanism called "Deferred Binding". It allows you to replace or generate components at build time. Build time in our case means maven, and the easiest way to customize a maven build is by providing different sets of dependencies. Unlike the regular GWT compilation which is restricted to a subset of the JDK, the deferred binding has access to the full API, including classloading. It actually happens before the GWT compiler kicks in.

Great. That means our plugin loader can build upon the availability of plugins in the classloading scope.
Maven already provides the dependency management, then we are set.

Let's look at an example. Trunk contains a plugin-example that should get you going. Take a look at the pom.xml dependencies first. Any plugin has a dependency on the console plugin API:


<dependency>
     <groupId>org.jbpm.jbpm3</groupId>
     <artifactId>gwt-console-plugin</artifactId>
     <version>1.0.0-SNAPSHOT</version>
</dependency>

It ships with a few concepts that we need to clarify before we can move on. The console acts as a Workspace. A Workspace contains Editors, which are composed of different Views. Developing a plugin actually means implementing an Editor and providing a workspace configuration (workspace.cfg).

When building the console, the deferred binding generates and links the code that is required to associate all Editors in a workspace configuration with the console framework. Sounds terrible, but a plugin developer only needs to know a few interfaces and create a workspace configuration.


public abstract class Editor extends Panel
{
 protected ApplicationContext appContext;

 public abstract String getEditorId();
 public abstract String getTitle();
 public abstract String getIconCSS();
 public abstract MenuSection provideMenuSection();
 
 [...]
}

An Editor is your plugin root implementation. It needs to be uniquely identified (editorId) and contribute to the main menu (MenuSection). It can contain one or more views:


public abstract class View extends Panel
{
  protected ApplicationContext appContext;

  public abstract String getViewId();
  public abstract String getIconCSS();
}

The editor and it's views share an ApplicationContext that gives you access to common things like authentication, configuration, logging, etc.


public interface ApplicationContext
{
  [...]
  void displayMessage(String message, boolean isError);
  Authentication getAuthentication();
}

Besides that, you can leverage anything GWT provides for implementing your plugin.
I think you get the idea. Take a look at the plugin example (Google Maps integration), especially the editor implementation.
In order to enable this workspace configuration you need to build with "mvn -Pplugin-example" switch.

Discussion
Please join us in the overlord forum for further discussions.

Typing a BPM system

2009-01-13T05:12:00.001-08:00

While working on the GWT console I came across two requirements that I couldn’t realize easily within the console: process variables inspection and task management forms. Both solutions should allow modification of data associated with a process through web based forms.

In order to build such a form and to process the data submitted through it you’d need some kind of data type description associated with the process definition. Even if you don’t intend to build those forms at runtime, but instead chose a design time approach you would still be left with the question of validation.

But not only that. The tooling that helps you build the task forms at design time would need some data type description as well. How would know what types to chose for UI elements and the keys to access it?

This leads to an additional requirement: Once you got the data type description and you successfully modeled your form, how do access the data at runtime? Somehow the process data needs to extracted and displayed through the forms.

One thing you may realize at this point is that a simple example like the task management forms already require type information across BPM components. It start’s with the tooling, is used to extract data at runtime and finally describes how the forms should be rendered.

One thing that jumps to my mind XML and XSD. Ok, I admit that I have been working on the JBoss web service stack, but other problem domains chose such a solution as well. Just think of heterogenous (i.e. ESB) or event based systems (i.e. CEP). And they do so for good reasons:

You get a data type model that supports structured and simple types
It can be validated through XSD
Access to the data can either be done programatically or through XPath
Data can be represented in Java or XML

If you add XForms to the picture we are one step closer to solving my initial task management problem: Describe your data types through XSD, extract it by using XPath and display it through XForms.

But there’s more to it: Decoupled producer and consumers in SOA, in which BPM plays in important role, would greatly benefit from such a type system as well. Any invocation between a process and a service could then be constrained to the type information at hand.

GWT console Beta2

2009-01-12T10:01:00.000-08:00

Some things changed in Beta2 others did improve and I’d like to give you a quick update on what happened the last two month.

TokenEditor

Something you may already know from the JSF console and what is commonly used during development is the possibility to signal tokens directly from the console. We’ve added that feature to the GWT console under the section “Development tools”. You’ll notice those by the little wrench icon in the upper corner. These tools will be available to a particular role only, once we add authorization to the editors.

Report server integration

Beta2 drops the gchart, the client side Javascript charting library and instead switched to BIRT. There have been several reasons leading to this decision.
On the one hand we’ve realized that biggest challenge with reporting is not rendering itself (which gchart could have provided) but the tooling that needs to go along with it.

We’ve realized that it’s close to impossible to come up with the one and only BPM report that pleases everyone. Most reporting and dashboard applications are actually quiet the opposite: Highly customized towards particular users and their problem domain. This is the approach we are taking with BIRT as well. The jBPM console ships with stock reports, that can be extended or even be replaced.

If you are interested in jBPM reporting a good starting point would be the report server project introduction.

Task management

We’ve also added the first task management functionality which currently does focus on task assignments.

The current release allows you to claim tasks, complete them by signaling or reassign tasks to a particular user or group. Future console version will embed task forms for human interaction, but this area in particular requires further discussion.

Stay tuned...

Guerilla Governance?

2008-11-28T07:18:00.000-08:00

An old friend of mine started talking about Guerilla SOA a few years back and got into some interesting debates around it. Ignoring the fact that some of G-SOA is flack about the need (or not) for vendor specific SOIs (always a good thing to push if you're working for a consultancy firm), one of the important aspects that I take from this is the incremental approach to SOA. Whatever you call it, the concept of taking things one step at a time when you approach SOA rather than trying to do a big bang approach is fairly obvious and intuitive. We're used to working in that way for many different things: you don't have to know all of Java in order to write HelloWorld; you don't have to understand how your car engine works in order to drive it etc. etc.

This incremental approach to SOA is something we've been pushing since the very early days of JBossESB and it's embodied in the latest release of our SOA Platform. The feedback we've been getting is overwhelmingly positive, so we've also been doing this within Overlord. I'm not sure if we'll call this Guerilla Governance (G-Governance?) but the principles are the same: start small and build up as you gain experience and as your needs evolve. A flexible SOA Governance infrastructure should adapt to you while at the same time encouraging good governance practices so that you can adapt to it. This is why we've started with things like DNA/Guvnor and Process Governance, as they satisfy well defined needs on an independent basis but also within the whole Overlord infrastructure as and when we get there. SAMM is a little more invasive as far as governance is concerned, as is the forthcoming work on Policy Enforcement Points and SLAs. But more on them in the future.

So if you're looking to deploy your favourite SOI and are concerned about governance, don't think you necessarily have to buy in a complete solution immediately. (I'm not even sure there is such a thing as a complete solution!) Look for something that can grow with you and even better, something that you can influence with your own policies and approaches to governance.

And if you're using JBossESB or our SOA Platform, take a look at the Process Governance work. As soon as you start to tie together more than a couple of services, something like this (helping you to write and enforce provable correct choreographies between arbitrary services and across arbitrary complexities) is essential.

Process Governance 1.0MR1 released

2008-11-25T04:47:00.000-08:00

The first milestone release of our Process Governance work has just been released. Please take the time to download it and try it out. This is a major contribution to the whole governance area and we hope to get your feedback. Congratulations to everyone involved.

Process Governance

2008-11-09T14:36:00.000-08:00

I've mentioned a few times before how important I think the WS-CDL effort has been and how overlooked it's been. But we started to put that right about a year ago when we looked to integrate the great tooling that the Hattrick guys have been working on with JBossESB. Well we're almost ready for the first milestone release and I've been going through what the team have been working on. It's very cool and I'm impressed. We'll be making an announcement soon, but this will really add an important differentiator to Overlord that we're calling Process Governance. I'll blog more on that later, but watch this space!

How does SAM relate to the world?

2008-09-01T01:44:00.000-07:00

We've discussed briefly what we're doing with SAM and it's relationship to BAM/BI, but it's worth going over some of this in a bit more detail.

For instance, how does SAM related to Business Service Management (BSM)? Well according to the common definitions of BSM, it is aimed at helping users (e.g., administrators, project managers) inform their management software which services, tasks etc. are the most important for the business. BSM then enables them to correlate the performance and availability of those systems with their business goals, identifying when an application, service etc. is not behaving as expected. Through some magic, the BSM identifies the cause(s) of the breach in contract and how to fix it (them).

So is SAM a solution to BSM? Not quite. But it should be easy to see how SAM can be a critical component in the development of BSM.

What about infrastructure monitoring? Well we did cover this in the very first posting about SAM. In essence the answer is the same as for BSM: SAM should be at the core of all infrastructure monitoring, receiving and collating information (data streams) from components, routers, processors, etc. Are these data streams real-time? Well let's ignore the fundamental problems with the limitation of the speed of light and simultanaeity. (Everyone should be made to read Lamport's classic paper on the subject as it relates to computing.) Let's also ignore the differences between hard real-time and soft real-time. The answer is yes AND no: of course most of the information streaming in to the SAM implementation will be coming "as it happens", but we're allowed to provide archival data that may have been taken hours, days or weeks previously. In fact, in order to support the right kind of correlation, archival data, whether provided by the SAM user or taken by SAM itself (since it is based on CEP principles), is critically important. This is also where the Bayesian Inference Network aspect to SAM comes in too. No longer will strict binary triggers be sufficient for the kinds of networks we see today and in the future.

So what constitutes the data of an event? In fact an event message contains multi-dimensional data that we need in order to correlate and visualize it. Of course there has to be a way of distinguishing "when" something (event) happened. That could be explicitly mentioned in the data stream or implied by the local time at which the message is received. Then we need to figure out "what" is being mentioned, i.e., what data is being analyzed. Which brings us to the actual data. Of course one message may contain multiple readings, e.g., the temperature at a sensor as recorded over 5 different intervals. Given all of this information (which is common to most other monitoring techniques), we can start to build up a map of what is going on in the system and trigger on any desired event, even if triggering requires correlating across a multitude of input streams. (In order to make the architecture symmetrical, we'll actually consider time as an input stream to SAM as well.)

But what about visualization? Well we've already seen the start of what we want to do with SAM. The BPM console is just one graphical view on to the data that we are accumulating and correlating. But there will be the equivalent of a BAM console: the SAM console. How you would view this information to obtain the best and most intuitive representation is an ongoing effort in its own right. For example, customizable home pages, displaying the most important graphs, data etc. per user. The notion of mimic diagrams will also be interesting to explore. In all likelihood, because of the inherent flexibility of the SAM infrastructure it'll be impossible to cater for all of the different ways in which the information may be displayed, so there'll need to be a combination of common out-of-the-box views as well as a toolkit of components that allow for the easy construction of other views.

In a later blog posting we'll look at how SAM can be important in the support of cloud computing.

First glimpse at the new BPM console

2008-08-21T03:11:00.000-07:00

The new BPM console approaches it's first milestone and I'd like to use that opportunity to introduce you to the most important changes and new features.

Managing Process Definitions

Moving to GWT
More and more JBoss projects are moving to GWT. And they do it for a good reason:

If you are familiar with Java development and don't want to become the next web developer expert, then GWT is a good choice. With GWT you can stick to eclipse, fire up a debugger and write unit tests.
It has a rich set of UI widgets that you can use out of the box. The widgets also force a common look and feel across implementations.

Another good example of a successful GWT implementation is the Drools console.

Managing process instances

GWT is already quiet modular and it will allow for integration of consoles across projects. Another side effect is, that you can easily take a GWT application, or pieces of it, and hook it up with existing web applications. For instance this would allow users to embed the task management functionality of the BPM console with their own intranet.

Process instance details

Improve on BAM and BI functionality
Probably the biggest drawback of the current console is lack of BAM and BI features.

Workload overview

Improving on BAM and BI is not going to happen within a day, but you could expect to see the first metrics and stats in early releases and we try to add more bits and pieces while we move towards a full fledged BAM console. Because this going to overlap in both functionality and technology with the Service Activity Monitoring project, interested readers should have an eye on SAM as well.

Performance metrics

How to move forward
To begin with, we are going to provide a replacement for the existing jBPM console based on GWT. It will retain the current features and provide additional BI functionality. Initially we are going to leverage the existing jBPM3 backend and then gradually enrich it with SAM components or even replace it at all.

Process Graph View

Stay tuned. Next time we dive into implementation details: gchart, gwt and gwt-ext.

We know the answer, but what was the question?

2008-08-08T12:28:00.000-07:00

Peter: "SAM means service activity monitoring".
Paul: "So, you are going to monitor service activities"
Peter: "Right. That's what I said."
Paul: "But it's two different things, isn't it?"

A typical hallway chatter. You just want to grab some coffee and then you bump into somebody, who completely turns your world upside down. It's a small question, but in this case an important one.

The last week I spend thinking about SAM infrastructure, basically getting an idea on how SAM might fit into different application/service landscapes. So you have SAM "monitoring" what your services do, and you put instrumentation code pushing "events" to SAM. Anything that might potentially be interesting to SAM, but it should figure our itself what's considered relevant. Hence the CEP backup behind it, right?

Fortunately I got distracted by some console work (moving the jBPM console to GWT), which made me think more thoroughly about what information will be relevant to people using the BPM console. There is process definitions, from which you create process instances, which again consists of work items (aka nodes). You can't tell upfront what path of execution will taken, neither you know what particular node types do.
At least from a monitoring perspective, you are forced to keep a more general look onto things. Coming from the technical side, this quiet naturally leads to a static view onto things: Process start and end dates, overall number of executions, average execution time. These sort of things.

But is it really what console users are interested in?
If I put on my "chief of sales operations EMEA" hat, I would probably like to see key performance factors relevant to my business. This might be anything from missed opportunities to SLA breaks of certain partners. But what are key performance factors?
Especially without knowing the business domain beforehand, how should I (the console developer) enable "chief of sales" to get the information relevant to him?

Let's look at the catchy title again: If I would have known upfront what you are going to ask me, I could have prepared myself. But I don't, and I never will.
I am just "Service Activity Monitoring" and you are monitoring service activities.

Now that we know "chief of sales" actually exists, we can start wondering about what actually enables him (or her) to do a successful job. IT is just the tool set here. If we do that, we will probably see that any business is goal driven (did we achieve it or not?), it has a number of participants (partners, systems, colleagues)
and suffers from derivations (exceptions, compensation). To "chief of sales" the underlying IT infrastructure is irrelevant. It exists and enables business, that's it.

Back to SAM.
To SAM the infrastructure exists but is irrelevant, too. Things that happen inside or outside your company can be relevant to the business. SAM just makes it visible to you and raises your attention.

Paul: "Great, sounds like a match."
Peter: "Yes, it is"
Paul: "But what's with the BPM console then?"

Well, the work on the BPM console made me realize that "chief of sales" questions are not restricted to BPM. It covers anything that enables particular business aspects. IT is just an implementation detail here. And so is BPM. Or ESB, or web services. But still all of these SOA components will provide information relevant to the business. And so does the BPM engine.

For the BPM console it means, that it doesn't need to be tight to the BPM domain. I believe that to a large degree general service information (goals, participants,derivations) which SAM can deliver would be sufficient. However, we may still add pluggable console components that are specific to BPM or even proprietary to jBPM. But to begin with, we should focus on a general monitoring concept.

Paul: "OK, back to work."
Peter: "Yep, talk to you later."

CDL and precision

2008-07-18T04:56:00.001-07:00

There's an interesting discussion on InfoQ related to WS-CDL and it obviously has direct relevance to the work we've been doing in the community for over a year now.

Design for correctness

2008-06-26T07:37:00.000-07:00

We started integrating WS-CDL into our design and runtime processes a while back. This work became one of the defining (and differentiating) factors behind our governance efforts (and therefore Overlord). Some people (users and analysts) just "get it" and understand the need behind CDL (let's drop the WS component to the name, because CDL is not limited to SOAP/HTTP by any means). However, others don't and still others ignore it entirely. Best case, this is a shame. Worst case, this is compromising integrity of the systems they develop.

Steve Ross-Talbot recently gave a presentation on CDL recently at the Cognizant Community Europe workshop, and used the analogy of a house architect to explain where CDL fits in. This is a good analogy, because CDL should be in any good Enterprise Architect's repertoire. Just as you don't throw together a straw-built house from a pencil drawing on the back of a napkin and expect it to withstand a hurricane, neither should you just cobble together components or services into a distributed system (irrespective of the scale) and expect it to be correct (and provably correct at that). In the housing example you would pull an architect into the solution and that architect would use best practices that have been developed over centuries of collective experience, to design a building that can withstand 100 mph winds. Software engineering should be no different. Some sectors of our industry have been able to get by with computing as an art rather than a science, and house designers did pretty much the same thing thousands of years ago. But we don't live in caves any more for good reasons (although there's still something to be said for using caves in a hurricane!)

Of course it means that there are more layers in between the act of deciding what needs to be done and actually realising that in an implementation, but those layers are pretty important. The days of just throwing something together and assuming it'll work as planned are well and truly over. Asynchronous systems, which really began life several decades ago but were muzzled by layers of synchronous abstractions, are back to stay. Yes, synchronous is easier to understand and reason about, but it's an unfortunate reality that if you want scale, real-time, loose coupling etc. we have to break through the synchronous barrier. That has a knock-on effect on how you design your systems and individual components (services) and ultimately how they are managed (by a person or by some autonomic mechanism). "Design for testability" was a buzz-phrase from many years ago. What we need now (and what CDL integration gives us) is "design for correctness".

Goodbye BAM/BI, hello SAM/SI

2008-06-23T04:13:00.000-07:00

OK, not quite, but it's a nice title ;-)

The term Business Activity Monitoring (BAM) is used to describe the real-time access to critical business performance metrics in order to improve the efficiency and effectiveness of business processes. Real-time process/service monitoring is a common capability supported in many distributed infrastructures. However, BAM differs in that it draws information from multiple sources to enable a broader and richer view of business activities. BAM also encompasses Business Intelligence (BI) as well as network and systems management. Plus BAM is often weighted toward the business side of the enterprise.

Although BAM was popularized by BPM, the fundamental basis behind it (monitoring the activities in an environment and informing interested parties when certain events are triggered) has been around since the early days of (distributed) system management and monitoring. BAM specializes this general notion and targets the business analyst. Using BAM within an autonomic infrastructure is often difficult if not impossible (depending upon the implementation in question).

Within a distributed environment (and many local environments) services are monitored by the infrastructure for a number of reasons, including performance and fault tolerance, e.g., detecting when services fail so that new instances can be automatically started elsewhere. Over the years distributed system implementations have typically provided different solutions to specific monitoring requirements, e.g., failure detection (or suspicion) would be implemented differently from that used to detect performance bottlenecks. For some types of event monitoring this leads to overlap and possible inefficiencies. For instance, some approaches to detecting (or suspecting) failures may also be used to detect services that are simply slow, indicating problems with the network or overloaded machine on which the service resides. But where these ad hoc approaches have differed from BAM/BI is in their intended target audience: other software components (e.g., a load balancer) rather than humans.

This separation of audience is useful from a high-level perspective: business analysts shouldn't have to be concerned about low-level infrastructural details. But in many cases this ad hoc (bolt-on) approach to BAM and BI can lead to less information being delivered to the entities that need it at the time they need it. Therefore, within the Overlord project we are working on Service Activity Monitoring (SAM) and associated Service Intelligence (SI), which will provide an architecture (and corresponding infrastructure) that brings together many different approaches to entity monitoring within distributed systems (where an entity could be a service, a machine, a network link or something else entirely) and particularly SOIs. The emergence of event processing has also seen an impact on this general entity monitoring, where some implementations treat failure, slowness to respond etc. as particular events. This uniform monitoring includes the following:

• Message throughput (the number of messages a service can process within a unit of time). This might also include the time taken to process specific types of messages (e.g., how long to do transformations).
• Service availability (whether or not the service is active).
• Service Mean Time To Failure (MTTF) and Mean Time To Recovery (MTTR).
• Information about where messages are sent.

The information is made available to the infrastructure so that it may be able to take advantage of it for improved QoS, fault tolerance etc. The streams may be pulled from existing infrastructure, such as availability probing messages that are typically used to detect machine or service failures, or may be created specifically for the SAM environment. Furthermore, streams may be dynamically generated in real-time (and perhaps persisted over time) or static, pre-defined information, where the SAM can be used to mine the data over time and based on explicit queries.

With the advent of SAM we will see BAM implementations that are built on it, narrowing the types of events of interest for the business analyst. The SAM approach offers more flexibility and power to monitoring and management over the traditional BAM approaches. As BPM and SOA move steadily towards each other, this kind of infrastructure will become more important to maintaining agility and flexibility.

WS-CDL and ESB integration

2008-05-14T08:09:00.000-07:00

At the start of the year we described some work around CDL. Well we've made a lot of progress. Check it out and provide feedback.