by David Luckham

Pretty much every sector of commerce is event-driven these days. Time and information are the two driving sources of competitive advantage. As a result managers of event-driven enterprises are becoming increasingly stressed as time to manufacture, time to supply, time to do anything, shrinks, while simultaneously the information tsunami rolls over them.

Correspondingly, the market for Business Activity Monitoring (BAM) tools to aid in enterprise management has exploded into what is properly described as a chaotic marketplace. Gartner lists over fifty vendors in the BAM space. These tools can be very useful, no doubt about it. So, why do I say chaotic?

Well, first of all, step back and take a look at what BAM is trying to accomplish.

There’s a global cloud of business events flowing through the IT layers of any enterprise. These events flow from all four corners of a distributed event-driven enterprise, as well as from outside partnering enterprises. And they contain a great deal of important information for management. There are many sources of events. Some of the events are related, others aren’t. Some events cause others, some happen at the same time, others happen independently, maybe in different time zones. But they happen fast – a medium sized stock brokerage may be dealing with 2000 business events per second. These events don’t fall into place in a nice linear one-after-the-other stream. They’re a mess, a cloud of events. BAM takes on the task of providing tools that help to make management sense from the information in an enterprise’s event cloud.

Over the past five years or so various classes of BAM tools have appeared. Many of these tools monitor events from the enterprise cloud in real-time. Those are the ones we’re talking about. The simplest are those that monitor metrics (often called Key Performance Indicators, KPIs) on your IT and business assets, display them on a dashboard with fancy graphics, and alert you when the metrics get critical – like “low inventory”, or “retail website overload”. Slightly more advanced tools also give you rules that trigger on alerts and let you specify proactive repairs to your business. At the other end of the scale are BPM/BAM suites that can monitor everything on your IT Infrastructure, decide how your business processes depend upon various IT assets, deliver an up to the second view of how your business processes are performing, and let you go back to remodeling the processes at any time. Quite a stovepipe, this latter category!

There is a whole spectrum of B*M tools (where “*” is any letter of the alphabet) between the dashboards and the complete BPM/BAM systems. For example, there are classes of fraud detection and conformance monitoring tools. These contain both real-time tools, and COB time tools. Also a new breed of data (or event) streams management tools called variously DSM or ESP has entered the BAM space. These take the view that the events of interest to you are presented as linear streams such as stockmarket feeds. They can compute complex metrics on streams, such as: “all stocks with high volatility and average volume surge in the past 30 minutes”. So they can supply the kind of information from stock feeds that algorithmic trading or arbitrage programs need. At high throughput rates too! They deal with a very specialized BAM problem since market feeds are a small part of the event cloud on an enterprise IT infrastructure. The general management problem is much harder. That is, making management sense out of the cloud of business events, discovering the patterns of events that are really significant. Connecting the dots in the event cloud is what BAM is really about.

Why is the BAM tools space chaotic? Well, what would we expect if it was orderly?

First of all, we might want classifications of BAM tools that help us compare one tool with another. Secondly, standards for event processing that help us integrate one BAM tool with another. Thirdly, technology checklists whereby we would know how a tool actually works as opposed to what the marketing department says it does. None of this exists yet.

A similar chaos in software applications is just what led to the EAI space. But it can be argued that chaos at the beginning of a new market is to be expected. Chaotic development is the first step towards establishing that a market for a new kind of IT tool exists. The market for BAM is now well established, especially as event driven architectures have become prevalent. So perhaps now we can hope for a little order to come out of the chaos?

Standards usually involve protracted politics, but the first attempts at a Common Event Infrastructure standard for BAM appeared in 2004. That’s a start! Classifying tools may be even tougher. How about a BAM technology checklist? Does your tool do this or that –check the list. It might help consumers understand what kind of event processing goes on inside a tool.

Here’s a start at a BAM event processing checklist. I hope you find it useful!

1. Real-time event data computation. This means that the tool can use the data contained in events to compute metrics (i.e., KPIs) in real-time and continuously update them on the graphics displays or feed the metrics to other applications. Typically DSM tools do this.

2. Single event triggering. The tool can react to single events, either predefined or defined by the user, e.g., alerts indicating critical KPIs, and then take actions also predefined or specified by the user. Actions may involve sending email messages, etc. Usually the tool will supply a capability to define event triggered reactive rules, and may combine this with capabilities under item #1.

3. Event streams processing. An event stream is a real-time, continuous, ordered (by arrival time at the tool, or by a time stamping mechanism) sequence of events. Here, the tool assumes the event input is a stream. This assumption allows optimization of some kinds of event processing that fall under item #1. It is an appropriate way to handle stockmarket feeds, for example.

However, it is impossible to control the order in which events from an enterprise cloud arrive at the tool. So the stream order cannot reflect the relationships that exist between events as a result of enterprise activities. We’re talking about relationships such as which events caused other events, which events happened independently, or their creation times – the times at which the events actually happened. If a tool turns the cloud into a stream before it starts processing events, then it can only detect simple patterns of events in the event – essentially Boolean and’s and or’s of events.

4. Complex event pattern triggering. Tools that satisfy this checklist item can detect complex patterns of many events in the global event cloud of the enterprise and then react to them in various ways. Typically, a complex pattern is made up of many events, often created at different locations in an enterprise and possibly in different time zones. Complex patterns may involve events that are causally related and other events that happened independently. A complex pattern consists not only of the events but also of the relationships between the events. For example, the pattern of activity in a set of concurrent or collaborating business processes. Other common examples can be found in patterns of events in fraudulent activity, or violations of SEC regulations or other conformance issues.

These tools have to capture information about the event cloud in order to detect complex patterns. For example, causality between events, independence and the creation time of each event. Typically, such tools encode this information in the events themselves so that each event contains its own genetic information. When a tool processes an event it can tell from the encoded genetic data how that event is related to other events in the cloud. Many enterprise IT systems already put some of this encoding into events when they are created by using various identifiers such as message Ids, transaction Ids, etc. So, encoding the cloud is not always very difficult.

5. Event pattern abstraction. Tools that satisfy item #4 may go a step further by providing rules to create new events whenever an event pattern matches. The new events can be constructed to contain some of the information contained in the events that matched the pattern. And the new events can be processed by the tool just like other events in the cloud. This is called event pattern abstraction. It is a step towards supplying higher level views e.g., executive reports of patterns of complicated business activity. Important data is included in the new event, and unnecessary details are omitted, the new event is an abstraction.

Right now, most BAM tools will only check off items #1 and #2. Even so, they’re proving to be very useful. DSM tools mainly focus on items #3 and #1. New tools are entering the marketplace that can check off the first four items. If a BAM tool can check off four or five items on this list, it is certainly doing some complex event processing.

However, a word of caution. This checklist is only a beginning at understanding what a tool does and how useful it is. The devil is always in the details, isn’t it?

Many tools use finite state machines or Java scripts to define patterns of events and reactive rules. These methods, even when supported by GUIs, are too low level. As a result they will lead to problems later on. As a tool gets used on more demanding management problems, the sets of event patterns and event pattern triggered rules will grow in size. Patterns and rules management then becomes an issue. It is not easy to check a bunch of Java scripts to uncover mutually exclusive or redundant rules (e.g., if A matches a set of events then B will match the same set of events). High level, declarative event pattern languages would be preferable.

But, on the other hand, these are early days in a fledgling industry and one mustn’t ask for too much at once!

Do I think this checklist will stand up to extensive discussion? No, it obviously needs a lot of refinement. Have I made enemies with this checklist? Yes. But someone has to start somewhere! So just consider this checklist as a start, a suggestion to get the ball rolling. We have to get some order into the BAM tools space. The main thing is to get the marketplace thinking about these issues and not just believing what the vendors tell them.

© David Luckham 2005