A large variety of commercial and open source event processing software is available to architects and developers who are building event processing applications. These are sometimes called event processing platforms, complex-event processing (CEP) systems, event stream processing (ESP) systems, or distributed stream computing platforms (DSCPs).
Some examples are:
- Apache Samza
- Apache Spark
- Apache Storm
- Codehaus/EsperTech’s Esper, Nesper
- DataTorrent RTS (Real-time Streaming)
- FeedZai Pulse
- FujitsuInterstage Big Data Complex Event Processing Server
- Hitachi uCosminexus Stream Data Platform
- IBM InfoSphere Streams
- IBM Operational Decision Manager (ODM)
- Informatica RulePoint
- LG CNS’ EventPro
- Microsoft StreamInsight
- OneMarketData OneTick CEP
- Oracle Event Processor
- RedHat Drools Fusion/JBoss Enterprise BRMS
- SAP Event Stream Processor
- SAS DataFlux
- ScaleOut Software
- SQLStream s-Server
- Software AG Apama Event Processing Platform
- Tibco BusinessEvents
- Tibco StreamBase
- Vitria Technology Operational Intelligence Analytic Server
- WS02 CEP Server
These subsystems offer a wide range of different features, but all perform CEP in a technical sense. CEP (or ESP) is a technique in which incoming data about what is happening (event data) is processed more or less as it arrives to generate higher-level, more-useful, summary information (complex events). Event processing platforms have built-in capabilities for filtering incoming data, storing windows of event data, computing aggregates and detecting patterns. In formal terminology, CEP software is any computer program that can generate, read, discard or perform calculations on complex events. A complex event is an abstraction of one or more base (input) events. Complex events may signify threats or opportunities that require a response from the business. One complex event may be the result of calculations performed on a few or on millions of base events from one or more event sources.
Event processing platforms are a varied lot:
- Established, widely-used commercial products, such as IBM’s ODM and InfoSphere Streams, SAP’s ESP (formerly Sybase Aleri), Software AG’s Apama, TIBCO’s Business Events and StreamBase, and others, are multifaceted development and runtime software suites that include adapters to integrate with event sources, development and testing tools, dashboard and alerting tools, and administration tools.
- Other event processing platforms are combined with features, such as query, reporting, interactive analytics, alerting, or key performance indicator tools, that are specifically directed at operational intelligence applications. Examples include FeedZai Pulse, SQLStream s-Server, and Vitria Technology’s Operational Intelligence.
- Emerging DSCPs, such as Apache Samza, Spark and Storm, are general-purpose platforms without full native CEP analytic functions and associated accessories. They are highly scalable and extensible so developers can add the logic to address many kinds of stream processing applications.
- Those DSCPs, along with Esper/Nesper and RedHat’s Drools Fusion/JBoss Enterprise BRMS (both of which are CEP systems), are open source offerings.
- DataTorrent’s RTS and IBM’s InfoSphere Streams are classified as either DSCPs or CEP systems, depending on the context and the author’s point of view.
- Some event processing platforms are bundled with a rule engine. Examples include IBM’s ODM, RedHat Drools Fusion/JBoss Enterprise BRMS, and TIBCO Business Events.
CEP usage is growing rapidly because CEP, in a technical sense, is the only way to get information from event streams in real-time or near-real time. The system has to process the event data more or less as it arrives so that the appropriate action can be taken quickly. The subsystems listed in this article are general purpose development and runtime tools that are used by developers to build custom, event-processing applications without having to re-implement the core algorithms for handling event streams. However, these subsystems are only one of several ways to get CEP functionality, and in fact, most applications that implement CEP logic don’t use these subsystems.
Most CEP is obtained as part of a larger product. Companies acquire a packaged application or subscribe to a SaaS service that has embedded CEP under the covers. The company is buying a solution that happens to require event processing, and it may not realize that CEP is being used. For example, supply chain visibility products; security information and even management (SIEM) products; some kinds of fraud detection and governance, risk and compliance (GRC) products; system and network monitoring systems; business activity monitoring (BAM) tools; and many other categories of software implement some greater or lesser amount of CEP logic. In a few cases, the developers of these products or SaaS offerings have leveraged the general purpose event processing platforms listed above to reduce the amount of code they have to write. But in most cases, the developers implement a specialized subset of event processing algorithms in new code to suit their application purposes.
Some user companies have written custom applications with CEP logic rather than leveraging an off-the-shelf event processing platform. This was especially common in the 1990s and early 2000s before the subsystems listed above were widely available, and some developers still choose to write their own CEP logic for performance or cost reasons. For example, large banks and related financial services companies have built front-office systems for capital markets trading with their own embedded CEP logic.
Every company has places where it already uses applications or SaaS solutions that implement some amount of CEP under the covers. A growing number of companies need the kind of event processing subsystems listed here to support demanding, high-throughput, low-latency applications, where event processing logic is primary to the business problem. In either case, the more you know about CEP, the better will be your application architecture.