Information Management Report

The market size for complex event processing (CEP) systems in the financial industry has grown to roughly $115 million this year, and is expected to grow by around 30 percent every year for the next two years, according to a report released by the research and consulting firm Celent.

In the report, titled “Complex Event Processing: Looking Beyond Algorithmic Trading,” data volumes are increasing exponentially due to growing automation. Trading venues and exchanges are responding to the challenge by upgrading their IT infrastructure to handle large data volumes.

Trading firms too are pressed to expand their data processing capabilities, and therefore there is a growing need for sophisticated analytics for mining large volumes of data.

Such analytics includes complex event processing, a category of technology that allows algo traders to respond to a variety of market data, including current events and news, in order to generate trading orders, by providing high-speed responses to specific patterns or triggers in real-time data.

The financial crisis of 2008 was a driver for greatly increasing demand for this technology, as CEP is being used for continuous and more granular risk monitoring applications.  ………………

…………………..

Moving forward, the report says that complex event processing will be used increasingly for multiple applications within a single firm. As a result, enterprise architects will need to rethink their infrastructures and evolve them in ways so they can handle multiple instances of data being used by variety of different modules in real time.  Report

MIT Technology Review

A new analysis of research about the predictive power of social media challenges claims that Twitter can predict the outcomes of elections. The University of Oviedo’s Daniel Gayo-Avello contends the research on Twitter and elections is riddled with flaws.

Gayo-Avello says researchers have assumed that all tweets are trustworthy when political statements are often littered with rumors, propaganda, and humor. He takes issue with Twitter’s demographics, considering most tweeters are younger, which will bias any results.

“Social media is not a representative and unbiased sample of the voting population,” Gayo-Avello says. Self selection is a problem in that people who make political remarks are the most interested in politics, and Gayo-Avello says more work needs to be done to understand the silent majority.

Moreover, he notes that all of the papers on elections so far have been done after the fact. Consequently, “there are elections virtually all the time, thus, if you are claiming you have a prediction method you should predict an election in the future,” Gayo-Avello notes.  Report

Market Watch, WSJ

Fujitsu today announced the development of new software offerings that help customers utilize big data. The new products consist of two product families: a line of parallel distributed processing and complex event processing products -which are standard technologies in big data applications – and a line of products to employ big data in a wide range of uses. Fujitsu plans to gradually roll out the new products worldwide.

The new offerings include two new products: Interstage Big Data Complex Event Processing Server V1, which is a complex event processing solution, and Interstage eXtreme Transaction Processing Server V1, an extreme transaction solution. Furthermore, Fujitsu is releasing an enhanced version of its Symfoware Server V11 database solution.

Delivered as on-premise software for even greater convenience, these new products represent a commercialization of the technologies and operational know-how honed by Fujitsu through its world-leading cloud services for utilizing big data, Data Utilization Platform Services(1). In addition to enabling high reliability and performance by incorporating many proprietary Fujitsu technologies that boast a strong track record in mission-critical systems, the solutions can be easily deployed and operated. They can also be combined with products from other vendors, including open-source software, to build ecosystems(2) that support customers in using big data.

In recent years, the spread of technologies like smartphones, tablet devices and other smart devices, as well as various kinds of sensors and IC tags has caused the amount of data produced in our daily lives to grow at an explosive pace. At the same time, many companies are keen to use big data in their business activities. This includes, for example, discovering trends and predictions from various logs and data sources, and then leveraging this information in new business areas or to improve existing operations.

……………………

Overview of the New Products

1. Highly reliable parallel distributed processing – Interstage Big Data Parallel Processing Server V1 (available since February)

Interstage Big Data Parallel Processing Server is a parallel distributed processing software platform that enables improved data reliability by combining the Apache Hadoop open source big data parallel distributed processing platform with Fujitsu’s own proprietary distributed file system. At the same time, by sharing files between servers, the software eliminates the need for transferring data, thereby allowing for improved processing performance. Furthermore, the setup feature enables short turnaround for system deployment.

2. Easy-to-use complex event processing – Interstage Big Data Complex Event Processing Server V1 (new)

Report

by Linda Vu, ESnet

The U.S. Department of Energy’s Energy Sciences Network (ESnet) is laying the foundation for a high-speed network that can transport an increasing amount of scientific data. “Over the last decade, the amount of scientific data transferred over our network has increased at a rate of about 72 percent per year, and we see that trend potentially accelerating,” says ESnet director Greg Bell.

For example, climate researchers are producing some of the fastest growing datasets in science. Five years ago, the amount of information generated for the Nobel Prize-winning United Nations International Panel on Climate Change (IPCC) Fourth Assessment Report was 35 terabytes—equivalent to the amount of text in 35 million books, occupying a bookshelf 248 miles (399 km) long. By 2014, when the next IPCC report is published, experts predict that 2 petabytes of data will have been generated for it—that’s a 580 percent increase in data production.

ESnet researchers worked with the Internet2 consortium to develop the Advanced Networking Initiative (ANI), which is part of a 100 Gbps national prototype network and a wide-area network testbed. More than 25 groups have taken advantage of ESnet’s wide-area testbed, which connects the National Energy Research Scientific Computing Center, the Argonne Leadership Computing Facility, and the Oak Ridge Leadership Computing Facility.

“Our 100G testbed has been about 80 percent booked since it became available in January, which just goes to show that there are a lot of researchers hungry for a resource like this,” says ESnet’s Brian Tierney. For example, Brookhaven National Laboratory researchers have used the ANI testbed to design an ultra-high-speed, end-to-end file transfer protocol tool to move science data at 100 gigabits per second across a national network. Report

The real-time benefits of complex event processing: David Luckham explains

by Peter Schooff.

Listen to the podcast with David Luckham, emeritus professor of electrical engineering at Stanford University, and author of Event Processing for Business: Organizing the Real-Time Enterprise, published by Wiley. In this podcast we drill down on one of the keys to today’s real-time enterprise: complex event processing. Podcast

DCL: unfortunately the sound quality leaves something to be desired, probably due to the microphone I was using during the interview with Peter. But I think it is clear enough.

by Paul Francuch,  University of Illinois at Chicago

University of Illinois at Chicago (UIC) professor Jakob Eriksson is developing a traffic model that gathers and assembles data from smartphones, online maps, radio, and TV.

“The long-range ambition is to build something like a roadmap, but annotated with everything that’s going on at the moment,” Eriksson says. He says mathematical calculations that could predict alternate routes on side streets and secondary routes could help motorists bypass traffic jams. A more dynamic, real-time traffic picture also could help commuters decide when it would be more practical to use mass transit.

“This project asks, how do we combine it all into one coherent model showing the current status of the transportation network?” Eriksson notes. Initially, UIC researchers will use Chicago-area data for the project, but they plan to branch out into other cities in the future.

“There may be 10,000 road segments we want to simulate in some detail with perhaps a million cars on the road during rush hour,” Eriksson says. He notes that data not currently being fed into traffic reporting systems could be used for the model. Road sensors, tollway transponders mounted on windshields, and “jam cam” traffic cameras also could supply data.

Report:   http://tigger.uic.edu/htbin/cgiwrap/bin/newsbureau/cgi-bin/index.cgi?from=Releases&to=Release&id=3456&start=1325347568&end=1333123568&topic=0&dept=0

by James Bamford, Wired News 

The Utah desert is the site of a massive surveillance facility being built for the U.S. National Security Agency (NSA), with the purpose of secretly intercepting, deciphering, analyzing, and storing vast volumes of communications. The size of the center, combined with the data storage capacity of modern technologies, makes the volume of information that could be contained in the facility truly staggering.

However, the amount of intelligence data generated daily by NSA and other agencies is growing exponentially. Consequently, the Pentagon is expanding the Global Information Grid to handle data on the scale of yottabytes.

A key focus of the Utah center will be sifting through data housed in the deep Web, which is beyond public reach and includes the stolen classified secrets of potential adversaries. Other sources of data feeding into the center will include NSA’s eavesdropping satellites and overseas listening posts.

Under construction by contractors with top-secret clearances, the blandly named Utah Data Center is being built for the National Security Agency. A project of immense secrecy, it is the final piece in a complex puzzle assembled over the past decade. Its purpose: to intercept, decipher, analyze, and store vast swaths of the world’s communications as they zap down from satellites and zip through the underground and undersea cables of international, foreign, and domestic networks. The heavily fortified $2 billion center should be up and running in September 2013. Flowing through its servers and routers and stored in near-bottomless databases will be all forms of communication, including the complete contents of private emails, cell phone calls, and Google searches, as well as all sorts of personal data trails—parking receipts, travel itineraries, bookstore purchases, and other digital “pocket litter.” It is, in some measure, the realization of the “total information awareness” program created during the first term of the Bush administration—an effort that was killed by Congress in 2003 after it caused an outcry over its potential for invading Americans’ privacy.

Code-breaking also will be a critical function of the center, as much of the data that it will manage will be heavily encrypted, according to a senior intelligence official.

To aid in the decryption effort, scientists and computer engineers are building supercomputers and cryptanalytic applications at the Multiprogram Research Facility in Oak Ridge, Tennessee.  Read the article

DCL: This is an excellent article reporting on a very complex event processing facility for “information awareness.”

by RANDALL STROSS,  NYT Business Day

The U.S. Defense Advanced Research Projects Agency (DARPA) wants to confirm a user’s identity by the way they type on a keyboard instead of using passwords. That’s a vision that the Defense Advanced Research Projects Agency, part of the Defense Department, wants to turn into a reality. It will distribute research funds to develop software that determines, just by the way you type, that you are indeed the person you say you are.

“What I’d like to do is move to a world where you sit down at a console, you identify yourself, and you just start working, and the authentication happens in the background, invisible to you, while you continue to do your work without interruptions,” says DARPA’s Richard Guidorizzi.

Various research efforts are underway to authenticate users by the way they operate a computer. For example, Carnegie Mellon University professor Roy Maxion is researching keystroke dynamics, including the length of time a user holds down a given key and moves from one particular key to another. Pace University professor Charles Tappert has developed software that analyzes the distinctive patterns of keyboard pressure and can accurately confirm the identity of a test taker in 99.5 percent of cases.

Columbia University professor Salvatore J. Stolfo has developed software that uses a decoy document designed to lure and catch intruders. When a decoy file is opened, the system software checks to see whether the user has conducted file searches on the computer that fit the expected search pattern. If there is no close match, the system sets off an alarm and asks the user for an identity confirmation. ….. Report

DCL: Seems to amount to some research into developing sophisticated complex event processing technology.

from PC Mag.com

Artificial intelligence has recently seen some great developments: better search engines, improved speech recognition programs, IBM’s Watson project, and more.

Underlying all of these advances are models of machine learning. Often, these models are in turn based on mathematical frameworks created by Judea Pearl, professor emeritus at UCLA, that enable algorithms that incorporate real world experiences and human reasoning.

…….

ACM Executive Director John White told me that “Pearl’s research was instrumental in moving machine-based reasoning from the rules-bound expert systems of the 1980s to a calculus that incorporates uncertainty and probabilistic models.” In other words, he has figured out methods for trying to draw the best conclusion, even when there is a degree of uncertainty. It can be applied when trying to answer questions from a large amount of unstructured information, or trying to figure out what someone has said in languages that have lots of similar-sounding words—all things we do a lot today.

Vinton Cerf, an Internet pioneer and a former Turing Award recipient, explained that “Judea Pearl persisted in his deep analysis of Bayesian methods to extract useful information from partial data. In the face of skepticism, he persevered and, ultimately, his work has had enormous impact, not only on the theoretical basis for computational reasoning but in a very real sense, the basis for the successful business models of companies that search the World Wide Web.”……   Report

DCL: Judea’s thinking applies to event processing and decision making based upon event input. BTW, Judea Pearl is one of the nicest guys I’ve ever met.

by Andrew Brust,  ZDNet.com

In a recent post, I discussed MPP (Massively Parallel Processing) data warehouse technology and how it too is a Big Data technology. I also explained some things MPP has in common, architecturally, with Hadoop’s MapReduce parallel computation approach to handling large data sets.

There’s another three-letter acronym that is at once its own category and yet also a legitimate Big Data player: CEP (Complex Event Processing). CEP engines process continuous data flows (like feeds, sensor readings and other event-driven data that is large in volume and very fast-flowing), in real-time. CEP engines are available from so-called mega-vendors like Oracle, Sybase/SAP and Microsoft, integration players like TIBCO and pure play companies like StreamBase and OneMarketData.

As with other data-related domains, there’s a lot of cross-pollination and conflation happening between CEP and Hadoop/MapReduce. As an extreme example, I’ve seen one blog post that asserted Hadoop is a CEP Engine. But there are other, perhaps less-controversial examples as well.

In August of 2011, Twitter acquired a company called BackType, and a product of theirs called Storm. Storm processes streaming data in parallel, over a cluster of machines, using a processing architecture called topologies. Technology like this works well for a social media data hotbed like Twitter. Just think of the real-time processing necessary to determine trending topics, and you’ll start to get the idea. The keepers of Storm point out that MapReduce processes “jobs” in batch while Storm processes streams in real-time. Storm and MapReduce have a divide-and-conquer parallelism in common, and Storm even uses the ZooKeeper distributed process coordinator that began as an Apache sub-project of Hadoop. Twitter made Storm open source in September, 2011. So look for mashups of Storm and other data-centric technologies on the horizon.

Another interesting CEP-MapReduce crossover comes from Microsoft. It has devised a way for its StreamInsight CEP engine (which ships as part of the SQL Server relational database) to work in concert with the Microsoft-Hortonworks Hadoop on Windows distribution. Interestingly, Microsoft has figured out a way to shim the StreamInsight engine into the reducer step of Hadoop’s MapReduce jobs. This is explained, complete with examples, in a Microsoft blog post.

These are just a couple of examples of how CEP and Big Data technologies can be combined in ways that are logical and beneficial. Lots of other data technologies, like data visualization, predictive analytics and more, fit in with Big Data, and stretch its definition, in a likewise manner. We’ll certainly look at more of them in future posts. Brust’s Blog