|
|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
| (slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix | |||||||
|
|
John Morrell
InfoManagement Direct, November 3, 2006Complex event processing (CEP) involves the continuous processing and analysis of high-volume, high-speed data streams from inside and outside an organization to detect business-critical issues as they happen. In comparison to traditional intelligence processes, which provide delayed analysis, CEP software processes data streams and detects business events in real-time. Some examples of CEP applications are:
- Real-time financial market data analysis and enrichment,
- Financial trade auditing and compliance,
- IT security event correlation,
- Asset management and tracking using RFID, and
- Manufacturing process, power grid or energy pipeline monitoring.
The vast majority of event processing applications today are custom-coded. Much of this custom coding effort, however, can be eliminated by using CEP software; the level of time and cost savings corresponds with the complexity of the event processing application. The remainder of this article will articulate a framework by which you can understand where and to what degree CEP software can offer cost savings over custom development.
What Does CEP Offer?
CEP software offers two major components: a high-level language for programmers to easily describe how to process the streams, and an infrastructure engine for processing and analyzing high-volume data streams. Although CEP software performs different functions, the component structure is mildly analogous to database software, where there is a language (SQL) and an engine (the database server).
Because some of the operations a programmer wants to perform on data streams are similar to a relational model, a select number of CEP vendors offer a language that is based on SQL. This provides a familiar programming environment, speeding the creation of event processing applications.
The engine provides the core components to execute the analysis at run-time. The engine takes on many complex tasks typical in data management infrastructure software as well as those unique to event processing:
- Stream management: Data streams are analogous to a database table of infinite size, with each new event appending a row onto the table. As streams often travel over a network, there can be issues such as dropped, delayed or out-of-order messages. A good CEP engine will automatically handle all these issues without requiring programmer intervention, ensure reliable message delivery and generate a valid, dependable stream for processing.
- Memory management: Data streams can become very large and have many queries running against them. A good CEP engine needs to optimize how memory is managed to ensure high throughput. Special care must be taken to avoid copying and ensure that every piece of data is only stored once.
- Parallel execution and synchronization: To maintain performance, a CEP engine will perform operations in parallel and synchronize data between the threads. Excess synchronization can hurt performance. Thus, a CEP engine not only has to automatically perform state synchronization for the programmer, but it must also balance the synchronization rates for efficient execution.
- Windows: Processing on data streams is performed in "windows," typically, units of time. An efficient CEP engine must be able to expire messages properly, both on new events and timer events.
- Indexing: Fast-moving data streams require indexes to be continually updated at a similar high rate for efficient processing. A good CEP engine will automatically manage these indexes so the programmer does not have to deal with such issues.
These and many more functions are abstracted from the programmer, making the development of CEP applications easier.
Types of Event Processing Applications
If a developer were to create a custom-coded event processing application, he or she would need to code some if not all of the CEP engine features mentioned above, depending on the complexity of the event processing application.
To simplify the framework for determining the applicability of CEP software, let's examine event processing applications in four tiers:
- Tier One: simple event processing applications,
- Tier Two: event processing applications involving multiple streams and/or stored data,
- Tier Three: complex analysis and pattern matching across event streams, and
- Tier Four: multiple, enterprise-class event processing applications.
Abstract: Yemanja is a model-based event correlation engine for multi-layer fault diagnosis. It targets complex propagating fault scenarios, and can smoothly correlate low-level network events with high-level application performance alerts related to quality of service violations. Entity models that represent devices or abstract components encapsulate entity behavior. Distantly associated entities are not explicitly aware of each other, and communicate through event propagation chains.
Cited by: More
Non-deterministic Diagnosis of End-to-End Service Failures in a.. - Steinder (2001) (Correct)
The present and future of event correlation: A need for.. - Steinder, Sethi (2001) (Correct)
Combinatorial Designs In Multiple Faults Localization For.. - Fecko, Steinder (2001) (Correct)
Active bibliography (related documents): More All
0.8: End-to-end Service Failure Diagnosis Using Belief Networks - Steinder, Sethi (2002) (Correct)
0.6: Increasing Robustness of Fault Localization Through Analysis.. - Steinder, Sethi (2002) (Correct)
0.3: IP Fault Localization Via Risk Modeling - Ramana Rao Kompella (2005) (Correct)
Similar documents based on text: More All
0.4: Intelligent Search of Correlated Alarms for GSM Networks.. - Zheng, Xu, Lv, Ma (2002) (Correct)
0.3: A Conceptual Framework for Network Management Event.. - Masum Hasan Binay (Correct)
0.3: GulfStream - a System for Dynamic Topology.. - Fakhouri.. (2001) (Correct)
Related documents from co-citation: More All
6: IFIPIEEE International Symposium Integrated Network Management (context) - IFIP, Symposium et al. - 2001
3: Alarm correlation (context) - Jakobson, Weissman - 1993
3: High speed and robust event correlation (context) - Yemini, Kliger - 1996
Citations (may not include all citations):
107 Remote Network Monitoring Management Information Base - Waldbusser - 1995
46 Oceano -- SLA-based management of computing utility (context) - Appleby, Fakhouri et al.
36 Alarm correlation (context) - Jakobson, Weissman - 1993
30 Schemes for fault identification in communication networks - Katzela, Schwartz - 1995
25 High speed and robust event correlation (context) - Yemini, Kliger et al. - 1996
23 Event correlation using rule and object based techniques (context) - Nygate - 1995
22 and Internetworking Protocols (context) - Perlman, Second et al. - 1999
20 GEM -- a generalised event monitoring language for distribut.. (context) - Mansouri-Samani, Sloman - 1997
17 A Complete Guide to DB2 Universal Database (context) - Chamberlin - 1998
15 Definition of Managed Objects for Bridges (context) - Decker, Langille et al. - 1993
14 A case-based reasoning approach to the resolution of faults .. (context) - Lewis - 1993
11 Event correlation in heterogeneous networks using the OSI ma.. (context) - Jordaan, Paterok - 1993
10 Layered model for supporting fault isolation and recovery (context) - Gopal - 2000
10 A conceptual framework for network management event correlat.. - Hasan, Sugla et al. - 1999
9 Scaling Internet services by dynamic allocation of connectio.. (context) - Goldszmidt, Hunt - 1999
9 Composite events for network event correlation - Liu, Mok et al. - 1999
8 Towards a practical alarm correlation system (context) - Houck, Calo et al. - 1995
6 Alarm correlation engine (context) - Wu, Bhatnagar et al. - 1998
3 Service Level Agreements : Managing Cost and Quality in Serv.. (context) - Hiles - 1993
2 Value-oriented network management (context) - Schwartz, Zager - 2000
1 A modeling framework for integrated distributed systems faul.. (context) - Katker - 1996
1 A Simple Network Management Protovol (context) - Case, Fedor et al. - 1990
1 IBM Internal Article (context) - Appleby, Fakhouri et al.
1 Management Information Base Network Mangement TCPIP based in.. (context) - Rose, Base et al. - 1991
http://www.adventnet.com
Documents on the same site (http://www.cis.udel.edu/~steinder/PAPERS/index.html): More
End-to-end Service Failure Diagnosis Using Belief Networks - Steinder, Sethi (2002) (Correct)
Increasing Robustness of Fault Localization Through Analysis.. - Steinder, Sethi (2002) (Correct)
The present and future of event correlation: A need for.. - Steinder, Sethi (2001)
With the increasing complexity of enterprise networks and the Internet, event correlation is playing an increasingly important role in network as well as integrated system management systems. Even though the timing of events often reveals important diagnostic information about event relationships and should therefore be represented in event correlation rules or models, most extant approaches lack a formal mechanism to define complex temporal relationships among correlated events. In this paper, we discuss the formal use of composite events for event correlation and present a composite event specification approach that can precisely express complex timing constraints among correlated event instances, for which efficient compilation and detection algorithms have been developed in [13, 14]. A Java implementation of this approach, called Java Event CorrelaTOR (JECTOR), is described, and some preliminary experimental results of using JECTOR in an experimental network management environment are also discussed in the paper.
Abstract: This paper addresses the problem of efficient management of events, in particular in those environments where events carry information useful to multiple applications, possibly operating in different domains and at different levels of abstraction. We investigate the problems and opportunities offered by such environments, and define a framework that enables a semantic mapping of events, i.e., enables the processing and successive refinement of events at different levels of abstraction, so that they can be understood and efficiently consumed by business applications. We identify the requirements of an event mapping system and present a specification language, integrating high-level Petri nets and database query languages, which provides the required expressive power to specify complex event processing functions and includes a set of constructs that support the design process and allows efficient implementations.
Event correlation simplifies and speeds the monitoring of network events by consolidating events and error logs into a short, easy-to-understand package. A network administrator can deal with, say, 25 events based on cross-referencing intrusion events against firewall entries and host/asset databases much more efficiently than when he must scan 10,000 mostly normal log entries.The benefits can be very real: more efficient use of staff time and skills, as well as the prevention of revenue loss resulting from downtime.
According to Marcus Ranum, an independent computer and communications security consultant in Woodbine, Md., "Correlation is something everyone wants, but nobody even knows what it is. It's like liberty or free beer -- everyone thinks it's a great idea and we should all have it, but there's no road map for getting from here to there." Still, a variety of technologies and operations are associated with event correlation:
Compression takes multiple occurrences of the same event, examines them for duplicate information, removes redundancies and reports them as a single event. So 1,000 "route failed" events become a single events that says "route failed 1,000 times."
Counting reports a specified number of similar events as one. This differs from compression in that it doesn't just tally the same event and that there's a threshold to trigger a report.
Suppression associates priorities with events and lets the system suppress an alarm for a lower-priority event if a higher-priority event has occurred.
Generalization associates events with some higher-level events, which are what's reported. This can be useful for correlating events involving multiple ports on the same switch or router in the event that it fails. You don't need to see each specific failure if you can determine that the entire unit has problems.
Time-based correlation can be helpful establishing causality -- for instance, tracing a connectivity problem to a failed piece of hardware. Often more information can be gleaned by correlating events that have specific time-based relationships. Some problems can be determined only through such temporal correlation. Examples of time-based relationships include the following:
• Event A is followed by Event B.
• This is the first Event A since the recent Event B.
• Event A follows Event B within two minutes.
• Event A wasn't observed within Interval I.
Winning Users Over
"Event correlation, in its basic form, is becoming almost a commodity product," says Drogseth. "Where you want to reduce the number of events and events and have some level of topological awareness to eliminate duplicates -- that's pretty standard and working today." Buyers are skeptical, but Drogseth says many event-correlation products work well out of the box or with minimal customization.
"There are any number of more sophisticated approaches that are all about diagnostics, finding out what is the real cause of a problem," Drogseth says. "Here, you have to address a lot more complexity in network infrastructure." When you start trying to isolate a problem and get at the true root cause, he says, "you have a high level of investment and complexity, but also a high level of value."
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|
|
You can use PayPal to to buy a cup of coffee for authors of this site |
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: March 12, 2019
