Tuesday, August 4, 2009

Complex Event processing- The Smart Enterprise

Complex Event Processing or CEP, is a technology in transition- a precursor to more sophisticated autonomous decision processing emerging within the future enterprise.

CEP systems collect data from numerous sources and raw events within an environment such as a physical emergency situation or a company’s operations and uses algorithms and rules to determine in real time the interconnected trends and patterns that create complex scenarios. The results of this analysis is then channelled to the appropriate decision-maker for action.

This process should therefore be recognised as the beginning of the emergence of the ‘Smart Enterprise’. In time most major decisions within the enterprise will be associated with CEP events.

We are now also entering the era of the ‘Smart Planet' revolution. This is IBM’s mantra, but also that of Cisco, Google, Microsoft, Oracle, SAP, GE and every other major information services player. Adaptive and responsive techniques, largely autonomously managed, are beginning to be applied to the optimisation of the design, maintenance and operation of infrastructure and business processes. These include electricity and communication grids, healthcare, financial, transport, investment, building, engineering, emergency response and supply chain systems.

But in order for this revolution to occur, the enterprise must also evolve to be equivalently ‘smart’. Smart infrastructure without smart enterprise management won't compute.

Collecting the raw data for CEP will inevitably create information overload for the enterprise as sciences such as astronomy, biology and particle physics have already discovered, genrating massive datasets. Traditional relational databases and SOA architectures are not optimised for real time event processing, particularly as much of the data will be unstructured and garnered from heterogeneous sources such as web pages, videos, RSS feeds, market intelligence, statistical data, electronic devices and instrumentation, control systems and sensors.

In addition, CEP overlaps with the business intelligence domain. The latest CEP toolsets allow users to apply high level modelling tools, AI and query languages that allow them to implement business logic while processing event streams.

However, no matter how much filtering, pattern matching and analytic processing is applied to CEP data, human decision-making will still be a significant bottleneck. The future smart enterprise must have the flexibility to focus and deploy its cooperative intelligence in real time and autonomously, at all levels of the organisation in response to opportunities and competitive pressures in the marketplace.

The level of complexity of CEP and decision-making will continually and rapidly increase over time in response to the changing social and technological environment. The resulting complexity of networks of interactions involving customers, supply chains, services, markets and logistics will make it impossible for humans to respond effectively. It will become just too complex and time-consuming even for dedicated teams of humans to manage.

Real-time integration of disparate data and applications is a key challenge facing the future enterprise. Conventional approaches such as building a data warehouse to consolidate all data sources are expensive, slow and highly intrusive. A number of innovative CEP platforms are being developed in this sector based on enterprise information streaming models. These provide a virtual unified view of the data stream without first transferring it to a central repository.

The future smart enterprise will also be required to be proactive rather than reactive in order to optimise its response to a fast changing environment. That is it will be required to actively search for solutions and be knowledge driven. This will place further pressure on the need for real-time quality decision-making.

In the near future humans will be partners in the decision processes powered by CEP and smart algorithms, but over time their input, as for airline pilots and fast train drivers, will be largely symbolic.