Technologies

A Tunable Fault-Tolerant Distributed Stream Processing System for Big Data Online Analytics


Abstract/Technology Overview

Conventional big data analytics systems (e.g., MapReduce, Dryad, Spark) are designed to work in an offline, batch-based manner originally. All data needs to be available in advance and will be processed as a whole. However, data is often generated continuously and needs to be processed in real time, for instance, network traffic data in the telecommunication environment.

 

The team developed a novel system for big data online distributed stream processing. It provides a high-performance, fault-tolerant, and generic analytics platform for various analytics applications, such as data synopsis, stream database queries, and online machine learning for telecommunication, big data analytics industries and IT service operators.


Technology Features, Specifications and Advantages

  • Our system realizes a novel concept called approximate fault tolerance, which reduces the number of backup operations to mitigate performance overheads for fault tolerance maintenance while ensuring that the stream processing errors upon failures are bounded
  • To address diverse application needs, the system can easily tune the trade-off between performance and accuracy with only a few parameters. Thus, can process more data and faster than other systems without fault tolerance


Potential Applications

The technology could be used for network measurements (example: anomaly detection, flow size distribution, failure diagnosis), data mining and machine learning (example: frequent pattern mining, classification, regression, prediction). Applications can be found in preventive maintenance of heavy traffic servers, in which abnormal or specific patterns can be identified for early detection of a potential failure.


OVERVIEW
Technology Owner

Cathy Wong

Company

The Chinese University of Hong Kong

Technology Category
  • Big Data, Data Analytics, Data Mining & Data Visualisation
  • Data Processing
Technology Status
  • Licensed but Available
Technology Readiness Level
  • TRL 7
Keywords

big data, data processing, data analytics, machine learning, distributed stream processing