流处理的特征与对比

 

——读The 8 Requirements of Real-Time Stream Processing笔记

 

       这篇文章介绍了8条实时流处理所需要的要求与特征，然后对比了传统DBMS（或者是内存DBMS），Rule engine与SPE在处理流数据方面所能达到的上面8条特征中的几条。

 

1.   Eight Rules for stream processing

Rule 1: Keep the data moving

       The first requirement for a real-time stream processing system is to process messages “in-stream”, without any requirement to store them to perform any operation or sequence of operations. Ideally the system should also use an active (i.e., non-polling) processing model.

 

Rule 2: Query using SQL on Streams(StreamSQL)

       The second requirement is to support a high-level “StreamSQL” language with built-in extensible stream-oriented primitives and operators. 

 

Rule 3: Handle stream imperfections(delayed, missing, and out-of-order data)

       The third requirement is to have built-in mechanisms to provide resiliency against stream “imperfections”, including missing and out-of-order data, which are commonly present in real-world data streams. 

 

Rule 4: Generate Predictable Outcomes

       The fourth requirement is that a stream processing engine must guarantee predictable and repeatable outcomes.

 

Rule 5: Integrate Stored and Streaming Data

       The fifth requirement is to have the capability to efficiently store, access, and modify state information, and combine it with live streaming data. For seamless integration, the system should use a uniform language when dealing with either type of data. 

 

Rule 6: Guarantee Data Safety and Availability

       The sixth requirement is to ensure that the applications are up and available, and the integrity of the data maintained at all times, despite failures. 

 

Rule 7: Partition and Scale Applications Automatically

       The seventh requirement is to have the capability to distribute processing across multiple processors and machines to achieve incremental scalability. Ideally, the distribution should be automatic and transparent. 

 

Rule 8:  Process and Respond Instantaneously

       The eighth requirement is that a stream processing system must have a highly-optimized, minimal-overhead execution engine to deliver real-time response for high-volume applications. 

 

2.   DBMS, Rule Engine, SPE对比

      

 

DBMS在处理数据上是先存储后处理的，即“process-after-store” model. 所以在处理实时数据流方面天生就不是适合,尽管可以利用内存数据库来缓和效率方面的弱势， 同时其也具备trigger的特性， 但所有这些都不显得不够可扩展。

       Rule Engine 虽然某种程度上能够处理实时的数据流， 但其在Rule Language方面有欠缺， 不能够拥有类似SQL的表达能力。对数据流的处理操作有限。

       只有SPE是专门为处理实时流数据定做的。有许多天生的特性，专门用来处理和操作流数据。

       下面就是它们的一个对比：

	DBMS	Rule engine	SPE
Keep the data moving	No	Yes	Yes
SQL on streams	No	No	Yes
Handle stream imperfections	Difficult	Possible	Possible
Predictable outcome	Difficult	Possible	Possible
High availability	Possible	Possible	Possible
Stored and streamed data	No	No	Yes
Distribution and scalability	Possible	Possible	Possible
Instantaneous response	Possible	Possible	Possible