AMAZON Kinesis(1)Introduction (Old Note)

AMAZON Kinesis(1)Introduction (Old Note)

1 Introduction
Price:
Send Throughput:
Receive Throughput:
Message Latency:
Message Size: MB/S
Batch Size:
Persist Time: 24 hours

Producers: EC2 Instances, Clients, Mobile Clients, Traditional Servers
Consumers: S3, Amazon Redshift and etc.

The data in the stream are distributed into shards. Each record in the stream has a sequence number. The data blob can be up to 1MB.
The retention period of data can be from 24 hours to 7 days.

Each shard can support up to 5 transactions per second for reads, up to 2 MB per second 
1,000 records per second writes, up to 1 MB per second

number_of_shards = max(incoming_write_bandwidth_in_KB/ outgoing_read_bandwidth_in_KB/2000)

Consumers can be auto scaling as well.

Limitation:
1.  50 shards for US East(N. Virginia)    US West (Oregon)   EU(Ireland). All other regions have a default shard limit of 25.
2.  1 to 7 days.
3.  Data blob is up to 1 MB.
http://docs.amazonaws.cn/en_us/kinesis/latest/dev/service-sizes-and-limits.html

Kinesis -> S3 -> Redshift, We can put selected columns into Redshift, we can also add logic in lambda if we want.
http://www.csdn.net/article/2015-10-09/2825871

Producer —> Kinesis —> Redis Cluster Persist
http://www.csdn.net/article/2015-01-07/2823464

Kinesis and Spark Streaming
https://github.com/aiyanbo/spark-programming-guide-zh-cn/blob/master/spark-streaming/basic-concepts/kinesis-integration.md


Reference:
http://aws.amazon.com/kinesis/faqs/
search for
"When should I use Amazon Kinesis, and when should I use Amazon SQS"

https://github.com/aiyanbo/spark-programming-guide-zh-cn/blob/master/spark-streaming/basic-concepts/kinesis-integration.md
http://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-create-stream.html