sensei cluster

http://senseidb.github.com/sensei/cluster.html


sensei中在node上层有一个broker的概念，会拦截data-event和query request.
每个broker都会从zookeeper那里得到一份shard-id to node-id map.

在索引的时候：
broker接受到事件的时候，会使用ShardingStrategy分发事件。
sensei.sharding.strategy定义进行分发的类。

在查询的时候：
broker接受到请求，会使用LoadBalancerFactory得到LoadBalancerFactory进行分桶。把不同的请求打到不同的node.
sensei.search.router.factory配置进行分发的类。


Under the hood

Each Sensei node reports to Zookeeper, our cluster manager, its state and which shards it has. Zookeeper propagates this information through-out the entire cluster, especially to the brokers. The brokers maintains an inverse map of shard id to node id list. This list is kept up to date with cluster topology changes from newly introduced nodes and/or node failures.
Sharding Strategy

At indexing time, the ShardingStrategy is applied from the data events streamed-in from the gateway for each node. So only data-events belong to a specific shard is added. ShardingStrategy can be configured with sensei.sharding.strategy setting in the sensei.properties file.
Load Balancer Factory

At query time, the broker uses the SenseiLoadBalancerFactory to get a SenseiLoadBalancer and generates a routing map to get a list of nodes to send the requests to. By default, broker uses consistent hash on the routing parameter provided by the request. Load balancer factory can be configured with sensei.search.router.factory setting in the sensei.properties file.

评论

3 楼 tsxm 2012-10-10  

shixin42 写道

tsxm 写道

用kafka的时候，kafka里面的数据会被所有相关shard所在node消费，还是仅仅随机只被一个node消费呢？（senseidb做了集群）

会在指定的shard消费！

这个我知道的，比如我有2个node，每个node都有2个shard，node0-0,1,   node1-0,1

现在加入被shard到0这个上面，那么node0和node1都应该会进行消费建立索引吧

2 楼 shixin42 2012-10-10  

tsxm 写道

用kafka的时候，kafka里面的数据会被所有相关shard所在node消费，还是仅仅随机只被一个node消费呢？（senseidb做了集群）

会在指定的shard消费！

1 楼 tsxm 2012-10-10  

用kafka的时候，kafka里面的数据会被所有相关shard所在node消费，还是仅仅随机只被一个node消费呢？（senseidb做了集群）