S4-多节点部署

AliKevin 写道

   本系列文章不涉及过多的S4的理论内容，因为s4论文中描述相当清楚（我认为我实在是说的不会比论文中更清楚了：）呵呵），论文信息请看论文http://dl.iteye.com/topics/download/704e5924-0dd8-34df-b44f-2efbc91de071

S4多节点(node)部署
前面几个章节我们主要描述介绍了S4细节上的开发，接下来的两章我们将在宏观上了感性认识一下S4多节点运行和与ZooKeeper的整合配置。
多节点运行的关键在于最clusters.xml文件的配置，那下面我们对查看一下单节点配置，并进行多节点配置和运行。
1.单节点配置
我们之前的运行的例子的配置是：

<config version="-1">
  <cluster name="s4" type="s4" mode="unicast">
    <node>
      <partition>0</partition>
      <machine>localhost</machine>
      <port>5077</port>
      <taskId>s4node-0</taskId>
    </node>
  </cluster>
<cluster name="client-adapter" type="s4" mode="unicast">
    <node>
      <partition>0</partition>
      <machine>localhost</machine> <!-- used only in red button mode -->
      <taskId>client-adapter-0</taskId>
      <port>6077</port>
    </node>
  </cluster>
</config>

配置中我们发现只有一个名字为s4类型为s4的集群配置，里面有1个节点。
2.多节点配置

<config version="-1">
  <cluster name="s4" type="s4" mode="unicast">
    <node>
      <partition>0</partition>
      <machine>localhost</machine>
      <port>5077</port>
      <taskId>s4node-0</taskId>
    </node>
    <node> <!-- this is the new node element -->
      <partition>1</partition>
      <machine>localhost</machine> <!-- used only in red button mode -->
      <port>5088</port>
      <taskId>s4node-1</taskId>
    </node>
  </cluster>
<cluster name="client-adapter" type="s4" mode="unicast">
    <node>
      <partition>0</partition>
      <machine>localhost</machine> <!-- used only in red button mode -->
      <taskId>client-adapter-0</taskId>
      <port>6077</port>
    </node>
  </cluster>
</config>

OK，我们在同一台机器上增加了一个新的节点.好了不用更改其他地方，S4就能进行多node的部署了。（似乎S4也满强大和灵活哦）
3.运行服务和测试
a.启动S4和Adapter

root@slave:/kevin/sellMoniter# rm -fr $S4_IMAGE/s4-apps/* 
root@slave:/kevin/sellMoniter# rm $S4_IMAGE/s4-core/logs/s4-core/*
root@slave:/kevin/sellMoniter# cp -r $S4_IMAGE/s4-example-apps/s4-example-speech02 $S4_IMAGE/s4-apps/
root@slave:/# $S4_IMAGE/scripts/start-s4.sh -r client-adapter 
appName=s4
dequeuer number: 6
[]
[/kevin/s4/build/s4-image/s4-apps/s4-example-speech02/s4-example-speech02-conf.xml]
Adding processing element with bean name eventCatcher, id eventCatcher
adding pe: io.s4.example.speech01.SentenceReceiverPE@c2ee15
Using ConMapPersister ..
Adding processing element with bean name rerouteSentencePE, id rerouteSentencePE
adding pe: io.s4.processor.ReroutePE@149105b
Using ConMapPersister ..
Adding processing element with bean name rerouteSpeechPE, id rerouteSpeechPE
adding pe: io.s4.processor.ReroutePE@196f4b5
Using ConMapPersister ..
Adding processing element with bean name sentenceJoinPE, id sentenceJoinPE
adding pe: io.s4.processor.JoinPE@1bfc4fc
Using ConMapPersister ..


root@slave:/kevin/s4# $S4_IMAGE/scripts/run-client-adapter.sh -s client-adapte-g s4 -d $S4_IMAGE/s4-core/conf/default/client-stub-conf.xml 
....
....
d/s4-image/s4-core/conf/default/client-stub-conf.xml
appName=s4
dequeuer number: 12
Adding InputStub genericStub
Adding OutputStub genericStub

如上信息证明启动成功。
b.运行测试类和查看日志
测试类

root@slave:/kevin/s4/build/s4-image/s4-core/logs/s4-core# head -20 $S4_IMAGE/s4-example-testinput/speeches.txt | sh $S4_IMAGE/s4-tools-loadgenerator/scripts/generate-load.sh -r 2 -a localhost:2334 -d 1500 -
[: 27: ==: unexpected operator
[: 57: xlocalhost:2334: unexpected operator
[: 61: x2: unexpected operator
[: 65: x1500: unexpected operator
[: 69: x: unexpected operator
{0=io.s4.tools.loadgenerator.LoadGenerator$EventTypeInfo@60420f, 1=io.s4.tools.loadgenerator.LoadGenerator$EventTypeInfo@19106c7, 2=io.s4.tools.loadgenerator.LoadGenerator$EventTypeInfo@540408}
Emitted 19 events
root@slave:/kevin/s4/build/s4-image/s4-core/logs/s4-core# 

服务端

.....
Sentence is 'It is altogether fitting and proper that we should do this.', location gettysburg, pa, us
Sentence is 'The brave men, living and dead who struggled here have consecrated it far above our poor power to add or detract.', location gettysburg, pa, us
Sentence is 'The world will little note nor long remember what we say here, but it can never forget what they did here.', location gettysburg, pa, us
Sentence is 'It is for us the living rather to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced.', location gettysburg, pa, us
.....

日志

root@slave:/kevin/s4/build/s4-image/s4-core/logs/s4-core# find $S4_IMAGE/s4-core/logs/s4-core -name "s4-core_*.log" -print -exec sh -c 'grep -i "count by" {} | tail -4' \;
/kevin/s4/build/s4-image/s4-core/logs/s4-core/s4-core_17078.log
/kevin/s4/build/s4-image/s4-core/logs/s4-core/s4-core_16917.log
2011-10-17 20:41:16,605 s4 INFO (PEContainer.java:362) Count by Sentence speechId: 4
2011-10-17 20:41:16,605 s4 INFO (PEContainer.java:362) Count by Speech id: 1
2011-10-17 20:41:16,605 s4 INFO (PEContainer.java:362) Count by SentenceJoined speechId: 4
2011-10-17 20:41:16,606 s4 INFO (PEContainer.java:362) Count by RawHighlight : 4
root@master:/kevin/s4/build/s4-image/s4-core/logs/s4-core# 

整体演示图如下：

这里没有显示出负载均衡，需要继续不同数据数据测试因为该例子采用的是Speechid和id作为key值，采用FNV1-64进行hash计算节点。：）