S4-环境搭建

AliKevin 写道

   本系列文章不涉及过多的S4的理论内容，因为s4论文中描述相当清楚（我认为我实在是说的不会比论文中更清楚了：）呵呵），论文信息请看论文http://dl.iteye.com/topics/download/704e5924-0dd8-34df-b44f-2efbc91de071

一、s4版本管理工具git的安装（ubuntu系统）

• a.执行如下命令：

root@slave:~# sudo apt-get install git git-core       
Reading package lists... Done        
Building dependency tree       
Reading state information... Done      
The following extra packages will be installed:       
emacsen-common git-man liberror-perl       ....

• b.安装后检查是否安装成功，执行如下git命令如下：

root@slave:~# git help
usage: git [--version] [--exec-path[=<path>]] [--html-path]
           [-p|--paginate|--no-pager] [--no-replace-objects]
           [--bare] [--git-dir=<path>] [--work-tree=<path>]
           [-c name=value] [--help]
           <command> [<args>]

The most commonly used git commands are:
   add        Add file contents to the index
   bisect     Find by binary search the change that introduced a bug
   branch     List, create, or delete branches
   checkout   Checkout a branch or paths to the working tree
   clone      Clone a repository into a new directory
   commit     Record changes to the repository
   diff       Show changes between commits, commit and working tree, etc
   fetch      Download objects and refs from another repository
   grep       Print lines matching a pattern
   init       Create an empty git repository or reinitialize an existing one
   log        Show commit logs
   merge      Join two or more development histories together
   mv         Move or rename a file, a directory, or a symlink
   pull       Fetch from and merge with another repository or a local branch
   push       Update remote refs along with associated objects
   rebase     Forward-port local commits to the updated upstream head
   reset      Reset current HEAD to the specified state
   rm         Remove files from the working tree and from the index
   show       Show various types of objects
   status     Show the working tree status
   tag        Create, list, delete or verify a tag object signed with GPG

See 'git help <command>' for more information on a specific command.

二、s4构建工具的依赖安装

在V0.3.x版本采用gradle进行构建代码，gradle能够对现有的maven机制进行利用。
1).安装Maven
a.download mavenhttp://mirror.bjtu.edu.cn/apache//maven/binaries/apache-maven-3.0.3-bin.tar.gz
b.安装参考http://maven.apache.org/download.html#Installation
2).安装Gradle
注：在进行s4构建前会自动下载,对构建s4这一步骤不是必须的，但我们开发自己程序时候仍然需要（可选）gradle：）
a.下载gradlehttp://repo.gradle.org/gradle/distributions/gradle-1.0-milestone-3-all.zip]
b.将gradle解压到你安装目录(gradle)，我这里是/opt/gradle
c.修改prefile文件，增加环境变量S4_IMAGE。

root@slave:/kevin/s4# vi /etc/profile
       ...
       export S4_IMAGE=/kevin/s4/build/s4-image
       ...
   root@slave:source /etc/profile

d.验证s4的构建安装情况，
1).启动s4

root@slave:/kevin/s4# $S4_IMAGE/scripts/start-s4.sh
root@slave:/kevin/s4# $S4_IMAGE/scripts/start-s4.sh
Temp is /tmp/tmp.XNzI2ZfztP
CORE_HOME='/kevin/s4/build/s4-image/s4-core'
APPS_HOME='/kevin/s4/build/s4-image/s4-apps'
EXTS_HOME='/kevin/s4/build/s4-image/s4-exts'
GC_OPTS='-server -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit'
MEM_OPTS='-Xms800m -Xmx2000m'
JAVA_OPTS=' -Dcommlayer.mode=static  -Dzk.session.timeout=5000  -Dlock_dir=/kevin/s4/build/s4-image/s4-core/lock  -Dlog_loc=/kevin/s4/build/s4-image/s4-core/logs '
java location is /usr/jdk1.6.0_26/bin/
JAVA VERSION=java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode)

RUNNING /usr/jdk1.6.0_26/bin/java -server -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit  -Xms800m -Xmx2000m  -Dcommlayer.mode=static  -Dzk.session.timeout=5000  -Dlock_dir=/kevin/s4/build/s4-image/s4-core/lock  -Dlog_loc=/kevin/s4/build/s4-image/s4-core/logs  -classpath /kevin/s4/build/s4-image/s4-core:/kevin/s4/build/s4-image/s4-core/lib/activation-1.1.jar:/kevin/s4/build/s4-image/s4-core/lib/jakarta-regexp-1.4.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-jexl-1.1.jar:/kevin/s4/build/s4-image/s4-core/lib/bcel-5.2.jar:/kevin/s4/build/s4-image/s4-core/lib/minlog-1.2.jar:/kevin/s4/build/s4-image/s4-core/lib/asm-3.2.jar:/kevin/s4/build/s4-image/s4-core/lib/log4j-1.2.15.jar:/kevin/s4/build/s4-image/s4-core/lib/gson-1.6.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-logging-1.1.1.jar:/kevin/s4/build/s4-image/s4-core/lib/mail-1.4.jar:/kevin/s4/build/s4-image/s4-core/lib/junit-3.8.1.jar:/kevin/s4/build/s4-image/s4-core/lib/reflectasm-0.8.jar:/kevin/s4/build/s4-image/s4-core/lib/kryo-1.01.jar:/kevin/s4/build/s4-image/s4-core/lib/s4-comm-0.3.0.jar:/kevin/s4/build/s4-image/s4-core/lib/json-20090211.jar:/kevin/s4/build/s4-image/s4-core/lib/spring-2.5.6.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-cli-1.2.jar:/kevin/s4/build/s4-image/s4-core/lib/jline-0.9.94.jar:/kevin/s4/build/s4-image/s4-core/lib/flexjson-2.1.jar:/kevin/s4/build/s4-image/s4-core/lib/s4-core-0.3.0.jar:/kevin/s4/build/s4-image/s4-core/lib/zookeeper-3.3.1.jar::::/tmp/tmp.XNzI2ZfztP:/kevin/s4/build/s4-image/s4-core/conf/default -DDequeuerCount=6 -Dlog4j.configuration=file:/kevin/s4/build/s4-image/s4-core/conf/default/log4j.xml io.s4.MainApp -c /kevin/s4/build/s4-image/s4-core -a /kevin/s4/build/s4-image/s4-apps -e /kevin/s4/build/s4-image/s4-exts -t default -d wall
appName=s4
dequeuer number: 6
[]
[]

如上信息证明s4启动成功，但此时没有部署任何应用在s4中。

2).启动adapter

root@slave:~# $S4_IMAGE/scripts/run-client-adapter.sh -s client-adapter \
> -g s4 -d $S4_IMAGE/s4-core/conf/default/client-stub-conf.xml
configuration location is /kevin/s4/build/s4-image/s4-core/conf/default
commlayer mode is static
Cluster manager localhost:2181
Sender cluster name client-adapter
Listener cluster name s4
java location is /usr/jdk1.6.0_26/bin/
JAVA VERSION=java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode)

Temp is /tmp/tmp.syup7UCLoB
appName=client-adapter
listenerAppName=s4
zk_address=localhost:2181
listener_max_queue_size=8000
Running /usr/jdk1.6.0_26/bin/java  -Dlock_dir=/kevin/s4/build/s4-image/s4-core/lock  -Dlog_loc=/kevin/s4/build/s4-image/s4-core/logs  -Dzk_session_timeout=5000 -Dcommlayer.mode=static -Dlog4j.configuration=file:/kevin/s4/build/s4-image/s4-core/conf/default/log4j.xml -classpath /kevin/s4/build/s4-image/s4-core/lib/activation-1.1.jar:/kevin/s4/build/s4-image/s4-core/lib/jakarta-regexp-1.4.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-jexl-1.1.jar:/kevin/s4/build/s4-image/s4-core/lib/bcel-5.2.jar:/kevin/s4/build/s4-image/s4-core/lib/minlog-1.2.jar:/kevin/s4/build/s4-image/s4-core/lib/asm-3.2.jar:/kevin/s4/build/s4-image/s4-core/lib/log4j-1.2.15.jar:/kevin/s4/build/s4-image/s4-core/lib/gson-1.6.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-logging-1.1.1.jar:/kevin/s4/build/s4-image/s4-core/lib/mail-1.4.jar:/kevin/s4/build/s4-image/s4-core/lib/junit-3.8.1.jar:/kevin/s4/build/s4-image/s4-core/lib/reflectasm-0.8.jar:/kevin/s4/build/s4-image/s4-core/lib/kryo-1.01.jar:/kevin/s4/build/s4-image/s4-core/lib/s4-comm-0.3.0.jar:/kevin/s4/build/s4-image/s4-core/lib/json-20090211.jar:/kevin/s4/build/s4-image/s4-core/lib/spring-2.5.6.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-cli-1.2.jar:/kevin/s4/build/s4-image/s4-core/lib/jline-0.9.94.jar:/kevin/s4/build/s4-image/s4-core/lib/flexjson-2.1.jar:/kevin/s4/build/s4-image/s4-core/lib/s4-core-0.3.0.jar:/kevin/s4/build/s4-image/s4-core/lib/zookeeper-3.3.1.jar:::/kevin/s4/build/s4-image/s4-core/conf/default:/tmp/tmp.syup7UCLoB io.s4.client.Adapter -t default -c /kevin/s4/build/s4-image/s4-core -d /kevin/s4/build/s4-image/s4-core/conf/default/client-stub-conf.xml
appName=client-adapter
dequeuer number: 12
Adding InputStub genericStub
Adding OutputStub genericStub

如上信息证明adapter已经启动。

e.s4环境配置完毕


三、错误解决

   s4的安装中可能出现找不到依赖的 core和comm的jar包,如：comm-xxxx.jar,kryo-1.01.jar,reflectasm-0.8.jar和minlog-1.2.jar，这时需要利用maven添加s4core模块和comm模块的maven库。构建时候发出如下错误警告：

:: problems summary ::
:::: ERRORS
        Server access Error: Connection reset url=http://google-gson.googlecode.com/svn/mavenrepo/com/esotericsoftware/kryo/1.01/kryo-1.01.pom

        Server access Error: Connection reset url=http://google-gson.googlecode.com/svn/mavenrepo/com/esotericsoftware/kryo/1.01/kryo-1.01.jar

        Server access Error: Connection reset url=http://google-gson.googlecode.com/svn/mavenrepo/com/esotericsoftware/reflectasm/0.8/reflectasm-0.8.pom

        Server access Error: Connection reset url=http://google-gson.googlecode.com/svn/mavenrepo/com/esotericsoftware/reflectasm/0.8/reflectasm-0.8.jar

        Server access Error: Connection reset url=http://google-gson.googlecode.com/svn/mavenrepo/com/esotericsoftware/minlog/1.2/minlog-1.2.pom

        Server access Error: Connection reset url=http://google-gson.googlecode.com/svn/mavenrepo/com/esotericsoftware/minlog/1.2/minlog-1.2.jar

我们要进行添加：

1.添加comm及相关依赖

root@slave:/# cd kevin
     root@slave:/kevin# git clone git://github.com/s4/comm.git
     root@slave:/kevin#cd comm
     root@slave:/kevin/comm# mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=zookeeper -Dversion=3.1.1 -Dpackaging=jar -Dfile=lib/zookeeper-3.1.1.jar
     root@slave:/kevin/comm# mvn install
  ....
     [INFO] [jar:jar {execution: default-jar}]
     [INFO] Building jar: /kevin/comm/target/comm-0.2.1.0.jar
     [INFO] [install:install {execution: default-install}]
     [INFO] Installing /kevin/comm/target/comm-0.2.1.0.jar to /root/.m2/repository/io/s4/comm/comm/0.2.1.0/comm-0.2.1.0.jar
     [INFO] ------------------------------------------------------------------------
     [INFO] BUILD SUCCESSFUL
     [INFO] ------------------------------------------------------------------------
     [INFO] Total time: 2 minutes 9 seconds
     [INFO] Finished at: Sun Oct 16 23:26:53 PDT 2011
     [INFO] Final Memory: 17M/42M
     [INFO] ------------------------------------------------------------------------

2.添加core及相关依赖

root@slave:/# cd kevin
     root@slave:/kevin# git clone git://github.com/s4/core.git
     root@slave:/kevin#cd core
     root@slave:/kevin/core# mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=kryo -Dversion=1.01 -Dpackaging=jar -Dfile=lib/kryo-1.01.jar
     root@slave:/kevin/core# mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=reflectasm -Dversion=0.8 -Dpackaging=jar -Dfile=lib/reflectasm-0.8.jar
     root@slave:/kevin/core# mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=minlog -Dversion=1.2 -Dpackaging=jar -Dfile=lib/minlog-1.2.jar
     root@slave:/kevin/core# mvn assembly:assembly install
      ....
      ....
      [INFO] [jar:jar {execution: default-jar}]
      [INFO] [install:install {execution: default-install}]
      [INFO] Installing /kevin/core/target/s4_core-0.3.0.0.jar to /root/.m2/repository/io/s4/s4_core/0.3.0.0/s4_core-0.3.0.0.jar
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESSFUL
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 1 minute 28 seconds
      [INFO] Finished at: Sun Oct 16 23:43:05 PDT 2011
      [INFO] Final Memory: 16M/49M
      [INFO] ------------------------------------------------------------------------

至此，相信你已经能将S4的环境配置成功了。：）