【Jstorm 入门例子】

gaojingsong

浏览: 1211081 次
性别:
来自: 深圳

最近访客更多访客>>

ZZ_lll

boveysmith

zah5897

xckouy

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

阿里JStorm框架

Jstorm 入门例子

一、检查硬盘空间，启动jstorm环境

[root@hadoop0 ~]# cd /opt/bigdata/

[root@hadoop0 bigdata]# ls

apache-flume-1.6.0-bin apache-hive-2.0.1-bin.tar.gz hadoop272

hbase-1.1.5-bin.tar.gz jstorm-0.9.6.2 kafka sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz taokeeper-monitor.tar.gz zookeeper apache-flume-1.6.0-bin.tar.gz apache-tomcat-7.0.69.zip hbase-1.1.5 hive2.0 jstorm-0.9.6.2.zip sqoop-1.4.6 stomr096 tomcat7

[root@hadoop0 bigdata]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root 18G 9.3G 7.2G 57% /

tmpfs 250M 72K 250M 1% /dev/shm

/dev/sda1 485M 33M 427M 8% /boot

1）启动zookeeper环境

[root@hadoop0 bigdata]# cd zookeeper/bin/

[root@hadoop0 bin]# ./zkServer.sh start

JMX enabled by default

Using config: /opt/bigdata/zookeeper/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

2)启动Tomcat环境

[root@hadoop0 bin]# cd ../../tomcat7/bin/

[root@hadoop0 bin]# ./startup.sh

Using CATALINA_BASE: /opt/bigdata/tomcat7

Using CATALINA_HOME: /opt/bigdata/tomcat7

Using CATALINA_TMPDIR: /opt/bigdata/tomcat7/temp

Using JRE_HOME: /opt/jdk1.7.0_79

Using CLASSPATH: /opt/bigdata/tomcat7/bin/bootstrap.jar:/opt/bigdata/tomcat7/bin/tomcat-juli.jar

Tomcat started.

3)启动storm的nimbus 和supervisor

[root@hadoop0 bin]# cd ../../jstorm-0.9.6.2/bin/

[root@hadoop0 bin]# ./jstorm nimbus &

[1] 1985

[root@hadoop0 bin]# ./jstorm sRunning: java -server -Djstorm.home=/opt/bigdata/jstorm-0.9.6.2 -Dstorm.options= -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Xms2g -Xmx2g -Xmn768m -XX:PermSize=128m -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=20 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSFullGCsBeforeCompaction=5 -XX:+HeapDumpOnOutOfMemoryError -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=5000 -Dlogfile.name=nimbus.log -Dlog4j.configuration=File:/opt/bigdata/jstorm-0.9.6.2/conf/jstorm.log4j.properties -cp /opt/bigdata/jstorm-0.9.6.2/jstorm-client-extension-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/jstorm-client-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/jstorm-server-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/minlog-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-logging-1.1.3.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-cli-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/slf4j-api-1.7.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/libthrift7-0.7.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jline-0.9.94.jar:/opt/bigdata/jstorm-0.9.6.2/lib/objenesis-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/disruptor-3.2.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/curator-client-2.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/fastjson-1.1.41.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-exec-1.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/kryo-2.17.jar:/opt/bigdata/jstorm-0.9.6.2/lib/clj-time-0.4.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/json-simple-1.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/httpclient-4.3.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-utils-1.5.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-compiler-javac-1.8.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/zookeeper-3.4.6.jar:/opt/bigdata/jstorm-0.9.6.2/lib/joda-time-2.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/slf4j-log4j12-1.7.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/log4j-1.2.16.jar:/opt/bigdata/jstorm-0.9.6.2/lib/carbonite-1.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-lang-2.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/netty-3.9.0.Final.jar:/opt/bigdata/jstorm-0.9.6.2/lib/http-20070405.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-codec-1.6.jar:/opt/bigdata/jstorm-0.9.6.2/lib/metrics-core-3.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/asm-4.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/metrics-jvm-3.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/guava-16.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/clojure-1.5.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jgrapht-0.8.3.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-compiler-api-1.8.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jzmq-2.1.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/curator-framework-2.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-io-2.4.jar:/opt/bigdata/jstorm-0.9.6.2/lib/servlet-api-2.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/httpcore-4.3.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/reflectasm-1.07-shaded.jar:/opt/bigdata/jstorm-0.9.6.2/lib/snakeyaml-1.11.jar:/opt/bigdata/jstorm-0.9.6.2/conf: com.alibaba.jstorm.daemon.nimbus.NimbusServer

[root@hadoop0 bin]# ./jstorm supervisor &

[2] 2072

[root@hadoop0 bin]# Running: java -server -Djstorm.home=/opt/bigdata/jstorm-0.9.6.2 -Dstorm.options= -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Xms512m -Xmx512m -Xmn128m -XX:PermSize=64m -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSFullGCsBeforeCompaction=5 -XX:+HeapDumpOnOutOfMemoryError -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=5000 -Dlogfile.name=supervisor.log -Dlog4j.configuration=File:/opt/bigdata/jstorm-0.9.6.2/conf/jstorm.log4j.properties -cp /opt/bigdata/jstorm-0.9.6.2/jstorm-client-extension-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/jstorm-client-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/jstorm-server-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/minlog-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-logging-1.1.3.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-cli-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/slf4j-api-1.7.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/libthrift7-0.7.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jline-0.9.94.jar:/opt/bigdata/jstorm-0.9.6.2/lib/objenesis-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/disruptor-3.2.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/curator-client-2.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/fastjson-1.1.41.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-exec-1.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/kryo-2.17.jar:/opt/bigdata/jstorm-0.9.6.2/lib/clj-time-0.4.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/json-simple-1.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/httpclient-4.3.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-utils-1.5.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-compiler-javac-1.8.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/zookeeper-3.4.6.jar:/opt/bigdata/jstorm-0.9.6.2/lib/joda-time-2.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/slf4j-log4j12-1.7.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/log4j-1.2.16.jar:/opt/bigdata/jstorm-0.9.6.2/lib/carbonite-1.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-lang-2.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/netty-3.9.0.Final.jar:/opt/bigdata/jstorm-0.9.6.2/lib/http-20070405.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-codec-1.6.jar:/opt/bigdata/jstorm-0.9.6.2/lib/metrics-core-3.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/asm-4.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/metrics-jvm-3.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/guava-16.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/clojure-1.5.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jgrapht-0.8.3.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-compiler-api-1.8.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jzmq-2.1.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/curator-framework-2.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-io-2.4.jar:/opt/bigdata/jstorm-0.9.6.2/lib/servlet-api-2.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/httpcore-4.3.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/reflectasm-1.07-shaded.jar:/opt/bigdata/jstorm-0.9.6.2/lib/snakeyaml-1.11.jar:/opt/bigdata/jstorm-0.9.6.2/conf com.alibaba.jstorm.daemon.supervisor.Supervisor

./jstorm supervisor &kill 32721: No such process

二、开发应用程序

参见附件，storm程序不用更改任何代码直接部署使用

永久链接: http://gaojingsong.iteye.com/blog/2289928

三、部署应用

[root@hadoop0 bin]# ./jstorm jar /opt/bigdata/stormtest-0.0.1-SNAPSHOT.jar stormdemo.stormtest.App

Running: java -client -Xms256m -Xmx256m -Djstorm.home=/opt/bigdata/jstorm-0.9.6.2 -Dstorm.options= -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.jar=/opt/bigdata/stormtest-0.0.1-SNAPSHOT.jar -Dstorm.root.logger=INFO,stdout -Dlog4j.configuration=File:/opt/bigdata/jstorm-0.9.6.2/conf/aloha_log4j.properties -cp /opt/bigdata/jstorm-0.9.6.2/jstorm-client-extension-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/jstorm-client-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/jstorm-server-0.9.6.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/minlog-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-logging-1.1.3.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-cli-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/slf4j-api-1.7.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/libthrift7-0.7.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jline-0.9.94.jar:/opt/bigdata/jstorm-0.9.6.2/lib/objenesis-1.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/disruptor-3.2.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/curator-client-2.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/fastjson-1.1.41.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-exec-1.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/kryo-2.17.jar:/opt/bigdata/jstorm-0.9.6.2/lib/clj-time-0.4.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/json-simple-1.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/httpclient-4.3.2.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-utils-1.5.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-compiler-javac-1.8.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/zookeeper-3.4.6.jar:/opt/bigdata/jstorm-0.9.6.2/lib/joda-time-2.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/slf4j-log4j12-1.7.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/log4j-1.2.16.jar:/opt/bigdata/jstorm-0.9.6.2/lib/carbonite-1.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-lang-2.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/netty-3.9.0.Final.jar:/opt/bigdata/jstorm-0.9.6.2/lib/http-20070405.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-codec-1.6.jar:/opt/bigdata/jstorm-0.9.6.2/lib/metrics-core-3.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/asm-4.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/metrics-jvm-3.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/guava-16.0.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/clojure-1.5.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jgrapht-0.8.3.jar:/opt/bigdata/jstorm-0.9.6.2/lib/plexus-compiler-api-1.8.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/jzmq-2.1.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/curator-framework-2.5.0.jar:/opt/bigdata/jstorm-0.9.6.2/lib/commons-io-2.4.jar:/opt/bigdata/jstorm-0.9.6.2/lib/servlet-api-2.5.jar:/opt/bigdata/jstorm-0.9.6.2/lib/httpcore-4.3.1.jar:/opt/bigdata/jstorm-0.9.6.2/lib/reflectasm-1.07-shaded.jar:/opt/bigdata/jstorm-0.9.6.2/lib/snakeyaml-1.11.jar:/opt/bigdata/stormtest-0.0.1-SNAPSHOT.jar:/root/.jstorm:/opt/bigdata/jstorm-0.9.6.2/bin:/opt/bigdata/jstorm-0.9.6.2/conf/jstorm.log4j.properties stormdemo.stormtest.App

====Bolt==declarer========

======Spout=====declareOutputFields========

[INFO 1999-06-12 20:05:14 CuratorFrameworkImpl:238 main] Starting

[INFO 1999-06-12 20:05:15 ConnectionStateManager:194 main-EventThread] State change: CONNECTED

[INFO 1999-06-12 20:05:16 ThriftClient:62 main] masterHost:hadoop0:7627

[INFO 1999-06-12 20:05:16 ThriftClient:86 main] Begin to connect hadoop0:7627

[INFO 1999-06-12 20:05:16 CuratorFrameworkImpl:238 main] Starting

[INFO 1999-06-12 20:05:16 ConnectionStateManager:194 main-EventThread] State change: CONNECTED

[INFO 1999-06-12 20:05:17 ThriftClient:62 main] masterHost:hadoop0:7627

[INFO 1999-06-12 20:05:17 ThriftClient:86 main] Begin to connect hadoop0:7627

[INFO 1999-06-12 20:05:17 CuratorFrameworkImpl:238 main] Starting

[INFO 1999-06-12 20:05:17 ConnectionStateManager:194 main-EventThread] State change: CONNECTED

[INFO 1999-06-12 20:05:18 ThriftClient:62 main] masterHost:hadoop0:7627

[INFO 1999-06-12 20:05:18 ThriftClient:86 main] Begin to connect hadoop0:7627

[INFO 1999-06-12 20:05:18 StormSubmitter:244 main] Jar not uploaded to master yet. Submitting jar...

[INFO 1999-06-12 20:05:18 StormSubmitter:286 main] Uploading topology jar /opt/bigdata/stormtest-0.0.1-SNAPSHOT.jar to assigned location: /opt/bigdata/jstorm-0.9.6.2/data/nimbus/inbox/b44ccc23-d47d-47ac-92e8-1b260310a268/stormjar-b44ccc23-d47d-47ac-92e8-1b260310a268.jar

[INFO 1999-06-12 20:05:18 StormSubmitter:305 main] Successfully uploaded topology jar to assigned location: /opt/bigdata/jstorm-0.9.6.2/data/nimbus/inbox/b44ccc23-d47d-47ac-92e8-1b260310a268/stormjar-b44ccc23-d47d-47ac-92e8-1b260310a268.jar

[INFO 1999-06-12 20:05:18 StormSubmitter:135 main] Submitting topology simple in distributed mode with conf {"user.password":null,"user.name":null,"topology.workers":1,"topology.debug":true,"user.group":null}

[INFO 1999-06-12 20:05:18 StormSubmitter:149 main] Finished submitting topology: simple

[root@hadoop0 bin]#

四、验证