idea打包sbt项目

转载：https://blog.csdn.net/coder__cs/article/details/79344839

 

前提条件是创建好了wordcount项目，可以参考Scala官方IDE教程

 

Getting Started with Scala in IntelliJ

Building a Scala Project with IntelliJ and sbt

Spark Quick Start https://spark.apache.org/docs/latest/quick-start.html#self-contained-applications

 

我们需要做的是打包项目为jar包并发布到集群中去运行，下面逐一讲解打包步骤。 

第一步 打开项目结构配置页面 可以使用快捷键 Ctrl+Alt+Shift+S 

 

第二步 添加jar包配置 

 

第三步 去除额外的lib包依赖，不将其他依赖打包到jar文件中，只保留class编译文件及META-INF文件夹 

 

 

第四步 编译构建 生成jar包 

 

 

 

 

 

● jar包解压后内部结构 

 

第五步 最后将得到的jar包，可上传到集群中执行

 

sftp> put wordcount.jar

Uploading wordcount.jar to /home/elon/workspace/wordcount/wordcount.jar

  100% 5KB      5KB/s 00:00:00     

C:/Users/yilon/Documents/wordcount.jar: 5452 bytes transferred in 0 seconds (5452 bytes/s)

1

2

3

4

 

[elon@hadoop spark]$ ./bin/spark-submit --class WordCount --master local ~/workspace/wordcount/wordcount.jar 

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

18/02/22 00:38:59 INFO SparkContext: Running Spark version 2.2.1

18/02/22 00:39:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

18/02/22 00:39:01 INFO SparkContext: Submitted application: WordCount

18/02/22 00:39:01 INFO SecurityManager: Changing view acls to: elon

18/02/22 00:39:01 INFO SecurityManager: Changing modify acls to: elon

18/02/22 00:39:01 INFO SecurityManager: Changing view acls groups to: 

18/02/22 00:39:01 INFO SecurityManager: Changing modify acls groups to: 

18/02/22 00:39:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(elon); groups with view permissions: Set(); users  with modify permissions: Set(elon); groups with modify permissions: Set()

18/02/22 00:39:02 INFO Utils: Successfully started service 'sparkDriver' on port 44048.

18/02/22 00:39:02 INFO SparkEnv: Registering MapOutputTracker

18/02/22 00:39:02 INFO SparkEnv: Registering BlockManagerMaster

18/02/22 00:39:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information

18/02/22 00:39:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up

18/02/22 00:39:02 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-df8c9e80-53ba-42e5-98f9-6010211bac1c

18/02/22 00:39:02 INFO MemoryStore: MemoryStore started with capacity 413.9 MB

18/02/22 00:39:03 INFO SparkEnv: Registering OutputCommitCoordinator

18/02/22 00:39:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.

18/02/22 00:39:03 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.115:4040

18/02/22 00:39:04 INFO SparkContext: Added JAR file:/home/elon/workspace/wordcount/wordcount.jar at spark://192.168.1.115:44048/jars/wordcount.jar with timestamp 1519231144007

18/02/22 00:39:04 INFO Executor: Starting executor ID driver on host localhost

18/02/22 00:39:04 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 32785.

18/02/22 00:39:04 INFO NettyBlockTransferService: Server created on 192.168.1.115:32785

18/02/22 00:39:04 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy

18/02/22 00:39:04 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.115, 32785, None)

18/02/22 00:39:04 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.115:32785 with 413.9 MB RAM, BlockManagerId(driver, 192.168.1.115, 32785, None)

18/02/22 00:39:04 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.115, 32785, None)

18/02/22 00:39:04 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.115, 32785, None)

wordCounts: 

(package,1)

(For,3)

(Programs,1)

(processing.,1)

(Because,1)

(The,1)

(page](http://spark.apache.org/documentation.html).,1)

(cluster.,1)

(its,1)

([run,1)

(than,1)

(APIs,1)

(have,1)

(Try,1)

(computation,1)

(through,1)

(several,1)

(This,2)

(graph,1)

(Hive,2)

(storage,1)

(["Specifying,1)

(To,2)

("yarn",1)

(Once,1)

(["Useful,1)

(prefer,1)

(SparkPi,2)

(engine,1)

(version,1)

(file,1)

(documentation,,1)

(processing,,1)

(the,24)

(are,1)

(systems.,1)

(params,1)

(not,1)

(different,1)

(refer,2)

(Interactive,2)

(R,,1)

(given.,1)

(if,4)

(build,4)

(when,1)

(be,2)

(Tests,1)

(Apache,1)

(thread,1)

(programs,,1)

(including,4)

(./bin/run-example,2)

(Spark.,1)

(package.,1)

(1000).count(),1)

(Versions,1)

(HDFS,1)

(Data.,1)

(>>>,1)

(Maven,1)

(programming,1)

(Testing,1)

(module,,1)

(Streaming,1)

(environment,1)

(run:,1)

(Developer,1)

(clean,1)

(1000:,2)

(rich,1)

(GraphX,1)

(Please,4)

(is,6)

(guide](http://spark.apache.org/contributing.html),1)

(run,7)

(URL,,1)

(threads.,1)

(same,1)

(MASTER=spark://host:7077,1)

(on,7)

(built,1)

(against,1)

([Apache,1)

(tests,2)

(examples,2)

(at,2)

(optimized,1)

(3"](https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3).,1)

(usage,1)

(development,1)

(Maven,,1)

(graphs,1)

(talk,1)

(Shell,2)

(class,2)

(abbreviated,1)

(using,5)

(directory.,1)

(README,1)

(computing,1)

(overview,1)

(`examples`,2)

(example:,1)

(##,9)

(N,1)

(set,2)

(use,3)

(Hadoop-supported,1)

(running,1)

(find,1)

(contains,1)

(project,1)

(Pi,1)

(need,1)

(or,3)

(Big,1)

(high-level,1)

(Java,,1)

(uses,1)

(<class>,1)

(Hadoop,,2)

(available,1)

(requires,1)

((You,1)

(more,1)

(see,3)

(Documentation,1)

(of,5)

(tools,1)

(using:,1)

(cluster,2)

(must,1)

(supports,2)

(built,,1)

(tests](http://spark.apache.org/developer-tools.html#individual-tests).,1)

(system,1)

(build/mvn,1)

(Hadoop,3)

(this,1)

(Version"](http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version),1)

(particular,2)

(Python,2)

(Spark,16)

(general,3)

(YARN,,1)

(pre-built,1)

([Configuration,1)

(locally,2)

(library,1)

(A,1)

(locally.,1)

(sc.parallelize(1,1)

(only,1)

(Configuration,1)

(following,2)

(basic,1)

(#,1)

(changed,1)

(More,1)

(which,2)

(learning,,1)

(first,1)

(./bin/pyspark,1)

(also,4)

(info,1)

(should,2)

(for,12)

([params]`.,1)

(documentation,3)

([project,1)

(mesos://,1)

(Maven](http://maven.apache.org/).,1)

(setup,1)

(<http://spark.apache.org/>,1)

(latest,1)

(your,1)

(MASTER,1)

(example,3)

(["Parallel,1)

(scala>,1)

(DataFrames,,1)

(provides,1)

(configure,1)

(distributions.,1)

(can,7)

(About,1)

(instructions.,1)

(do,2)

(easiest,1)

(no,1)

(project.,1)

(how,3)

(`./bin/run-example,1)

(started,1)

(Note,1)

(by,1)

(individual,1)

(spark://,1)

(It,2)

(tips,,1)

(Scala,2)

(Alternatively,,1)

(an,4)

(variable,1)

(submit,1)

(-T,1)

(machine,1)

(thread,,1)

(them,,1)

(detailed,2)

(stream,1)

(And,1)

(distribution,1)

(review,1)

(return,2)

(Thriftserver,1)

(developing,1)

(./bin/spark-shell,1)

("local",1)

(start,1)

(You,4)

(Spark](#building-spark).,1)

(one,3)

(help,1)

(with,4)

(print,1)

(Spark"](http://spark.apache.org/docs/latest/building-spark.html).,1)

(data,1)

(Contributing,1)

(in,6)

(-DskipTests,1)

(downloaded,1)

(versions,1)

(online,1)

(Guide](http://spark.apache.org/docs/latest/configuration.html),1)

(builds,1)

(comes,1)

(Tools"](http://spark.apache.org/developer-tools.html).,1)

([building,1)

(Python,,2)

(Many,1)

(building,2)

(Running,1)

(from,1)

(way,1)

(Online,1)

(site,,1)

(other,1)

(Example,1)

([Contribution,1)

(analysis.,1)

(sc.parallelize(range(1000)).count(),1)

(you,4)

(runs.,1)

(Building,1)

(higher-level,1)

(protocols,1)

(guidance,2)

(a,8)

(guide,,1)

(name,1)

(fast,1)

(SQL,2)

(that,2)

(will,1)

(IDE,,1)

(to,17)

(get,1)

(,71)

(information,1)

(core,1)

(web,1)

("local[N]",1)

(programs,2)

(option,1)

(MLlib,1)

(["Building,1)

(contributing,1)

(shell:,2)

(instance:,1)

(Scala,,1)

(and,9)

(command,,2)

(package.),1)

(./dev/run-tests,1)

(sample,1)

--------------------- 

作者：Coder__CS 

来源：CSDN 

原文：https://blog.csdn.net/coder__cs/article/details/79344839 

版权声明：本文为博主原创文章，转载请附上博文链接！