similar to other open source projects,spark has several shells are listed there
sbin | server side shells | ||
start-all.sh | start the whole spark daemons | (ie. start-master.sh,start-slaves.sh) | |
start-master.sh | startup the spark's master process | deliver to "spark-daemon.sh start master.Master" | |
start-slaves.sh | starts all workers | deliver to sbin/slaves.sh | |
spark-daemon.sh | spawn up any daemons,e.g. spark-class.sh,spark-submit.sh.usage: spark-daemon.sh [--config <conf-dir>] (start|stop|status) <spark-command> <spark-instance-number> <args...> |
for start-master.sh,here will deliver to bin/spark-class.sh | |
slaves.sh | login into all slaves dfined in conf/slaves,then issue sbin/start-slave.sh | ||
spark-config.sh | export some global variables,e.g. SPARK_HOME | ||
start-slave.sh | deliver to spark-daemon.sh start deploy.worker.Worker |
some settings,eg. SPARK_WORKER_INSTANCES |
|
bin | |||
load-spark-env.sh | load the file spark-env.sh if exists | ||
spark-class.sh[end] |
finally,spawn up a daemon(class) returned by executing "org.apache.spark.launcher.Main",the class is specified by caller.(eg. spark-daemon.sh) Usage: spark-class <class> [<args>] |
||
spark-shell |
an interactive interface with spark to test,demo,committing spark app. Help msg is grapped from spark-submit.it uses athird-party jar named Jline to simulate a shell style. |
it will deliver to spark-submit shell.ie. bin/spark-submit --class org.apache.spark.repl.Main |
|
spark-submit | submit a spark app |
differs with start-master.sh and start-slave.sh,here will issue new class: "spark-class org.apache.spark.deploy.SparkSubmit .." |
|
run-example | runs the examples by given a subfix class name. |
deliver to "spark-submit .." |
|
conf | |||
spark-env.sh |
misc spark cluster settings corresponding to cluster manager,e.g. SPARK_LOCAL_IP, SPARK_CLASSPATH,SPARK_MASTER_IP.. |
eg.
hadoop@GZsw04:~$ spark-daemon.sh status org.apache.spark.deploy.master.Master 1 org.apache.spark.deploy.master.Master is running.
launch.Main.java
the union entry of issuing any class/daemon,e.g.if u wanna submit a workcount app,u can do like this:
run-example JavaWordCount /file/to/count
then the concrete class named "org.apache.spark.examples.JavaWordCount" is executed by inflecting by Main.java.also,u can dig into the src of JavaWordCount for more details.
conclusions;
a.complete flow of startup master
[start-all.sh]>start-master.sh>spark-daemon.sh>spark-class.sh > launch.Main>issue app master.Master
and the slaves(workers)
[start-all.sh]>start-slaves.sh>slaves.sh>start-slave.sh>spark-daemon.sh[ worker.Worker]>same steps with above
b.it's different from hadoop,hbase.since it will supply a union entry of launch.Main to fix several params,or coming them.
c.advances step by step will reduce the reduplicate code,improves the reusability
相关推荐
da---- 2020/3/31 13:39 2-3OSPF Adjacency Relationships and Various Types of LSAs da---- 2020/3/31 13:39 2-4OSPF Stub Area and NSSA da---- 2020/3/31 13:39 2-5OSPF VCs and Inter-Area Route Filtering da-...
在Laravel框架中,"bw-magic-relationships"可能是一个关于模型关系的教程或项目,旨在深入探讨如何在Laravel中高效地管理数据库之间的关联。在这个主题中,我们将重点讨论Laravel中的各种关系类型,以及如何利用...
Peer-to-Peer Network Relationships Client/ Server Network Relationships Comparing Peer-to-Peer and Client/ Server Networks Learning Network Features File Sharing Printer Sharing Application ...
在Laravel框架中,"Laravel开发-laravel-request-relationships"是一个重要的概念,它涉及到模型(Model)和请求(Request)之间的关系管理。Laravel是一个优雅的PHP Web应用开发框架,它提供了许多便利的功能,其中...
$ git clone https://github.com/DanielArturoAlejoAlvarez/Eloquent-ORM-Laravel-8.5-Relationships-Advanced[NAME APP] $ composer install $ copy .env.example .env $ php artisan key:generate $ ...
- `--relationships ../relation.csv`: 导入关系文件,若文件中包含关系类型,则无需再次指定。 ##### 2. Linux环境 - **停止Neo4j数据库服务**: - 使用`bin/neo4j stop`命令来停止服务。 - **检查并删除目标...
在Laravel框架中,"Relationships"是数据库交互的核心部分,它允许开发者通过ORM(对象关系映射)轻松处理复杂的数据库操作。Laravel的关系模型提供了许多预定义的关系类型,如一对一、一对多、多对多等,使得在...
`./neo4j-admin import --database <db_name> --mode <mode> --report-file <report_file> --nodes <node_files> --relationships <relationship_files> --id-type <id_type> --input-encoding <encoding>` ...
atlas-relationships-api-0.9.0-sources.jar
atlas-relationships-api-0.8.0-sources.jar
atlas-relationships-api-0.8.1-sources.jar
atlas-relationships-api-0.7.0-sources.jar
atlas-relationships-api-0.9.0.jar
atlas-relationships-api-0.8.0.jar
Contents Page 0 Introduction................................6 Source, coded, decoded and output data formats, scanning processes, and neighbouring relationships.......... 14 6.1 Bitstream formats...........
Lesson 24: A REAL-TIME APPLICATION OF MULTIPLE WAVE RELATIONSHIPS - 100 - Lesson 25: FIBONACCI TIME SEQUENCES - 105 - Lesson 26: LONG TERM WAVES - 111 - Lesson 27: THE WAVE PATTERN UP TO 1978 - 114 - ...
Graph Algorithms: Practical Examples in Apache Spark and Neo4j By 作者: Mark Needham – Amy E. Hodler ISBN-10 书号: 1492047686 ISBN-13 书号: 9781492047681 Edition 版本: 1 出版日期: 2019-01-04 pages ...
ACL 2015论文数据集 该存储库包含Nikos Voskarides,Edgar Meij,Manos Tsagkias,Maarten de Rijke和Wouter Weerkamp撰写的ACL 2015论文“学习以知识图形式说明实体关系”的数据集。 本文提出了一种自动解释实体...
- **配置一个新的组织关系**(正确答案B): - 通过**Exchange Admin Center (EAC)**创建一个组织关系,以允许两个组织之间的日历共享。 - 在**EAC**中选择“共享”->“组织关系”,然后添加一个新的组织关系,...
E.2.1 Example 1 Matrix with predefined values 52 E.2.3 Example 3 Assessing a value for the likelihood and the possible consequences of risks... 54 Annex F(informative) Constraints for risk ...