`

spark standalone结构

 
阅读更多

 

 

1 standalone架构下的角色介绍：
Master:负责任务分发；负责分配资源； 老大
Worker:运行应用代码的节点,接收Master请求，启动任务；向Master汇报信息，资源信息和健康状况。 项目经理
Executor:worker node的一个进程,接收worker的任务执行请求，执行任务；向worker汇报任务执行情况。 工程师

<!--5f39ae17-8c62-4a45-bc43-b32064c9388a:W3siYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImxpbmUtaGVpZ2h0IjoxLjg3NSwiYWxpZ24iOiJsZWZ0IiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MH0sImJsb2NrSWQiOiJhZDcyYjUxYS1jMTMzLWExZWUtMTNhMS0wMWI3NzEwNDE5NDgiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbeyJjaGFyIjoicyJ9LHsiY2hhciI6InQifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoibiJ9LHsiY2hhciI6ImQifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoibCJ9LHsiY2hhciI6Im8ifSx7ImNoYXIiOiJuIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6IuaetiJ9LHsiY2hhciI6IuaehCJ9XX19LHsiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImxpbmUtaGVpZ2h0IjoxLjg3NSwiYWxpZ24iOiJsZWZ0IiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MH0sImJsb2NrSWQiOiJkNjYyNWMzMy00MTMzLWExZWUtMTNhMS03NThhMjhhZmRmMDMiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbeyJjaGFyIjoiTSJ9LHsiY2hhciI6ImEifSx7ImNoYXIiOiJzIn0seyJjaGFyIjoidCJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiOiJ9LHsiY2hhciI6Iui0nyJ9LHsiY2hhciI6Iui0oyJ9LHsiY2hhciI6IuS7uyJ9LHsiY2hhciI6IuWKoSJ9LHsiY2hhciI6IuWIhiJ9LHsiY2hhciI6IuWPkSJ9LHsiY2hhciI6Iu+8myJ9LHsiY2hhciI6Iui0nyJ9LHsiY2hhciI6Iui0oyJ9LHsiY2hhciI6IuWIhiJ9LHsiY2hhciI6IumFjSJ9LHsiY2hhciI6Iui1hCJ9LHsiY2hhciI6Iua6kCJ9LHsiY2hhciI6Iu+8myJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIiwic3R5bGVzIjp7ImNvbG9yIjoiI0ZGMDAwMCJ9fSx7ImNoYXIiOiLogIEiLCJzdHlsZXMiOnsiY29sb3IiOiIjRkYwMDAwIn19LHsiY2hhciI6IuWkpyIsInN0eWxlcyI6eyJjb2xvciI6IiNGRjAwMDAifX1dfX0seyJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsibGluZS1oZWlnaHQiOjEuODc1LCJhbGlnbiI6ImxlZnQiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowfSwiYmxvY2tJZCI6ImI0Yjc0ODYwLTYxMzMtYTFlZS0xM2ExLTc1ODMyNDFmNjhlOSIsInJpY2hUZXh0Ijp7ImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWUsImRhdGEiOlt7ImNoYXIiOiJXIn0seyJjaGFyIjoibyJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiJrIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiI6In0seyJjaGFyIjoi6L+QIn0seyJjaGFyIjoi6KGMIn0seyJjaGFyIjoi5bqUIn0seyJjaGFyIjoi55SoIn0seyJjaGFyIjoi5LujIn0seyJjaGFyIjoi56CBIn0seyJjaGFyIjoi55qEIn0seyJjaGFyIjoi6IqCIn0seyJjaGFyIjoi54K5In0seyJjaGFyIjoiLCJ9LHsiY2hhciI6IuaOpSJ9LHsiY2hhciI6IuaUtiJ9LHsiY2hhciI6Ik0ifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6InQifSx7ImNoYXIiOiJlIn0seyJjaGFyIjoiciJ9LHsiY2hhciI6IuivtyJ9LHsiY2hhciI6IuaxgiJ9LHsiY2hhciI6Iu+8jCJ9LHsiY2hhciI6IuWQryJ9LHsiY2hhciI6IuWKqCJ9LHsiY2hhciI6IuS7uyJ9LHsiY2hhciI6IuWKoSJ9LHsiY2hhciI6Iu+8myJ9LHsiY2hhciI6IuWQkSJ9LHsiY2hhciI6Ik0ifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6InQifSx7ImNoYXIiOiJlIn0seyJjaGFyIjoiciJ9LHsiY2hhciI6IuaxhyJ9LHsiY2hhciI6IuaKpSJ9LHsiY2hhciI6IuS/oSJ9LHsiY2hhciI6IuaBryJ9LHsiY2hhciI6Iu+8jCJ9LHsiY2hhciI6Iui1hCJ9LHsiY2hhciI6Iua6kCJ9LHsiY2hhciI6IuS/oSJ9LHsiY2hhciI6IuaBryJ9LHsiY2hhciI6IuWSjCJ9LHsiY2hhciI6IuWBpSJ9LHsiY2hhciI6IuW6tyJ9LHsiY2hhciI6IueKtiJ9LHsiY2hhciI6IuWGtSJ9LHsiY2hhciI6IuOAgiJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiLpobkiLCJzdHlsZXMiOnsiY29sb3IiOiIjRkYwMDAwIn19LHsiY2hhciI6IuebriIsInN0eWxlcyI6eyJjb2xvciI6IiNGRjAwMDAifX0seyJjaGFyIjoi57uPIiwic3R5bGVzIjp7ImNvbG9yIjoiI0ZGMDAwMCJ9fSx7ImNoYXIiOiLnkIYiLCJzdHlsZXMiOnsiY29sb3IiOiIjRkYwMDAwIn19XX19LHsiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImxpbmUtaGVpZ2h0IjoxLjg3NSwiYWxpZ24iOiJsZWZ0IiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MH0sImJsb2NrSWQiOiJvZjc2ZTY0MS03MTMzLWExZWUtMTNhMS03NTg1YzIyMzI5MWYiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbeyJjaGFyIjoiRSJ9LHsiY2hhciI6IngifSx7ImNoYXIiOiJlIn0seyJjaGFyIjoiYyJ9LHsiY2hhciI6InUifSx7ImNoYXIiOiJ0In0seyJjaGFyIjoibyJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiI6In0seyJjaGFyIjoidyJ9LHsiY2hhciI6Im8ifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiayJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6Im4ifSx7ImNoYXIiOiJvIn0seyJjaGFyIjoiZCJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiLnmoQifSx7ImNoYXIiOiLkuIAifSx7ImNoYXIiOiLkuKoifSx7ImNoYXIiOiLov5sifSx7ImNoYXIiOiLnqIsifSx7ImNoYXIiOiIsIn0seyJjaGFyIjoi5o6lIn0seyJjaGFyIjoi5pS2In0seyJjaGFyIjoidyJ9LHsiY2hhciI6Im8ifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiayJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoi55qEIn0seyJjaGFyIjoi5Lu7In0seyJjaGFyIjoi5YqhIn0seyJjaGFyIjoi5omnIn0seyJjaGFyIjoi6KGMIn0seyJjaGFyIjoi6K+3In0seyJjaGFyIjoi5rGCIn0seyJjaGFyIjoi77yMIn0seyJjaGFyIjoi5omnIn0seyJjaGFyIjoi6KGMIn0seyJjaGFyIjoi5Lu7In0seyJjaGFyIjoi5YqhIn0seyJjaGFyIjoi77ybIn0seyJjaGFyIjoi5ZCRIn0seyJjaGFyIjoidyJ9LHsiY2hhciI6Im8ifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiayJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoi5rGHIn0seyJjaGFyIjoi5oqlIn0seyJjaGFyIjoi5Lu7In0seyJjaGFyIjoi5YqhIn0seyJjaGFyIjoi5omnIn0seyJjaGFyIjoi6KGMIn0seyJjaGFyIjoi5oOFIn0seyJjaGFyIjoi5Ya1In0seyJjaGFyIjoi44CCIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IuW3pSIsInN0eWxlcyI6eyJjb2xvciI6IiNGRjAwMDAifX0seyJjaGFyIjoi56iLIiwic3R5bGVzIjp7ImNvbG9yIjoiI0ZGMDAwMCJ9fSx7ImNoYXIiOiLluIgiLCJzdHlsZXMiOnsiY29sb3IiOiIjRkYwMDAwIn19LHsiY2hhciI6IiAiLCJzdHlsZXMiOnsiY29sb3IiOiIjRkYwMDAwIn19LHsiY2hhciI6IiAiLCJzdHlsZXMiOnsiY29sb3IiOiIjRkYwMDAwIn19XX19LHsiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImxpbmUtaGVpZ2h0IjoxLjg3NSwiYWxpZ24iOiJsZWZ0IiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MH0sImJsb2NrSWQiOiJoZGVjZjNhNC01MTMzLWExZWUtMTNhMS03NThlNmI0OWUwMjAiLCJyaWNoVGV4dCI6eyJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlLCJkYXRhIjpbeyJjaGFyIjoiRCJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiJpIn0seyJjaGFyIjoidiJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiOiJ9LHsiY2hhciI6IuaPkCJ9LHsiY2hhciI6IuS6pCJ9LHsiY2hhciI6IuS7uyJ9LHsiY2hhciI6IuWKoSJ9LHsiY2hhciI6Iu+8myJ9LHsiY2hhciI6IuebkSJ9LHsiY2hhciI6IuaOpyJ9LHsiY2hhciI6IuS7uyJ9LHsiY2hhciI6IuWKoSJ9LHsiY2hhciI6Iu+8myJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IuWuoiIsInN0eWxlcyI6eyJjb2xvciI6IiNGRjAwMDAifX0seyJjaGFyIjoi5oi3Iiwic3R5bGVzIjp7ImNvbG9yIjoiI0ZGMDAwMCJ9fV19fV0=-->

Driver:提交任务；监控任务； 客户


 
 
 
  • 大小: 49.1 KB
分享到：
| spark笔记
评论
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

    Spark机器学习案例实战.pdf

    同时，文件还介绍了如何在Spark环境配置、使用spark-shell进行交互式编程，以及如何利用Spark Standalone模式运行Spark应用。此外，还展示了如何在Spark中导入CSV数据，这是处理实际数据集的常见操作。整个文档强调...

    spark全套视频教程

    2. **Spark安装与配置**：学习如何在不同的操作系统上安装Spark，并配置分布式环境，如在本地模式、standalone模式、YARN或Mesos集群上运行Spark。 3. **Spark Shell与Spark SQL**：Spark Shell是交互式探索数据的...

    Spark简介以及其生态圈

    其生态圈还包含一系列处理不同数据源和数据类型的应用程序库，如Spark SQL用于处理结构化数据，Spark Streaming用于实时数据流处理，MLlib用于机器学习，以及GraphX用于图形处理。 Spark运行架构由几个核心组件构成...

    Spark The Definitive Guide-201712

    2. **Spark SQL**：Spark SQL是Spark处理结构化数据的主要模块，它允许用户通过SQL或DataFrame API进行查询，同时兼容Hive metastore，使得Hadoop生态中的数据可以直接在Spark中使用。 3. **Spark Streaming**：...

    Spark生态圈介绍

    4. 随处运行：Spark 具有很强的适应性，能够读取 HDFS、Cassandra、HBase、S3 和 Techyon 等持久层读写原生数据，能够以 Mesos、YARN 和自带的 Standalone 作为资源管理器调度 job，来完成 Spark 应用程序的计算。...

    Spark开发指南

    在Spark的源码分析方面，本指南提供了对Spark RDD分析、Spark持久化分析、Spark DAG调度分析、Spark Standalone Master分析和Spark Standalone Worker分析等更深层次的内容，让读者能够深入理解Spark的运行机制和...

    spark-2.0.1集群安装及编写例子提交任务

    2. **Spark SQL**：使用DataFrame API进行结构化数据处理，可以与多种数据源交互，如HDFS、Cassandra、Hive等。 3. **Spark Streaming**：处理实时数据流，例如从Kafka或TCP套接字读取数据，并进行实时分析。 4. *...

    Spark-Core学习知识笔记整理

    Spark-Core文档是本人经三年总结笔记汇总而来，对于自我学习Spark核心基础知识非常方便，资料中例举完善，内容丰富。具体目录如下： ...2.1 Spark on Standalone运行过程 59 2.2 Spark on YARN运行过程 60

    ApacheSpark设计与实现.pdf+ApacheSpark源码剖析.pdf+Spark原著中文版.pdf

    1. **源码结构**：Spark源代码的主要模块和包划分，以及关键类的职责。 2. **调度机制**：DAGScheduler和TaskScheduler的工作流程，如何调度任务到Executor。 3. **存储系统**：BlockManager如何管理内存和磁盘上的...

    spark-3.2.1 安装包 集成 hadoop2.7

    5. 使用`spark-submit`脚本提交Spark作业到YARN上运行，或者在本地模式或standalone模式下启动Spark Shell进行交互式测试。 Spark的使用场景广泛，涵盖了数据批处理、实时流处理、机器学习和图计算等。在大数据领域...

    spark-3.2.1 安装包 下载 hadoop3.2

    Spark不仅仅是一个批处理引擎，它还包括Spark Streaming用于实时流处理，MLlib库用于机器学习，GraphX用于图计算，以及Spark SQL用于结构化数据处理。这些组件共同构建了一个全面的大数据处理平台，覆盖了从数据摄入...

    Spark技术内幕：深入解析Spark内核架构设计与实现原理

    3. **Cluster Manager**: 负责资源调度，如YARN、Mesos或独立的Spark Standalone模式。 4. **Executor**: 在每个工作节点上运行，执行任务并缓存数据，提供内存和磁盘存储。 **DAG执行模型** Spark通过DAG执行模型...

    Spark源码剖析

    《Spark源码剖析》PDF 文件很可能会深入到这些技术细节，包括类结构、算法实现以及关键代码的解析，帮助读者更好地理解和优化 Spark 应用。通过深入学习 Spark 源码，开发者可以更好地掌握 Spark 内部工作原理，从而...

    Spark安装包、安装文档

    2. **Spark SQL**：用于结构化和半结构化数据处理，与SQL兼容，可以与Hive等传统SQL工具集成。 3. **Spark Streaming**：处理连续的数据流，支持微批处理，适用于实时数据处理。 4. **MLlib**：机器学习库，提供多种...

    Spark The Definitive Guide, 1st Edition

    6. **Spark部署和优化**：书中还详细讲解了如何在不同的环境（如Standalone、YARN、Mesos）下部署Spark，以及如何配置和优化Spark应用程序以获得最佳性能。这包括内存管理、任务调度、数据分区策略等。 7. **案例...

    46488-Spark大数据技术与应用(1-3).pdf

    3. Spark SQL和DataFrame：介绍如何使用Spark SQL进行结构化数据处理，DataFrame API的使用和好处。 4. Spark生态系统：探讨Spark的生态系统组件，如Spark Streaming、MLlib（机器学习库）和GraphX（图计算）。 5. ...

    Spark官方文档中文翻译

    Spark SQL是Spark用于处理结构化数据的模块，它可以与Hive、HDFS、Cassandra等外部数据源集成，并提供DataFrame和Dataset API，使得SQL查询和DataFrame/Dataset操作无缝结合。 **4. Spark Streaming** Spark ...

    spark-2.3.0-bin-hadoop2.7版本.zip

    3. **多模式支持**：Spark支持多种工作模式，包括本地模式、Standalone集群模式、YARN模式（与Hadoop2.7集成）以及Mesos模式，适应不同的部署环境。 4. **组件丰富**：Spark包含多个模块，如Spark Core、Spark SQL...

    Apache Spark 2.0.2 中文文档 - v0.1.0

    9. **部署和配置**：文档会介绍如何设置和配置Spark集群，包括本地模式、standalone模式、Mesos和YARN等集群管理器的配置。 10. **性能优化**：优化指南可能包含如何调整内存设置、并行度、数据序列化等策略，以...

    spark 2.6.3

    9. **部署和扩展性**：Spark支持standalone模式、Mesos、YARN和Kubernetes等多种部署方式，易于扩展和管理。 10. **编程语言支持**：Spark提供了Java、Scala、Python和R的API，满足不同开发者的需求。 在Linux环境...

Magicbox
Global site tag (gtag.js) - Google Analytics