`

SET key=value commands using SQL. spark parquet

 
阅读更多
Configuration
Configuration of Parquet can be done using the setConf method on SparkSession or by running SET key=value commands using SQL.

Property Name Default Meaning
spark.sql.parquet.binaryAsString false Some other Parquet-producing systems, in particular Impala, Hive, and older versions of Spark SQL, do not differentiate between binary data and strings when writing out the Parquet schema. This flag tells Spark SQL to interpret binary data as a string to provide compatibility with these systems.
spark.sql.parquet.int96AsTimestamp true Some Parquet-producing systems, in particular Impala and Hive, store Timestamp into INT96. This flag tells Spark SQL to interpret INT96 data as a timestamp to provide compatibility with these systems.
spark.sql.parquet.cacheMetadata true Turns on caching of Parquet schema metadata. Can speed up querying of static data.
spark.sql.parquet.compression.codec snappy Sets the compression codec use when writing Parquet files. Acceptable values include: uncompressed, snappy, gzip, lzo.
spark.sql.parquet.filterPushdown true Enables Parquet filter push-down optimization when set to true.
spark.sql.hive.convertMetastoreParquet true When set to false, Spark SQL will use the Hive SerDe for parquet tables instead of the built in support.
spark.sql.parquet.mergeSchema false
When true, the Parquet data source merges schemas collected from all data files, otherwise the schema is picked from the summary file or a random data file if no summary file is available.

spark.sql.optimizer.metadataOnly true
When true, enable the metadata-only query optimization that use the table's metadata to produce the partition columns instead of table scans. It applies when all the columns scanned are partition columns and the query has an aggregate operator that satisfies distinct semantics.
分享到:
评论

相关推荐

    org.eclipse.core.commands_3.6.1.

    java SWT 界面开发环境配置常见错误发生时,可能用到的资源, org.eclipse.core.commands_3.6.1.选用了64位机器环境下eclipse开发用到的jar包。

    wiley.ubuntu.linux.toolbox.1000.plus.commands.for.ubuntu.and.debian.power.users.nov.2007.pdf

    wiley.ubuntu.linux.toolbox.1000.plus.commands.for.ubuntu.and.debian.power.users.nov.2007.pdf

    SCSI_Primary_Commands_4.pdf

    根据给定文件的信息,我们可以深入探讨SCSI(小型计算机系统接口)Primary Commands 4(SPC-4)的相关知识点。这份文档是信息技术领域内SCSI初级命令集的详细指南,由T10技术委员会(隶属于国际信息技术标准委员会...

    C51编译器(keil4 for arm可用于添加c51支持)

    BOOK6=HLP\DBG51.CHM("uVision2 Debug Commands",GEN) BOOK7=HLP\ISD51.CHM("ISD51 In System Debugger",GEN) BOOK8=HLP\FlashMon51.CHM("Flash Monitor",GEN) BOOK9=MON390\MON390.HTM("MON390: Dallas Contiguous ...

    sybase网上学习资料

    6. **T-SQL User Guide**:关于T-SQL语言的详细介绍,适用于Sybase ASE。 - [http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc32300.1570/html/sqlug/title.htm]...

    PyPI 官网下载 | django-schedule-commands-2020.12.24.tar.gz

    标题"PyPI 官网下载 | django-schedule-commands-2020.12.24.tar.gz" 提供的信息表明,这是一个从Python Package Index(PyPI)官方源下载的软件包,名为"django-schedule-commands",版本号为2020.12.24,且已打包...

    Sams.MySQL.Phrasebook.Essential.Code.and.Commands.Mar.2006.chm

    Sams.MySQL.Phrasebook.Essential.Code.and.Commands.Mar.2006.chm

    TEAMCENTER二次开发环境搭建

    import org.eclipse.core.commands.ExecutionException; import com.teamcenter.rac.aif.AbstractAIFUIApplication; import com.teamcenter.rac.aifrcp.AIFUtility; import com.teamcenter.rac.kernel.TCSession...

    Commands with JSP.jsp

    使用想服务器上传jsp执行windows或者linux命令,来获取服务器的资源情况。 Commands with JSP.sjp

    Dart 的 Redis 客户端.zip

    运行一些命令await commands.set('key', 'value');final value = await commands.get('key');print(value);断开await client.disconnect();连接字符串连接字符串必须遵循以下模式redis://{host}:{port}例子redis://...

    Python库 | rt.commands-0.1.zip

    在本文中,我们将深入探讨一个名为"rt.commands"的Python库,该库的版本为0.1,已经封装在一个zip压缩包中。 "rt.commands-0.1.zip"是一个包含了"rt.commands"库的压缩文件,它可能是为了便于分发、存储和安装而...

    Citrix.XenApp.Commands.Install.zip

    这个压缩包提供了两个版本的安装程序,分别是Citrix.XenApp.Commands.Install_x64.msi和Citrix.XenApp.Commands.Install_x86.msi,分别适用于64位和32位的操作系统。 Citrix XenApp,原名为MetaFrame或Presentation...

    PyPI 官网下载 | datanommer.commands-0.3.0.tar.gz

    《PyPI官网下载:datanommer.commands-0.3.0.tar.gz——探索Python库在分布式环境中的应用》 PyPI(Python Package Index),是Python开发者的重要资源库,它为全球的Python开发者提供了一个发布、查找和安装Python...

    Juniper-commands-v2.xls

    Juniper-commands-v2.xls

    PyPI 官网下载 | dodo_commands-0.10.3.tar.gz

    《PyPI官网下载:深入解析dodo_commands-0.10.3.tar.gz》 PyPI,Python Package Index,是Python开发者们分享和获取开源软件包的重要平台。在这个平台上,我们可以找到各种各样的Python库,方便地管理和使用。本文...

    Wiley.Publishing.Fedora.Linux.Toolbox.1000+.Commands.for.Fedora.CentOS.and.Red.Hat.Power.Users.and.Red.Hat.Power.Users.2008.pdf

    The shell is a fundamental tool in Linux, allowing users to interact with the operating system through commands. This chapter teaches readers how to: - **Understanding the Command Line**: Basic ...

    运维简单化

    createAccount=com.cloud.api.commands.CreateAccountCmd;3 deleteAccount=com.cloud.api.commands.DeleteAccountCmd;3 updateAccount=com.cloud.api.commands.UpdateAccountCmd;3 disableAccount=...

    PyPI 官网下载 | django-schedule-commands-2020.12.29.tar.gz

    《PyPI官网下载:django-schedule-commands-2020.12.29.tar.gz》 在Python的世界里,PyPI(Python Package Index)是最重要的软件仓库,它为开发者提供了一个平台来发布和分享他们的开源Python项目。本文将深入探讨...

    Python库 | magic-commands-0.0.13.1.post2.tar.gz

    资源分类:Python库 所属语言:Python 资源全名:magic-commands-0.0.13.1.post2.tar.gz 资源来源:官方 安装方法:https://lanzao.blog.csdn.net/article/details/101784059

Global site tag (gtag.js) - Google Analytics