`
wbj0110
  • 浏览: 1604709 次
  • 性别: Icon_minigender_1
  • 来自: 上海
文章分类
社区版块
存档分类
最新评论

Cloudera Impala: Real-Time Queries in Apache Hadoop, For Real

阅读更多

After a long period of intense engineering effort and user feedback, we are very pleased, and proud, to announce the Cloudera Impala project. This technology is a revolutionary one for Hadoop users, and we do not take that claim lightly.

When Google published its Dremel paper in 2010, we were as inspired as the rest of the community by the technical vision to bring real-time, ad hoc query capability to Apache Hadoop, complementing traditional MapReduce batch processing. Today, we are announcing a fully functional, open-sourced codebase that delivers on that vision – and, we believe, a bit more – which we call Cloudera Impala. An Impala binary is now available in public beta form, but if you would prefer to test-drive Impala via a pre-baked VM, we have one of those for you, too. (Links to all downloads and documentation are here.) You can also review the source code and testing harness at Github right now.

Impala raises the bar for query performance while retaining a familiar user experience. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time. Furthermore, it uses the same metadata, SQL syntax (Hive SQL), ODBC driver and user interface (Hue Beeswax) as Apache Hive, providing a familiar and unified platform for batch-oriented or real-time queries. (For that reason, Hive users can utilize Impala with little setup overhead.) The first beta drop includes support for text files and SequenceFiles; SequenceFiles can be compressed as Snappy, GZIP, and BZIP (with Snappy recommended for maximum performance). Support for additional formats including Avro, RCFile, LZO text files, and the Parquet columnar format is planned for the production drop.

To avoid latency, Impala circumvents MapReduce to directly access the data through a specialized distributed query engine that is very similar to those found in commercial parallel RDBMSs. The result is order-of-magnitude faster performance than Hive, depending on the type of query and configuration. (See FAQ below for more details.) Note that this performance improvement has been confirmed by several large companies that have tested Impala on real-world workloads for several months now.

A high-level architectural view is below:

There are many advantages to this approach over alternative approaches for querying Hadoop data, including::

  • Thanks to local processing on data nodes, network bottlenecks are avoided.
  • A single, open, and unified metadata store can be utilized.
  • Costly data format conversion is unnecessary and thus no overhead is incurred.
  • All data is immediately query-able, with no delays for ETL.
  • All hardware is utilized for Impala queries as well as for MapReduce.
  • Only a single machine pool is needed to scale.

We encourage you to read the documentation for further technical details.

Finally, we’d like to answer some questions that we anticipate will be popular:

Is Impala open source?
Yes, Impala is 100% open source (Apache License). You can review the code for yourself at Github today.

How is Impala different than Dremel?
The first and principal difference is that Impala is open source and available for everyone to use, whereas Dremel is proprietary to Google.

Technically, Dremel achieves interactive response times over very large data sets through the use of two techniques:

  • A novel columnar storage format for nested relational data/data with nested structures
  • Distributed scalable aggregation algorithms, which allow the results of a query to be computed on thousands of machines in parallel.

The latter is borrowed from techniques developed for parallel DBMSs, which also inspired the creation of Impala. Unlike Dremel as described in the 2010 paper, which could only handle single-table queries, Impala already supports the full set of join operators that are one of the factors that make SQL so popular.

In order to realize the full performance benefits demonstrated by Dremel, Hadoop will shortly have an efficient columnar binary storage format called Parquet. But contrary to Dremel, Impala supports a range of popular file formats. This lets users run Impala on their existing data without having to “load” or transform it. It also lets users decide if they want to optimize for flexibility or just pure performance.

To sum it up, Impala plus Parquet will achieve the query performance described in the Dremel paper, but surpass what is described there in SQL functionality.

How much faster are Impala queries than Hive ones, really?
The precise amount of performance improvement is highly dependent on a number of factors:

  • Hardware configuration: Impala is generally able to take full advantage of hardware resources and specifically generates less CPU load than Hive, which often translates into higher observed aggregate I/O bandwidth than with Hive. Impala of course cannot go faster than the hardware permits, so any hardware bottlenecks will limit the observed speedup. For purely I/O bound queries, we typically see performance gains in the range of 3-4x.
  • Complexity of the query: Queries that require multiple MapReduce phases in Hive or require reduce-side joins will see a higher speedup than, say, simple single-table aggregation queries. For queries with at least one join, we have seem performance gains of 7-45X.
  • Availability of main memory as a cache for table data: If the data accessed through the query comes out of the cache, the speedup will be more dramatic thanks to Impala’s superior efficiency. In those scenarios, we have seen speedups of 20x-90x over Hive even on simple aggregation queries.

Is Impala a replacement for MapReduce or Hive – or for traditional data warehouse infrastructure, for that matter?
No. There will continue be many viable use cases for MapReduce and Hive (for example, for long-running data transformation workloads) as well as traditional data warehouse frameworks (for example, for complex analytics on limited, structured data sets). Impala is a complement to those approaches, supporting use cases where users need to interact with very large data sets, across all data silos, to get focused result sets quickly.

Does the Impala Beta Release have any technical limitations?
As mentioned previously, supported file formats in the first beta drop include text files and SequenceFiles, with many other formats to be supported in the upcoming production release. Furthermore, currently all joins are done in a memory space no larger than that of the smallest node in the cluster; in production, joins will be done in aggregate memory. Lastly, no UDFs are possible at this time.

What are the technical requirements for the Impala Beta Release?
You will need to have CDH4.1 installed on RHEL/CentOS 6.2. We highly recommend the use of Cloudera Manager (Free or Enterprise Edition) to deploy and manage Impala because it takes care of distributed deployment and monitoring details automatically.

What is the support policy for the Impala Beta Release?
If you are an existing Cloudera customer with a bug, you may raise a Customer Support ticket and we will attempt to resolve it on a best-effort basis. If you are not an existing Cloudera customer, you may use our public JIRA instance or the impala-user mailing list, which will be monitored by Cloudera employees.

When will Impala be generally available for production use?
A production drop is planned for the first quarter of 2013. Customers may obtain commercial support in the form of a Cloudera Enterprise RTQ subscription at that time.

We hope that you take the opportunity to review the Impala source code, explore the beta release, download and install the VM, or any combination of the above. Your feedback in all cases is appreciated; we need your help to make Impala even better.

We will bring you further updates about Impala as we get closer to production availability. (Update: Read about Impala 1.0.)

Impala resources:
Impala source code
Impala downloads (Beta Release and VM)
Impala documentation
Public JIRA
Impala mailing list
- Free Impala training (Screencast)

(Added 10/30/2012) Third-party articles about Impala:
- GigaOm: Real-time query for Hadoop democratizes access to big data analytics (Oct. 22, 2012)
- Wired: Man Busts Out of Google, Rebuilds Top-Secret Query Machine (Oct. 24, 2012)
InformationWeek: Cloudera Debuts Real-Time Hadoop Query (Oct. 24, 2012)
GigaOm: Cloudera Makes SQL a First-Class Citizen on Hadoop (Oct. 24, 2012) 
- ZDNet: Cloudera’s Impala Brings Hadoop to SQL and BI (Oct. 25, 2012)
Wired: Marcel Kornacker Profile (Oct. 29, 2012)
- Dr. Dobbs: Cloudera Impala – Processing Petabytes at The Speed Of Thought (Oct. 29, 2012)

 

http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/

 

分享到:
评论

相关推荐

    基于servlet+jsp+mysql实现的影视管理系统课程设计

    【作品名称】:基于servlet+jsp+mysql实现的影视管理系统【课程设计】 【适用人群】:适用于希望学习不同技术领域的小白或进阶学习者。可作为毕设项目、课程设计、大作业、工程实训或初期项目立项。 【项目介绍】: 基于servlet+jsp+mysql实现的影视管理系统【课程设计】 基于servlet+jsp+mysql实现的影视管理系统【课程设计】 Java Web课程设计,基于servlet+jsp+ajax+mysql做的影视管理系统 运行环境: Tomcat 9.0 JDK 1.8 MySQL 8.0 后台管理账号密码均为:root,项目依赖:lib 目录 【资源声明】:本资源作为“参考资料”而不是“定制需求”,代码只能作为参考,不能完全复制照搬。需要有一定的基础看懂代码,自行调试代码并解决报错,能自行添加功能修改代码。

    kernel-5.15-ky10-x86.tar.gz

    kernel-5.15-ky10-x86.tar.gz

    基于AT89C51 单片机为核心器件,程序设计采用C 语言,Keil 软件编译程序,配以相关外围接口电路,实现了方波、锯齿波、正弦波、三角波、梯形波五种特定波形的产生【论文+源码】

    【作品名称】:基于AT89C51 单片机为核心器件,程序设计采用C 语言,Keil 软件编译程序,配以相关外围接口电路,实现了方波、锯齿波、正弦波、三角波、梯形波五种特定波形的产生【论文+源码】 【适用人群】:适用于希望学习不同技术领域的小白或进阶学习者。可作为毕设项目、课程设计、大作业、工程实训或初期项目立项。 【项目介绍】:本设计中的波形发生器系统要求基于51单片机,因此选用以AT89C51单片机作为整个系统的控制核心,应用其强大的接口功能,构成整个波形发生器的硬件系统。使用C 语言对单片机编程可产生相应的正弦波,方波,三角波,锯齿波梯形波波形信号。在程序运行时,当接收到按键信息后,需要输出某种波形时,调用相应的中断服务子程序和波形发生程序,经电路的数/模转换器和运算放大器处理后,从信号发生器的输出端口输出即可得到要求的波形。 当需要改变频率时只需要改变单片机的波形发生程序中的递增或者递减变量即可。 【资源声明】:本资源作为“参考资料”而不是“定制需求”,代码只能作为参考,不能完全复制照搬。需要有一定的基础看懂代码,自行调试代码并解决报错,能自行添加功能修改代码。

    基于java的法律咨询系统设计与实现.docx

    基于java的法律咨询系统设计与实现.docx

    适用于元营销 API 的 Python SDK.zip

    适用于元营销 API 的 Python SDK适用于 Python 的 Facebook Business SDK 介绍Facebook Business SDK是一站式服务,可帮助我们的合作伙伴更好地服务于他们的业务。合作伙伴正在使用多个 Facebook API 来满足其客户的需求。采用所有这些 API 并在各个平台上保持最新状态可能非常耗时,而且最终会造成高昂的成本。为此,Facebook 开发了 Business SDK,将其许多 API 捆绑到一个 SDK 中,以简化实施和维护。Business SDK 是 Marketing API SDK 的升级版,其中包括 Marketing API 以及来自不同平台(如 Pages、Business Manager、Instagram 等)的许多 Facebook API。快速入门商业SDK入门指南Python 目前是我们第三方开发人员最常用的语言。是一个 Python 包,它提供了您的 Python 应用程序与Business SDK 内的 Facebook APIfacebook_business之间的

    数学建模培训资料 数学建模实战题目真题答案解析解题过程&论文报告 公交车调度的运作数学模型 共12页.pdf

    数学建模培训资料 数学建模实战题目真题答案解析解题过程&论文报告 公交车调度的运作数学模型 共12页.pdf

    基于smart-socket实现的轻量级http服务器

    smart-http 是一款可编程的 Http 应用微内核,方便用户根据自身需求进行 Server 或 Client 的应用开发。支持GET、POST的 HTTP 请求。提供了 URL 路由组件,可以快速搭建一套静态服务器。支持部分 RFC2612 规范,后续会逐渐完善。支持 Https 协议,由 smart-socket 为其赋能。具备文件上传的能力。支持 websocket、Cookie支持 Server、Client 开发

    新闻资讯系统 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程.zip

    新闻资讯系统 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程 项目启动教程:https://www.bilibili.com/video/BV1oiBpYcEBp

    高校师生工作室-JAVA-基于微信小程序的高校师生工作室管理系统的设计与实现

    高校师生工作室-JAVA-基于微信小程序的高校师生工作室管理系统的设计与实现

    基于java的常见小儿疾病中医护理系统设计与实现.docx

    基于java的常见小儿疾病中医护理系统设计与实现.docx

    本教程播放列表涵盖了 Python 中的数据结构和算法 每个教程都有数据结构或算法背后的理论、BIG O 复杂性分析和可供练习的练习 .zip

    本教程播放列表涵盖了 Python 中的数据结构和算法。每个教程都有数据结构或算法背后的理论、BIG O 复杂性分析和可供练习的练习。使用 Python 的数据结构和算法本教程涵盖了 Python 中的数据结构和算法。每个教程都包含数据结构或算法背后的理论、BIG O 复杂度分析以及可供练习的练习。要观看视频,您可以访问播放列表https://www.youtube.com/playlist?list=PLeo1K3hjS3uu_n_a__MI_KktGTLYopZ12订阅 codebasics youtube 频道https://www.youtube.com/c/codebasics

    数学建模学习资料 蒙特卡罗方法课件教程 第2章.随机数 共29页.pptx

    数学建模学习资料 蒙特卡罗方法课件教程 第2章.随机数 共29页.pptx

    python实现基于CNN网络的新闻数据集文本分类源码+数据集(Python期末大作业)

    python实现基于CNN网络的新闻数据集文本分类源码+数据集(Python期末大作业),个人大三学期的期末大作业、经导师指导并认可通过的高分大作业设计项目,评审分98分。主要针对计算机相关专业的正在做大作业的学生和需要项目实战练习的学习者,可作为课程设计、期末大作业。 python实现基于CNN网络的新闻数据集文本分类源码+数据集(Python期末大作业)python实现基于CNN网络的新闻数据集文本分类源码+数据集(Python期末大作业),个人大三学期的期末大作业、经导师指导并认可通过的高分大作业设计项目,评审分98分。主要针对计算机相关专业的正在做大作业的学生和需要项目实战练习的学习者,可作为课程设计、期末大作业。python实现基于CNN网络的新闻数据集文本分类源码+数据集(Python期末大作业),个人大三学期的期末大作业、经导师指导并认可通过的高分大作业设计项目,评审分98分。主要针对计算机相关专业的正在做大作业的学生和需要项目实战练习的学习者,可作为课程设计、期末大作业。python实现基于CNN网络的新闻数据集文本分类源码+数据集(Python期末大作业),个人大

    中小学知识产权教育试点学校申报表.doc

    中小学知识产权教育试点学校申报表.doc

    基于django的音乐推荐系统.zip

    基于django的音乐推荐系统.zip

    在建工程涉及专项行动情况检查表.docx

    在建工程涉及专项行动情况检查表.docx

    毕设源码-python-django基于python技术的学生管理系统的设计与开发-期末大作业+说明文档.rar

    本项目是一个基于Python技术的学生管理系统,采用Django框架进行开发,旨在为计算机相关专业的学生提供一个实践性强、功能全面的管理系统,以帮助他们完成毕业设计或进行项目实战练习。 系统实现了对学生信息、课程信息、成绩、考勤等多方面的管理功能。学生信息管理包括学生基本信息的增删改查;课程信息管理允许管理员设置课程信息,包括课程名称、授课老师、学分等;成绩管理功能使学生和教师能够录入、查看和修改成绩;考勤管理则方便教师记录学生的出勤情况。 该项目采用B/S架构,前端使用HTML、CSS、JavaScript等技术,后端使用Python语言和Django框架,数据库采用MySQL。Django框架提供了强大的后台管理功能,使得系统管理更加便捷。 通过开发这个项目,学生不仅能提升自己的编程能力,还能学习到如何构建一个实际应用的系统,对于即将步入职场的学生来说,具有很高的实用价值。

    适用于 Python 的 Splunk 软件开发工具包.zip

    适用于 Python 的 Splunk 软件开发工具包参考文档适用于 Python 的 Splunk Enterprise 软件开发工具包版本 2.1.0适用于 Python 的 Splunk Enterprise 软件开发套件 (SDK) 包含库代码,旨在使开发人员能够使用 Splunk 平台构建应用程序。Splunk 平台是一个搜索引擎和分析环境,它使用分布式 map-reduce 架构来有效地索引、搜索和处理大型时变数据集。Splunk 平台深受系统管理员的欢迎,用于聚合和监控 IT 机器数据、安全性、合规性以及各种其他场景,这些场景都需要有效地从大量时间序列数据中索引、搜索、分析和生成实时通知。Splunk 开发者平台使开发人员能够利用 Splunk 平台所使用的相同技术来构建令人兴奋的新应用程序。开始使用 Python 版 Splunk SDK开始使用 Python 版 Splunk Enterprise SDKSplunk Enterprise SDK for Python 包含库代码,其示例位于splunk-app-examples存储库

    分布式事务练习.zip

    分布式事务练习

    家庭财务管理系统 微信小程序+SSM毕业设计 源码+数据库+论文+启动教程.zip

    家庭财务管理系统 微信小程序+SSM毕业设计 源码+数据库+论文+启动教程 项目启动教程:https://www.bilibili.com/video/BV1BfB2YYEnS

Global site tag (gtag.js) - Google Analytics