After a long period of intense engineering effort and user feedback, we are very pleased, and proud, to announce the Cloudera Impala project. This technology is a revolutionary one for Hadoop users, and we do not take that claim lightly.
When Google published its Dremel paper in 2010, we were as inspired as the rest of the community by the technical vision to bring real-time, ad hoc query capability to Apache Hadoop, complementing traditional MapReduce batch processing. Today, we are announcing a fully functional, open-sourced codebase that delivers on that vision – and, we believe, a bit more – which we call Cloudera Impala. An Impala binary is now available in public beta form, but if you would prefer to test-drive Impala via a pre-baked VM, we have one of those for you, too. (Links to all downloads and documentation are here.) You can also review the source code and testing harness at Github right now.
Impala raises the bar for query performance while retaining a familiar user experience. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time. Furthermore, it uses the same metadata, SQL syntax (Hive SQL), ODBC driver and user interface (Hue Beeswax) as Apache Hive, providing a familiar and unified platform for batch-oriented or real-time queries. (For that reason, Hive users can utilize Impala with little setup overhead.) The first beta drop includes support for text files and SequenceFiles; SequenceFiles can be compressed as Snappy, GZIP, and BZIP (with Snappy recommended for maximum performance). Support for additional formats including Avro, RCFile, LZO text files, and the Parquet columnar format is planned for the production drop.
To avoid latency, Impala circumvents MapReduce to directly access the data through a specialized distributed query engine that is very similar to those found in commercial parallel RDBMSs. The result is order-of-magnitude faster performance than Hive, depending on the type of query and configuration. (See FAQ below for more details.) Note that this performance improvement has been confirmed by several large companies that have tested Impala on real-world workloads for several months now.
A high-level architectural view is below:
There are many advantages to this approach over alternative approaches for querying Hadoop data, including::
- Thanks to local processing on data nodes, network bottlenecks are avoided.
- A single, open, and unified metadata store can be utilized.
- Costly data format conversion is unnecessary and thus no overhead is incurred.
- All data is immediately query-able, with no delays for ETL.
- All hardware is utilized for Impala queries as well as for MapReduce.
- Only a single machine pool is needed to scale.
We encourage you to read the documentation for further technical details.
Finally, we’d like to answer some questions that we anticipate will be popular:
Is Impala open source?
Yes, Impala is 100% open source (Apache License). You can review the code for yourself at Github today.
How is Impala different than Dremel?
The first and principal difference is that Impala is open source and available for everyone to use, whereas Dremel is proprietary to Google.
Technically, Dremel achieves interactive response times over very large data sets through the use of two techniques:
- A novel columnar storage format for nested relational data/data with nested structures
- Distributed scalable aggregation algorithms, which allow the results of a query to be computed on thousands of machines in parallel.
The latter is borrowed from techniques developed for parallel DBMSs, which also inspired the creation of Impala. Unlike Dremel as described in the 2010 paper, which could only handle single-table queries, Impala already supports the full set of join operators that are one of the factors that make SQL so popular.
In order to realize the full performance benefits demonstrated by Dremel, Hadoop will shortly have an efficient columnar binary storage format called Parquet. But contrary to Dremel, Impala supports a range of popular file formats. This lets users run Impala on their existing data without having to “load” or transform it. It also lets users decide if they want to optimize for flexibility or just pure performance.
To sum it up, Impala plus Parquet will achieve the query performance described in the Dremel paper, but surpass what is described there in SQL functionality.
How much faster are Impala queries than Hive ones, really?
The precise amount of performance improvement is highly dependent on a number of factors:
- Hardware configuration: Impala is generally able to take full advantage of hardware resources and specifically generates less CPU load than Hive, which often translates into higher observed aggregate I/O bandwidth than with Hive. Impala of course cannot go faster than the hardware permits, so any hardware bottlenecks will limit the observed speedup. For purely I/O bound queries, we typically see performance gains in the range of 3-4x.
- Complexity of the query: Queries that require multiple MapReduce phases in Hive or require reduce-side joins will see a higher speedup than, say, simple single-table aggregation queries. For queries with at least one join, we have seem performance gains of 7-45X.
- Availability of main memory as a cache for table data: If the data accessed through the query comes out of the cache, the speedup will be more dramatic thanks to Impala’s superior efficiency. In those scenarios, we have seen speedups of 20x-90x over Hive even on simple aggregation queries.
Is Impala a replacement for MapReduce or Hive – or for traditional data warehouse infrastructure, for that matter?
No. There will continue be many viable use cases for MapReduce and Hive (for example, for long-running data transformation workloads) as well as traditional data warehouse frameworks (for example, for complex analytics on limited, structured data sets). Impala is a complement to those approaches, supporting use cases where users need to interact with very large data sets, across all data silos, to get focused result sets quickly.
Does the Impala Beta Release have any technical limitations?
As mentioned previously, supported file formats in the first beta drop include text files and SequenceFiles, with many other formats to be supported in the upcoming production release. Furthermore, currently all joins are done in a memory space no larger than that of the smallest node in the cluster; in production, joins will be done in aggregate memory. Lastly, no UDFs are possible at this time.
What are the technical requirements for the Impala Beta Release?
You will need to have CDH4.1 installed on RHEL/CentOS 6.2. We highly recommend the use of Cloudera Manager (Free or Enterprise Edition) to deploy and manage Impala because it takes care of distributed deployment and monitoring details automatically.
What is the support policy for the Impala Beta Release?
If you are an existing Cloudera customer with a bug, you may raise a Customer Support ticket and we will attempt to resolve it on a best-effort basis. If you are not an existing Cloudera customer, you may use our public JIRA instance or the impala-user mailing list, which will be monitored by Cloudera employees.
When will Impala be generally available for production use?
A production drop is planned for the first quarter of 2013. Customers may obtain commercial support in the form of a Cloudera Enterprise RTQ subscription at that time.
We hope that you take the opportunity to review the Impala source code, explore the beta release, download and install the VM, or any combination of the above. Your feedback in all cases is appreciated; we need your help to make Impala even better.
We will bring you further updates about Impala as we get closer to production availability. (Update: Read about Impala 1.0.)
Impala resources:
– Impala source code
– Impala downloads (Beta Release and VM)
– Impala documentation
– Public JIRA
– Impala mailing list
- Free Impala training (Screencast)
(Added 10/30/2012) Third-party articles about Impala:
- GigaOm: Real-time query for Hadoop democratizes access to big data analytics (Oct. 22, 2012)
- Wired: Man Busts Out of Google, Rebuilds Top-Secret Query Machine (Oct. 24, 2012)
- InformationWeek: Cloudera Debuts Real-Time Hadoop Query (Oct. 24, 2012)
- GigaOm: Cloudera Makes SQL a First-Class Citizen on Hadoop (Oct. 24, 2012)
- ZDNet: Cloudera’s Impala Brings Hadoop to SQL and BI (Oct. 25, 2012)
- Wired: Marcel Kornacker Profile (Oct. 29, 2012)
- Dr. Dobbs: Cloudera Impala – Processing Petabytes at The Speed Of Thought (Oct. 29, 2012)
http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/
相关推荐
标题《Cloudera-JDBC-Driver-for-Impala-Install-Guide.pdf》涉及的知识点主要包括: 1. Cloudera公司的Impala产品和JDBC驱动程序介绍:文档中提到了Cloudera公司,以及它的主要产品Impala,这是一款由Cloudera开发...
- **使用Spark Shell探索RDDs**:Apache Spark是Cloudera CDH中的一部分,用于处理大规模数据集的快速通用引擎。RDD(弹性分布式数据集)是Spark中的基础抽象,这部分练习帮助学员理解RDD的概念以及如何在Spark ...
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
标题:“Impala A Modern, Open-Source SQL Engine for Hadoop”介绍了一个现代化的开源SQL引擎,专门针对Hadoop数据处理环境进行架构设计。接下来,我将围绕这个标题和文件中提供的信息展开详细的讲解。 知识点一...
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
大数据管理与监控:Cloudera Manager:大数据概论与Hadoop生态系统.docx
cloudera-manager-agent-5.9.0-1.cm590.p0.249.el7.x86_64.rpm cloudera-manager-daemons-5.9.0-1.cm590.p0.249.el7.x86_64.rpm cloudera-manager-server-5.9.0-1.cm590.p0.249.el7.x86_64.rpm cloudera-manager-...
标题提及的是“基于Hadoop的实时查询 Cloudera Impala”,这指的是一种在Hadoop生态系统中实现快速、低延迟查询的工具——Cloudera Impala。Cloudera是一家提供大数据管理和分析平台的公司,Impala是它们推出的一个...
cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm centos7CM包下载
Cloudera Impala 是一款基于 Apache Hadoop 的 SQL 查询引擎,旨在为用户提供快速、交互式的查询体验。Impala 支持直接对存储在 Hadoop 分布式文件系统 (HDFS) 或 HBase 中的数据进行 SQL 查询。其与 Apache Hive ...
大数据管理与监控:Cloudera Manager:Impala实时查询引擎使用.docx
cloudera-manager-agent-5.9.0-1.cm590.p0.249.el7.x86_64.rpm cloudera-manager-daemons-5.9.0-1.cm590.p0.249.el7.x86_64.rpm cloudera-manager-server-5.9.0-1.cm590.p0.249.el7.x86_64.rpm cloudera-manager-...
/usr/bin/docker-quickstart /home/cloudera/cloudera-manager --express && service ntpd start ``` 配置 HDFS 进入容器后,需要配置 HDFS 节点: ```bash vim /etc/hadoop/conf.cloudera.hdfs/hdfs-site.xml ``...
cloudera-impala.pdf-impala使用文档 cloudera-installation.pdf-安装CDH和CM的文档 cloudera-introduction.pdf-基本介绍文档 cloudera-operation.pdf-操作使用文档 cloudera-quickstart.pdf-虚拟机快速使用(免安装)...
标题中的"cloudera-hive/impala-odbc"指的是Cloudera提供的Hive和Impala的ODBC(Open Database Connectivity)驱动程序。ODBC是用于在不同数据库系统之间建立连接的标准接口,使得应用程序可以访问多种类型的数据库...
Cloudera Distribution Including Apache Hadoop (CDH) 是一个包含了 Hadoop 生态系统中多个组件的集成解决方案。Impala 作为 CDH 的一部分,与其他组件如 HDFS、HBase、Hive 等紧密集成。这种集成不仅使得 Impala ...
Impala 驱动包 Cloudera_ImpalaJDBC4_2.5.41.zip Cloudera_ImpalaJDBC41_2.5.41.zip Cloudera-JDBC-Driver-for-Impala-Install-Guide.pdf Cloudera-JDBC-Driver-for-Impala-Release-Notes.pdf
cloudera-manager-agent-7.4.4-15850731.el7.x86_64.rpm cloudera-manager-daemons-7.4.4-15850731.el7.x86_64.rpm cloudera-manager-server-7.4.4-15850731.el7.x86_64.rpm cloudera-manager-server-db-2-7.4.4-...
cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
Cloudera Impala 是一款由 Cloudera 开发的高性能分布式 SQL 查询系统,它为大数据提供了接近于传统数据库性能的速度,同时保持了 Hadoop 的扩展性和灵活性。Impala 的设计初衷是为了弥补当时市场上存在的查询工具在...