`
nlslzf
  • 浏览: 1046184 次
  • 性别: Icon_minigender_1
  • 来自: 北京
社区版块
存档分类
最新评论

How to Benchmark a Hadoop Cluster

阅读更多

How to Benchmark a Hadoop Cluster

http://answers.oreilly.com/topic/460-how-to-benchmark-a-hadoop-cluster/

Is the cluster set up correctly? The best way to answer this question is empirically: run some jobs and confirm that you get the expected results. Benchmarks make good tests, as you also get numbers that you can compare with other clusters as a sanity check on whether your new cluster is performing roughly as expected. And you can tune a cluster using benchmark results to squeeze the best performance out of it. This is often done with monitoring systems in place, so you can see how resources are being used across the cluster.

To get the best results, you should run benchmarks on a cluster that is not being used by others. In practice, this is just before it is put into service, and users start relying on it. Once users have periodically scheduled jobs on a cluster it is generally impossible to find a time when the cluster is not being used (unless you arrange downtime with users), so you should run benchmarks to your satisfaction before this happens.

Experience has shown that most hardware failures for new systems are hard drive failures. By running I/O intensive benchmarks—such as the ones described next—you can “burn in” the cluster before it goes live.

Hadoop Benchmarks

Hadoop comes with several benchmarks that you can run very easily with minimal setup cost. Benchmarks are packaged in the test JAR file, and you can get a list of them, with descriptions, by invoking the JAR file with no arguments:

% hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar

Most of the benchmarks show usage instructions when invoked with no arguments. For example:

% hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO

TestFDSIO.0.0.4

Usage: TestFDSIO -read | -write | -clean [-nrFiles N] [-fileSize MB] [-resFile 

resultFileName] [-bufferSize Bytes] 

Benchmarking HDFS with TestDFSIO

TestDFSIO tests the I/O performance of HDFS. It does this by using a MapReduce job as a convenient way to read or write files in parallel. Each file is read or written in a separate map task, and the output of the map is used for collecting statistics relating to the file just processed. The statistics are accumulated in the reduce, to produce a summary.

The following command writes 10 files of 1,000 MB each:

% hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -write -nrFiles 10

-fileSize 1000

At the end of the run, the results are written to the console and also recorded in a local file (which is appended to, so you can rerun the benchmark and not lose old results):

% cat TestDFSIO_results.log

----- TestDFSIO ----- : write

           Date & time: Sun Apr 12 07:14:09 EDT 2009

       Number of files: 10

Total MBytes processed: 10000

     Throughput mb/sec: 7.796340865378244

Average IO rate mb/sec: 7.8862199783325195

 IO rate std deviation: 0.9101254683525547

    Test exec time sec: 163.387

The files are written under the /benchmarks/TestDFSIO directory by default (this can be changed by setting thetest.build.data system property), in a directory called io_data.

To run a read benchmark, use the -read argument. Note that these files must already exist (having been written byTestDFSIO -write):

% hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -read -nrFiles 10 

-fileSize 1000

Here are the results for a real run:

----- TestDFSIO ----- : read

           Date & time: Sun Apr 12 07:24:28 EDT 2009

       Number of files: 10

Total MBytes processed: 10000

     Throughput mb/sec: 80.25553361904304

Average IO rate mb/sec: 98.6801528930664

 IO rate std deviation: 36.63507598174921

    Test exec time sec: 47.624

When you’ve finished benchmarking, you can delete all the generated files from HDFS using the -clean argument:

% hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -clean

Benchmarking MapReduce with Sort

Hadoop comes with a MapReduce program that does a partial sort of its input. It is very useful for benchmarking the whole MapReduce system, as the full input dataset is transferred through the shuffle. The three steps are: generate some random data, perform the sort, then validate the results.

First we generate some random data using RandomWriter. It runs a MapReduce job with 10 maps per node, and each map generates (approximately) 10 GB of random binary data, with key and values of various sizes. You can change these values if you like by setting the properties test.randomwriter.maps_per_host and test.randomwrite.bytes_per_map. There are also settings for the size ranges of the keys and values; see RandomWriter for details.

Here’s how to invoke RandomWriter (found in the example JAR file, not the test one) to write its output to a directory calledrandom-data:

% hadoop jar $HADOOP_INSTALL/hadoop-*-examples.jar randomwriter random-data

Next we can run the Sort program:

% hadoop jar $HADOOP_INSTALL/hadoop-*-examples.jar sort random-data sorted-data

The overall execution time of the sort is the metric we are interested in, but it’s instructive to watch the job’s progress via the web UI (http://jobtracker-host:50030/), where you can get a feel for how long each phase of the job takes.

As a final sanity check, we validate the data in sorted-data is, in fact, correctly sorted:

% hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar testmapredsort -sortInput random-data \

  -sortOutput sorted-data

This command runs the SortValidator program, which performs a series of checks on the unsorted and sorted data to check whether the sort is accurate. It reports the outcome to the console at the end of its run:

SUCCESS! Validated the MapReduce framework's 'sort' successfully.

Other benchmarks

There are many more Hadoop benchmarks, but the following are widely used:

  • MRBench (invoked with mrbench) runs a small job a number of times. It acts as a good counterpoint to sort, as it checks whether small job runs are responsive.

  • NNBench (invoked with nnbench) is useful for load testing namenode hardware.

  • Gridmix is a suite of benchmarks designed to model a realistic cluster workload, by mimicking a variety of data-access patterns seen in practice. See src/benchmarks/gridmix2 in the distribution for further details.[63]

User Jobs

For tuning, it is best to include a few jobs that are representative of the jobs that your users run, so your cluster is tuned for these and not just for the standard benchmarks. If this is your first Hadoop cluster and you don’t have any user jobs yet, then Gridmix is a good substitute.

When running your own jobs as benchmarks you should select a dataset for your user jobs that you use each time you run the benchmarks to allow comparisons between runs. When you set up a new cluster, or upgrade a cluster, you will be able to use the same dataset to compare the performance with previous runs.

[63In a similar vein, PigMix is a set of benchmarks for Pig available from http://wiki.apache.org/pig/PigMix.

Cover of Hadoop: The Definitive Guide
Learn more about this topic from Hadoop: The Definitive Guide. 

Apache Hadoop is ideal for organizations with a growing need to process massive application datasets.Hadoop: The Definitive Guide is a comprehensive resource for using Hadoop to build reliable, scalable, distributed systems. Programmers will find details for analyzing large datasets with Hadoop, and administrators will learn how to set up and run Hadoop clusters. The book includes case studies that illustrate how Hadoop is used to solve specific problems.

分享到:
评论

相关推荐

    How to Benchmark Your Linux System.mp4

    How to Benchmark Your Linux System.mp4How to Benchmark Your Linux System.mp4How to Benchmark Your Linux System.mp4

    A Benchmark Approach to Quantitative Finance.pdf

    《基准方法在量化金融中的应用》(A Benchmark Approach to Quantitative Finance)是量化金融领域的一本重要著作,由Eduard Platen与David Heath共同撰写,并于2006年出版。本书不仅为读者提供了对基准方法在量化...

    大数据之路选择Hadoop还是MaxCompute?Hadoop开源与MaxCompute对比材料

    TPC-DS (Transaction Processing Performance Council Benchmark D) 是一种专门用来衡量大规模数据仓库性能的标准测试方法。通过比较不同系统的TPC-DS测试结果,可以直观地看出它们在处理大规模数据集时的表现差异。...

    apache benchmark ab.exe

    -v verbosity How much troubleshooting info to print -w Print out results in HTML tables -i Use HEAD instead of GET -x attributes String to insert as table attributes -y attributes String to ...

    Benchmark.rar_benchmark_benchmark三代_benchmark模型_三代benchmark_主动控制

    总结来说,"Benchmark.rar_benchmark_benchmark三代_benchmark模型_三代benchmark_主动控制"是一个专注于主动控制算法评估的资源包,它提供了一个先进的三代Benchmark模型,用于比较和优化控制策略的性能。...

    Hadoop集群

    Hortonworks由雅虎与Benchmark Capital合资成立,其工程师团队对Hadoop的贡献很大,拥有Hadoop的核心代码。 Hadoop之所以受到如Yahoo!、Facebook、LinkedIn等公司青睐,是因为其强大的数据存储和分析能力。这些公司...

    RGBD Salient Object Detection A Benchmark and Algorithms

    ECCV2014最新论文RGBD Salient Object Detection A Benchmark and Algorithms。

    Ruby Performance Optimization, Why Ruby is Slow, and How to Fix It

    You’ll make sure slow code doesn’t creep back into your Ruby application by writing performance tests, and you’ll learn the right way to benchmark Ruby. And finally, you’ll dive into the Ruby ...

    HADOOP案例及测试资料

    "Hibench BenchMark suite.docx"涉及的是Hadoop的基准测试工具Hibench,它详细介绍了如何使用Hibench进行大数据处理性能的评估,包括各种工作负载的设定和结果分析,这对于评估和优化Hadoop集群性能至关重要。...

    Hadoop 安装文档 性能测试

    常用的测试工具有Hadoop自带的Benchmark工具如`hadoop dfsio`和`testDFSIO`等。通过执行这些工具的不同模式(如读、写、随机读等),可以评估Hadoop集群的性能表现,为后续的调优提供数据支持。 #### 四、结语 ...

    大数据技术之Hadoop(入门).doc

    Hortonworks则是由雅虎和Benchmark Capital合资成立的,其团队成员为Hadoop的早期开发者,对Hadoop代码库做出了巨大贡献。 Hortonworks的Hadoop发行版在企业中广泛应用,特别是在稳定性和支持方面表现出色。 总的来...

    PHASE II OF THE ASCE BENCHMARK STUDY ON SHM.rar_BENCHMARK ASCE_a

    PHASE II OF THE ASCE BENCHMARK STUDY ON SHM

    Fritz Chess Benchmark4.3.2完全汉化版

    - “坑爹的小A”不仅完成了Fritz Chess Benchmark4.3.2的汉化工作,还可能对软件进行了本地化适应,确保其在中国环境下的稳定运行。 - 汉化版的推出降低了语言障碍,使得更多用户能够理解和使用这款专业工具,促进...

    TeraByte Sort on Apache Hadoop

    ### TeraByte Sort on Apache Hadoop #### 概述 《TeraByte Sort on Apache Hadoop》是由Yahoo公司的Owen O’Malley撰写的一篇关于Hadoop基准测试方法的论文,该论文详细介绍了一种用于Hadoop平台的大规模数据排序...

    Benchmark etcd性能测试工具

    Benchmark etcd性能测试工具

    benchmarksql数据库测试工具--支持达梦数据库版

    ** BenchmarkSQL数据库测试工具概述 ** BenchmarkSQL是一款广泛使用的开源数据库性能测试工具,它能够对各种类型的数据库系统进行基准测试,以评估其在不同工作负载下的性能。该工具旨在提供一个标准化的方法来比较...

    大数据技术之Hadoop(入门).docx

    - Hortonworks成立于2011年,最初由雅虎和Benchmark Capital共同投资成立。 - 主打产品是HDP(Hortonworks Data Platform),同样是一个100%开源的发行版,包含了Ambari,用于简化安装和管理流程。 - 2019年,...

    Benchmark Factory.docx

    《Benchmark Factory TPC-C 测试指南详解》 Benchmark Factory是一款强大的性能测试工具,尤其在评估数据库系统的性能方面,它遵循了TPC-C标准,能够模拟真实的业务负载,从而准确地衡量系统性能。本文将深入探讨...

Global site tag (gtag.js) - Google Analytics