`

compile hadoop-2.5.x on OS X(macbook)

 
阅读更多

  same as compile hbase ,it 's necessary for u to compile your own dist of hadoop to get best performance specified to your hardwares.

  bellow are some steps needed by this theme:

1.install some tools 

   a.cmake 2.6+

      download and set the path variable 

      this is used only if u want to make with native option,same like:

 

 mvn package -Pdist,native -DskipTests=true  -Dmaven.javadoc.skip=true

 

 

   b.protobuf  2.5+

     download and set variable

   c.mavn 3.0+

    same as above

 

2.comile hadoop by issuing

 

 mvn package -Pdist,native -DskipTests=true  -Dmaven.javadoc.skip=true -Dtar

 the results will like below:

main:
     [exec] $ tar cf hadoop-2.5.2.tar hadoop-2.5.2
     [exec] $ gzip -f hadoop-2.5.2.tar
     [exec] 
     [exec] Hadoop dist tar available at: /home/xxx/build-hadoop/hadoop-2.5.2-src/hadoop-dist/target/hadoop-2.5.2.tar.gz
     [exec] 
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /home/xxx/build-hadoop/hadoop-2.5.2-src/hadoop-dist/target/hadoop-dist-2.5.2-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................ SUCCESS [3:01.134s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [1:05.262s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [36.726s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.232s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [40.162s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [42.839s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [2:45.883s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [36.327s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [13.556s]
[INFO] Apache Hadoop Common .............................. SUCCESS [3:46.794s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [4.543s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.037s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [3:32.422s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [28.563s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [31.706s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [3.193s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.042s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.032s]
[INFO] hadoop-yarn-api ................................... SUCCESS [1:15.932s]
[INFO] hadoop-yarn-common ................................ SUCCESS [41.828s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.032s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [19.335s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [52.061s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [2.190s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [4.083s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [10.974s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.469s]
[INFO] hadoop-yarn-client ................................ SUCCESS [4.002s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.032s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [2.106s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [1.710s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.028s]
[INFO] hadoop-yarn-project ............................... SUCCESS [4.258s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.067s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [14.625s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [15.159s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [3.097s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [7.088s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [6.639s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [13.638s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [1.573s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [4.095s]
[INFO] hadoop-mapreduce .................................. SUCCESS [3.397s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [8.712s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [25.432s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [1.784s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [4.306s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [3.500s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [2.054s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [2.298s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [7.284s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [3.687s]
[INFO] Apache Hadoop Client .............................. SUCCESS [8.373s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.121s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [10.919s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [6.671s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.033s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [32.206s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26:07.575s
[INFO] Finished at: Fri Dec 12 16:12:56 CST 2014
[INFO] Final Memory: 123M/940M
[INFO] ------------------------------------------------------------------------

 

 

3.problems

a.Exception in thread "main" java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()

  this is as the compiler needs a file named tools.jar but localted at classes.jar under dir Classes but found nothing.so u can do workaround:

  

cd `java_home`/
mkdir Classes
cd Classes
ln -s ../lib/tools.jar classes.jar

   but u should know,in macbook,the command 'ln' is ONLY effect when u locate in the target dir,otherwise u will see:

  

cd `java_home`
mkdir Classes
ln -s lib/tools.jar Classes/classes.jar
# or:ln -s lib/tools.jar /absolute/path/to/Classes/classes.jar
cp Classes/classes.jar /tmp
Error:no such file or dir found

 

2.failed to compile target:hadoop mavn doc

   this is not important for this theme,so u can add a param to ignore it:

-Dmaven.javadoc.skip=true

 

 Ref:

 
0
2
分享到:
评论
2 楼 leibnitz 2014-10-30  
hi guy,
  this is used as development mode.simple,convenient:)
string2020 写道
撸主真土豪,在苹果里面玩大数据.

1 楼 string2020 2014-10-30  
撸主真土豪,在苹果里面玩大数据.

相关推荐

    hadoop-2.9.2.tar.gz

    《Hadoop 2.9.2在CentOS7 64位系统上的构建与应用》 Hadoop,作为大数据处理的核心框架,其2.9.2版本是经过长期优化和测试的一个稳定版本,广泛应用于数据分布式存储和计算。本文将详细解析如何在基于CentOS7的64位...

    elasticsearch-head-compile-after.tar.gz+node-v8.1.2-linux-x64.ta

    标题中的"elasticsearch-head-compile-after.tar.gz"和"node-v8.1.2-linux-x64.ta"分别指的是两个不同的软件组件。首先,Elasticsearch是一个流行的开源搜索引擎和分析平台,广泛应用于大数据检索、日志分析和实时...

    hadoop-2.6.1.rar win7编译

    在本主题中,我们将深入探讨如何在Windows 7操作系统上编译Hadoop 2.6.1源码。Hadoop是Apache软件基金会的一个开源项目,主要用于处理和存储大量数据,是大数据领域的基石。Hadoop 2.6.1版本包含了众多优化和改进,...

    hadoop-lzo-master

    ant compile-native tar 将生成的 build/hadoop-lzo-0.4.15.jar cp 到 /usr/local/hadoop-1.0.2/lib 测试解压程序 bin/hadoop jar /usr/local/hadoop-1.0.2/lib/hadoop-lzo-0.4.15.jar ...

    compile-node-sass-0.0.3.zip

    【标题】"compile-node-sass-0.0.3.zip" 涉及的主要知识点是关于Node.js环境下的Sass编译工具,它允许开发者将Sass或SCSS语法转换成CSS,以便在Web开发中使用。Sass是一种强大的CSS预处理器,提供了变量、嵌套规则、...

    hadoop-compile-depend.zip

    本文将深入探讨"Hadoop-Compile-Depend"这一主题,解析Hadoop编译过程中所依赖的关键组件、环境配置以及编译步骤,旨在为读者提供全面的Hadoop编译知识。 首先,我们要明确的是,"hadoop-compile-depend.zip"是一个...

    编译hadoop-2.5.0-cdh5.3.6 + snappy 源码的native包

    - 运行`ant compile-native`编译native库,这会链接到你之前编译的Snappy库。 - 接下来,执行`ant package`生成Hadoop的jar包。 5. **配置Hadoop**: - 将编译生成的`lib/hadoop/lib/native`目录下的所有库文件...

    elasticsearch-head-compile-after.tar.gz

    1. **解压源码**:首先,将 "elasticsearch-head-compile-after.tar.gz" 解压到你的工作目录,例如 `tar -zxvf elasticsearch-head-compile-after.tar.gz`。 2. **运行elasticsearch-head**:解压后,找到包含 ...

    glfw-2.5.zip

    opengl工具glfw以及完整的使用说明文档,可以应用于多种操作系统 compile.ami config.log lib readme.html compile.bat docs images license.txt support compile.sh examples include Makefile

    hadoop-2.6.0-cdh5.14.2.tar _after_compile -src

    标题中的"hadoop-2.6.0-cdh5.14.2.tar_after_compile -src"揭示了这是一个关于Hadoop的源代码版本,经过编译后的结果。Hadoop是大数据处理领域的重要工具,其核心是分布式文件系统(HDFS)和MapReduce计算框架。这个...

    Python库 | req_compile-0.10.15-py2.py3-none-any.whl

    标题中的"req_compile-0.10.15-py2.py3-none-any.whl"是一个Python库的发行文件,这种格式是Python的Wheel(轮子)文件,它是一种预编译的Python软件包格式,旨在简化安装过程,提高安装速度。这个文件的版本号为...

    commons-fileupload-1.2.jar和commons-io-1.3.2.jar

    // text file, the second is unknown and is written to a file on // the server Iterator iter = fileItems.iterator(); // 正则匹配,过滤路径取文件名 String regExp = ".+\\\\(.+)$"; // 过滤掉的文件...

    aspectjrt-1.8.7.jar aspectjweaver-1.8.7.jar

    1. **编译时织入(CTW, Compile-Time Weaving)**:在编译阶段,AspectJ Weaver 可以与 Java 编译器集成,将切面代码与目标类合并,生成新的字节码。这通常通过 ajc 编译器实现。 2. **加载时织入(LTW, Load-Time ...

    apache-maven-3.8.1-bin.tar.gz

    例如,`mvn clean`可以清除目标目录,`mvn compile`将源代码编译为类文件,`mvn package`则将编译好的类打包成JAR、WAR或EAR文件。 此外,Maven还支持插件,这些插件可以扩展Maven的功能,比如生成项目文档、执行...

    Python-2.7.13-xcompile.patch

    Python-2.7.13-xcompile.patch

    ibus-table-wubi-1.2.0.20090715.tar.gz

    Before you compile the table, you must install requirements packages, to do: #yum -y install ibus-table ibus-libs ibus-table-additional ibus-devel Now you can compile it: #./configure --prefix=/usr...

    maven 下载 sqljdbc4-4.0.jar

    这样,当执行`mvn install`或`mvn compile`时,Maven会自动从指定的仓库下载并管理`sqljdbc4-4.0.jar`,无需手动操作。 使用`sqljdbc4-4.0.jar`驱动程序连接SQL Server数据库的基本步骤如下: 1. 引入JDBC驱动:在...

    qt-5.9.9-gcc485-libc217-static-qt-xcb.tar.gz

    ./configure -confirm-license -opensource -debug-and-release -static -prefix ../build -no-opengl -nomake tests -no-compile-examples -nomake examples -qt-xcb ———————————————— 原文链接:...

    Android-FFMPEG-With-IJKPLAYER-master.zip

    2.如果想支持x264和openssl请先运行compile-x264.sh,compile-openssl.sh 3.运行compile-ffmpeg.sh 4.运行compile-ijk.sh 5.最后生成ffmpeg_pic,ffmpeg_pie, libijkplayer.so, libijksdl.so, libijkffmpeg.so.

Global site tag (gtag.js) - Google Analytics