`
kabike
  • 浏览: 608687 次
  • 性别: Icon_minigender_1
  • 来自: 大连
社区版块
存档分类
最新评论

编译hadoop 2.3的native library

阅读更多
用了hadoop 2.3 以后,每次启动都出现
引用
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable


而且hadoop的cache和短路读(Short-Circuit Local Reads)都需要native library的支持

但是它自带的native library是32位的,需要自己编译适合自己的.
编译方法是
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/NativeLibraries.html

编译过程中可能出现
引用

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.3.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1]


这时需要在系统中安装protobuf
从http://code.google.com/p/protobuf/downloads/list下载源码,
./configure
make
make install
即可.
需要安装cmake,可以通过yum install cmake安装

编译过程中还出现

引用
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec dir="/usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:136 in /usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1
around Ant part ...<exec dir="/usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:136 in /usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 1
around Ant part ...<exec dir="/usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:136 in /usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: /usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml:4: exec returned: 1
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646)
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327)
... 21 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common


这样的问题

发现是少了几个必须的包的开发包.比如装了zlib和openssl,但实际需要的是zlib-devel和openssl-devel

最后运行mvn package -Pdist,native -Dskiptests -Dtar
在test阶段各种失败
这个应该是hadoop的bug,skiptests在native profile激活时没起作用
https://issues.apache.org/jira/browse/HADOOP-8480
最后走投无路灵机一动,运行mvn compile -Pdist,native -Dskiptests -Dtar
在./hadoop-common-project/hadoop-common/target/native/target/usr/local/lib
中找到了想要的libhadoop.so

HADOOP-8480的状态是已在2.0.2-alpha解决,但是hadoop2.3中还是出现,愤怒的在他们下面留言了
分享到:
评论

相关推荐

    在linux中编译64位的 hadoop 2.4.1native

    Once you installed the prerequisite packages use the standard hadoop pom.xml file and pass along the native flag to build the native hadoop library: $ mvn package -Pdist,native -Dskiptests -Dtar You...

    hadoop-native-64-2.7.0.tar

    解决:Unable to load native-hadoop library for your platform 问题 原因: Apache提供的hadoop本地库是32位的,而在64位的服务器上就会有问题,因此需要自己编译64位的版本。 将准备好的64位的lib包解压到已经...

    hadoop 2.8.0 macOS native library

    安装hadoop启动之后总有警告:Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 原因: Apache提供的hadoop本地库是32位的,而在64位的服务器上就会有问题...

    hadoop321 native lib for Catalina.zip

    根据hadoop3.2.1源码编译,native library,native库依赖的第三方库主要有openssl 1.1,protobuf 2.5.0,第三方库未包含,如果有缺失的,例如zlib等,可通过homebrew自行安装,用最新版即可,jdk8,hadoop完整包可用...

    64位 macOS High Sierra 10.13.2 编译的hadoop 2.8.3 native

    编译Hadoop native库的过程涉及多个步骤,包括配置环境、设置编译选项、解决依赖问题以及执行编译和测试。在macOS上,这可能涉及到安装Xcode工具、Homebrew等包管理器,以及安装必要的开发库,如OpenSSL、Java开发...

    hadoop2.8.1 native for macOS10

    macOS下使用hadoop2.8.1时, 执行hadoop fs 命令(如...解决方法是下载hadoop源代码重新编译, 但中间需要下载安装很多第三方工具, 改资源是在macOS下重新编译后得到的native, 将$HADOOP_HOME/lib/native文件夹替换掉即可.

    Native Hadoop3.2.1 Library 64位编译

    Native Hadoop3.2.1 Library 64位编译,大数据本地库. Hadoop是使用Java语言开发的,但是有一些需求和操作并不适合使用java,所以就引入了本地库(Native Libraries)的概念,通过本地库,Hadoop可以更加高效地执行...

    hadoop-2.6.0编译好的64bit的native库

    在64位系统上运行Hadoop 2.6.0会...这是因为hadoop默认编译好的native库是32bit的,所以要重新编译hadoop2.6的源代码,获取64位native库,将其复制到原来安装的hadoop的native目录中替代原来的类库,然后重新运行即可。

    hadoop-3.1.3-mac-native_lib.zip

    本压缩包文件“hadoop-3.1.3-mac-native_lib.zip”专门针对macOS系统,包含了在macOS环境下编译Hadoop 3.1.3源码后得到的本地库(native library)。这些本地库对于解决macOS用户在安装或运行Hadoop时可能出现的因...

    hadoop 2.6 native snappy

    在Hadoop 2.6版本中,集成Native Snappy需要进行编译,这个过程可能比较耗时,因为涉及到多个步骤,包括获取源代码、配置编译环境、解决依赖关系、编译源码以及安装生成的库文件。这个压缩包文件“native”可能包含...

    hadoop2.4.1 native x64 编译库

    hadoop2.4.1 native x64 编译库 hadoop2.4.1 native 库默认是32为编译, 这个64位的库能解决“util.NativeCodeLoader: Unable to load native-hadoop library for your platform”这个警告的问题

    hadoop-native-64-2.4.1

    解决:Unable to load native-hadoop library for your platform 问题 原因: Apache提供的hadoop本地库是32位的,而在64位的服务器上就会有问题,因此需要自己编译64位的版本。 将准备好的64位的lib包解压到已经...

    hadoop-2.8.1已经编译好的64bit的native库

    :WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform 而hadoop默认编译好的native库是32bit的,所以需要重新编译。现将编译好的hadoop-2.8.1的native库分享给大家,替代原来的库...

    hadoop-native-64-2.5.0.tar

    解决:Unable to load native-hadoop library for your platform 问题 原因: Apache提供的hadoop本地库是32位的,而在64位的服务器上就会有问题,因此需要自己编译64位的版本。 将准备好的64位的lib包解压到已经...

    hadoop2.x native library64位

    标题提到的“hadoop2.x native library64位”正是为了解决这个问题,它是专门为64位操作系统编译的Hadoop本地库。 通常,Hadoop的本地库包含了Java Native Interface (JNI) 库,这些库实现了Hadoop在Java层无法直接...

    hadoop-native-64-2.6.0.tar

    解决:Unable to load native-hadoop library for your platform 问题 原因: Apache提供的hadoop本地库是32位的,而在64位的服务器上就会有问题,因此需要自己编译64位的版本。 将准备好的64位的lib包解压到已经...

    centos7 64位下编译Hadoop2.7.2源码库文件

    以下是关于如何在CentOS 7 64位系统上编译Hadoop 2.7.2源码库文件的详细步骤及相关的知识点: 1. **环境准备**:首先确保你的系统安装了必要的开发工具,如GCC、Java开发工具包(JDK)、Maven和Git。可以使用以下...

    hadoop-native-64-2.5.2.tar

    解决:Unable to load native-hadoop library for your platform 问题 原因: Apache提供的hadoop本地库是32位的,而在64位的服务器上就会有问题,因此需要自己编译64位的版本。 将准备好的64位的lib包解压到已经...

    Ubuntu下hadoop-2.5.2编译好的64bit的native库

    6. **环境变量设置**:确保`HADOOP_LIBRARY_PATH`环境变量已经设置为`/usr/local/hadoop/lib/native`,如果没有,需要添加到`.bashrc`或`.bash_profile`文件中,并执行`source ~/.bashrc`或`source ~/.bash_profile`...

    hadoop-2.9.2的64位本地库

    本地库(Native Libraries)在Hadoop中扮演着关键角色,它们主要是C++编写的,用于提升Hadoop的性能,尤其是在文件系统操作和内存管理等方面。这些库包括如libhdfs和libhadoop等,它们提供了与操作系统的直接交互,...

Global site tag (gtag.js) - Google Analytics