Linux下编译Hadoop的Eclipse插件

虽然使用Java编写MapReduce作业有点繁琐，但是对于刚入门hadoop的新手来说，有利于理解hadoop的一些底层实现，这无疑也是一种巨大的收获。目前大部分的MapReduce作业都是使用Hive，Pig完成的，当然也有用其他语言实现的，通过使用HadoopStreaming的方式，用原生Java写MR作业的只有很少一部分。在Hadoop2.x中，已经不支持eclipse的插件了，不过可以在配置的XML文件里，加入远程监控的端口，来监控hadoop集群。


由于1.0之后hadoop已经不自带eclipse的插件包了，所以得需要我们自己编译源码生成插件包，建议在Linux下编译，Windows下可能有时候会报一些莫名其妙的错误，先介绍下散仙的环境，Centos6.4的版本，hadoop1.2.0的版本，
hadoop的目录在/root/hadoop-1.2.0下面
eclipse的目录在/root/eclipse下面
总结一下如下的四步来完成编译eclipse插件的过程

步骤	操作
1	进入/root/hadoop-1.2.0/src/contrib/eclipse-plugin下面，修改build.xml
2	进入/root/hadoop-1.2.0/src/contrib下面，修改build-contrib.xml
3	/root/hadoop-1.2.0/src/contrib/eclipse-plugin/META-INF下面，修改MANIFEST.MF文件
4	使用shell命令进入/root/hadoop-1.2.0/src/contrib/eclipse-plugin下面，执行ant命令进行构建

下面散仙就来详细介绍下，每一步具体怎么操作，在第一步里我们需要向build.xml里面，加入的内容有
1,eclipse的根目录.
2.hadoop的版本号.
3.hadoop的一些引用包.
4，在javac里加入  includeantruntime="on".
内容如下所示：

  <!-- 1,2步的配置  -->
  <property name="eclipse.home" location="/root/eclipse"/>
  <property name="version" value="1.2.0"/>
  <!--第3步的配置  -->
  <copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar"  tofile="${build.dir}/lib/commons-cli.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-configuration-1.6.jar"  tofile="${build.dir}/lib/commons-configuration.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar"  tofile="${build.dir}/lib/commons-httpclient.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-lang-2.4.jar"  tofile="${build.dir}/lib/commons-lang.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar"  tofile="${build.dir}/lib/jackson-core-asl.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar"  tofile="${build.dir}/lib/jackson-mapper-asl.jar" verbose="true"/>
 <!--第4步的配置  -->
 includeantruntime="on">

散仙会在下文贴出整个XML，现在我们看第二大步的配置，修改build-contrib.xml，内容如下：

 <!--  原始的为off -->
<property name="javac.deprecation" value="on"/>

第三步的修改MANIFEST.MF内容如下：

Bundle-ClassPath: classes/,lib/commons-cli.jar,lib/commons-httpclient.jar,lib/hadoop-core.jar,lib/jackson-mapper-asl.jar,lib/commons-configuration.jar,lib/commons-lang.jar,lib/jackson-core-asl.jar

第四步执行命令：截图如下：


最后，插件类生成的目录，截图如下：




下面散仙给出所有配置的全部信息，以供参考，下图的是build.xml里面的内容

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->

             <!-- build.xml -->
<project default="jar" name="eclipse-plugin">

  <import file="../build-contrib.xml"/>

   
  <property name="eclipse.home" location="/root/eclipse"/>
  <property name="version" value="1.2.0"/>
  
  <path id="eclipse-sdk-jars">
    <fileset dir="${eclipse.home}/plugins/">
      <include name="org.eclipse.ui*.jar"/>
      <include name="org.eclipse.jdt*.jar"/>
      <include name="org.eclipse.core*.jar"/>
      <include name="org.eclipse.equinox*.jar"/>
      <include name="org.eclipse.debug*.jar"/>
      <include name="org.eclipse.osgi*.jar"/>
      <include name="org.eclipse.swt*.jar"/>
      <include name="org.eclipse.jface*.jar"/>

      <include name="org.eclipse.team.cvs.ssh2*.jar"/>
      <include name="com.jcraft.jsch*.jar"/>
    </fileset> 
  </path>

  <!-- Override classpath to include Eclipse SDK jars -->
  <path id="classpath">
    <pathelement location="${build.classes}"/>
    <pathelement location="${hadoop.root}/build/classes"/>
	 <fileset dir="${hadoop.root}">
        <include name="**/*.jar" />
    </fileset>
    <path refid="eclipse-sdk-jars"/>
  </path>

  <!-- Skip building if eclipse.home is unset. -->
  <target name="check-contrib" unless="eclipse.home">
    <property name="skip.contrib" value="yes"/>
    <echo message="eclipse.home unset: skipping eclipse plugin"/>
  </target>

 <target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">
    <echo message="contrib: ${name}"/>
    <javac
     encoding="${build.encoding}"
     srcdir="${src.dir}"
     includes="**/*.java"
     destdir="${build.classes}"
     debug="${javac.debug}"
     deprecation="${javac.deprecation}"
	 includeantruntime="on">
     <classpath refid="classpath"/>
    </javac>
  </target>

  <!-- Override jar target to specify manifest -->
  <target name="jar" depends="compile" unless="skip.contrib">
    <mkdir dir="${build.dir}/lib"/>
  <!--  <copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/> -->
   <!--  <copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>-->
  <copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar"  tofile="${build.dir}/lib/commons-cli.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-configuration-1.6.jar"  tofile="${build.dir}/lib/commons-configuration.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar"  tofile="${build.dir}/lib/commons-httpclient.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-lang-2.4.jar"  tofile="${build.dir}/lib/commons-lang.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar"  tofile="${build.dir}/lib/jackson-core-asl.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar"  tofile="${build.dir}/lib/jackson-mapper-asl.jar" verbose="true"/>


  <jar
      jarfile="${build.dir}/hadoop-${name}-${version}.jar"
      manifest="${root}/META-INF/MANIFEST.MF">
      <fileset dir="${build.dir}" includes="classes/ lib/"/>
      <fileset dir="${root}" includes="resources/ plugin.xml"/>
    </jar>
  </target>

</project>

build-contrib.xml的核心配置如下：只有一个地方，notepad++，定位截图如下：

  <property name="javac.deprecation" value="on"/>

下面是MANIFEST.MF里面的内容

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: MapReduce Tools for Eclipse
Bundle-SymbolicName: org.apache.hadoop.eclipse;singleton:=true
Bundle-Version: 0.18
Bundle-Activator: org.apache.hadoop.eclipse.Activator
Bundle-Localization: plugin
Require-Bundle: org.eclipse.ui,
 org.eclipse.core.runtime,
 org.eclipse.jdt.launching,
 org.eclipse.debug.core,
 org.eclipse.jdt,
 org.eclipse.jdt.core,
 org.eclipse.core.resources,
 org.eclipse.ui.ide,
 org.eclipse.jdt.ui,
 org.eclipse.debug.ui,
 org.eclipse.jdt.debug.ui,
 org.eclipse.core.expressions,
 org.eclipse.ui.cheatsheets,
 org.eclipse.ui.console,
 org.eclipse.ui.navigator,
 org.eclipse.core.filesystem,
 org.apache.commons.logging
Eclipse-LazyStart: true
Bundle-ClassPath: classes/,lib/commons-cli.jar,lib/commons-httpclient.jar,lib/hadoop-core.jar,lib/jackson-mapper-asl.jar,lib/commons-configuration.jar,lib/commons-lang.jar,lib/jackson-core-asl.jar
Bundle-Vendor: Apache Hadoop

至此，已经ant编译完毕，截图如下



启动eclipse后，截图如下：





至此，插件已经完美生成，并运行，因为受上传限制的影响，所以散仙在这里上传不了这个jar包，有需要的朋友们，可以联系散仙。

评论

2 楼 gainyray 2014-01-02  

弄好了，将hadoop/hadoop-core-1.2.1.jar copy到hadoop/build下 然后执行命令： ant jar 就编译好了

1 楼 gainyray 2014-01-02  

楼主，最后编译得时候出现java.net.ConnectException: Connection timed out.
然后找到用ant compile -Doffline=true 这个命令编译，显示build sucessful.但是问题是在指定得路径下找不到编译好的jar包，楼主有时间看看，先谢谢了。。。