Tez: build from souce code

build from source code

 

1. download   from http://tez.apache.org/install.html

if you want to get the lattest codes through this command

#git clone https://git-wip-us.apache.org/repos/asf/tez.git

 

#tar xvf apache-tez-0.5.1-src.tar.gz

#cd apache-tez-0.5.1-src

#mvn package  -Dhadoop.version=2.3.0 -P\!hadoop24 -DskipTests

 

error:

DAGClientTimelineImpl.java:[38,33] package com.sun.jersey.api.client does not exist

solution: modify tez-api/pom.xml 

 

  <dependencies>
     <dependency>
            <groupId>com.sun.jersey</groupId>
            <artifactId>jersey-client</artifactId>
            <version>1.9</version>
      </dependency>

......

 

rerun

#mvn clean package  -Dhadoop.version=2.3.0 -P\!hadoop24 -DskipTests

 

all build jars in tez-dist/target/ dir.

-------------------------------------------

Depoly tez to hadoop clusters

1. make TEZ_HOME

#mkdir -p /home/inok/tez

#cp tez-dist/target/tez-0.5.1-minimal.tar.gz  /home/inok/tez/

#cp tez-dist/target/tez-0.5.1.tar.gz  /home/inok/tez/

#cd /home/inok/tez

#mkdir conf

#mkdir jars

#cd conf

#vi tez-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>tez.lib.uris</name>
  <value>${fs.defaultFS}/apps/tez-0.5.1/tez-0.5.1.tar.gz</value>
</property>

<property>
  <name>tez.use.cluster.hadoop-libs</name>
  <value>false</value>
</property>
</configuration>

 

#cd  ../

#tar xf tez-0.5.1-minimal.tar.gz -C  jars/

 

distribute tez dir to all cluster nodes /home/inok/

 

2. upload tez-0.5.1.tar.gz to hdfs

#hdfs dfs -mkdir -p /apps/tez-0.5.1

#hdfs dfs -put  tez-0.5.1.tar.gz  /apps/tez-0.5.1/

 

3. configure hadoop

# modify hadoop-2.3.0/etc/hadoop/mapred-site.xml

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn-tez</value>
</property>

# modify hadoop-2.3.0/etc/hadoop/hadoop-env.sh

export TEZ_CONF_DIR=/home/inok/tez/conf
export TEZ_JARS=/home/inok/tez/jars
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_CONF_DIR:$TEZ_JARS/*:$TEZ_JARS/lib/*

 

update mapred-site.xml and hadoop-env.sh  in all hadoop cluster nodes

 

4. restart hadoop clusters

#stop-all.sh

#start-all.sh

 

5. test tez

#hdfs dfs -mkdir test

# vi words

good morning 
good afternoon
thanks 
my boy

#hdfs dfs -put words test/

#hadoop jar hadoop-2.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar wordcount test/words test/output

IF you see the following reds words, congratuations, tez works.

 

 
14/11/13 09:58:58 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.131:2015
14/11/13 09:58:58 INFO client.TezClient: Submitting DAG application with id: application_1415843888183_0002
14/11/13 09:58:58 INFO client.TezClient: Tez system stage directory hdfs://192.168.0.131:2014/mr-history/am/staging/inok/.staging/job_1415843888183_0002/.tez/application_1415843888183_0002 doesn't exist and is created
14/11/13 09:58:58 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1415843888183_0002, dagName=word count
14/11/13 09:58:58 INFO impl.YarnClientImpl: Submitted application application_1415843888183_0002
14/11/13 09:58:58 INFO client.TezClient: The url to track the Tez AM: http://inok-c0:2019/proxy/application_1415843888183_0002/
14/11/13 09:58:58 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.131:2015
14/11/13 09:58:58 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.131:2015
 
-----------------

When I submit a common mapreduce job in hue through oozie, there is an error

Cannot initialize Cluster. Please check your configuration for
 mapreduce.framework.name and the correspond server addresses

 

#oozie job --oozie http://localhost:11000/oozie --config examples/apps/java-main/job.properties  -run
job: 0000003-141113143744303-oozie-inok-W

 

http://qnalist.com/questions/5187496/failing-to-run-hive-action-using-mapreduce-framework-name-yarn-tez

 

 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn-tez</value>
</property>

but  in mapred-default.xml  in hadoop-mapreduce-client-core-2.3.0.jar

<property>
  <name>mapreduce.framework.name</name>
  <value>local</value>
  <description>The runtime framework for executing MapReduce jobs.
  Can be one of local, classic or yarn.
  </description>
</property>

 

 ----
try these mothods

1. cp tez-site.xml to  etc/hadoop

2. cp all tez relavant jars to oozie share lib 's oozie dir and update to hdfs

3. check tez's example

 

 

References

http://blog.woopi.org/wordpress/?p=96

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.2/bk_installing_manually_book/content/rpm-chap-tez.html