datax 编译安装包 并进行正常使用 过程中各种错误 解决

centos 系统

------------------------------------------------------------------

首先必须安装 rpmbuild

 

yum　install rpm-build

 

----------------------------

接着运行编译命令，到datax源码的rpm目录下面

rpmbuild --ba t_dp_datax_engine.spec

------------报错提示缺少 dos2unix--------------

 yum install dos2unix

 

------------接着继续执行编译命令出现一下错误-------------------------------------------------------------

Processing files: t_dp_datax_engine-1.0.0-1.noarch
error: File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/bin
error: File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/conf
error: File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/engine
error: File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/common
error: File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/libs
error: File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/logs
error: File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/jobs

RPM build errors:
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/bin
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/conf
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/engine
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/common
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/libs
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/logs
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.i386/home/taobao/datax/jobs

 

-----------------------------------

这个问题 网上已经有人提出了解决方案：

http://yangyoupeng-cn-fujitsu-com.iteye.com/blog/1832143

请直接点击跳转即可

------------------------------------------------------------------------------------------

 

-----------------接下来继续执行 出现如下错误----------------

Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64

error: Installed (but unpackaged) file(s) found:

   /home/taobao/datax/jobs/HdfsReader_to_OracleWriter.xml

   /home/taobao/datax/jobs/MysqlReader_to_HdfsWriter.xml

   /home/taobao/datax/jobs/OracleReader_to_HdfsWriter.xml

   /home/taobao/datax/jobs/OracleReader_to_MysqlWriter.xml

   /home/taobao/datax/jobs/sample.xml

 

 

RPM build errors:

    Installed (but unpackaged) file(s) found:

   /home/taobao/datax/jobs/HdfsReader_to_OracleWriter.xml

   /home/taobao/datax/jobs/MysqlReader_to_HdfsWriter.xml

   /home/taobao/datax/jobs/OracleReader_to_HdfsWriter.xml

   /home/taobao/datax/jobs/OracleReader_to_MysqlWriter.xml

   /home/taobao/datax/jobs/sample.xml

 

------------------------

该问题的方案是：

进入   cd /usr/lib/rpm

         vi   macros

------------------------ ----------------------------------

#==============================================================================

# ---- Build configuration macros.

#

# Script gets packaged file list on input and buildroot as first parameter.

# Returns list of unpackaged files, i.e. files in $RPM_BUILD_ROOT not packaged.

#

# Note: Disable (by commenting out) for legacy compatibility.

 %__check_files         %{_rpmconfigdir}/check-files %{buildroot}    

 

#

# Should unpackaged files in a build root terminate a build?

#

# Note: The default value should be 0 for legacy compatibility.

%_unpackaged_files_terminate_build      1

 

------------------------------把红色的这一行注释掉即可----------------------------------------

 

再运行编译打包命令

------------------------------------------------------------

+ /usr/lib/rpm/brp-compress

+ /usr/lib/rpm/brp-strip

+ /usr/lib/rpm/brp-strip-static-archive

+ /usr/lib/rpm/brp-strip-comment-note

Processing files: t_dp_datax_engine-1.0.0-1.noarch

Wrote: /root/rpmbuild/SRPMS/t_dp_datax_engine-1.0.0-1.src.rpm

Wrote: /root/rpmbuild/RPMS/noarch/t_dp_datax_engine-1.0.0-1.noarch.rpm

Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.OYRna1

+ umask 022

+ cd /root/rpmbuild/BUILD

+ /bin/rm -rf /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64

 

+ exit 0

-----------------------------------------------------------------------------------------------------------

成功生成 rpm包。

-----------------------------------------------------------------------------------------------------------

安装生成的rpm包，执行以下命令

rpm -ivh t_dp_datax_engine-1.0.0-1.noarch.rpm

 

-----------------------出现一下日志表示已经成功安装-------------------------------------------------------

 Preparing...                ########################################### [100%]

   1:t_dp_datax_engine-1.0.0-1.noarch ########################################### [100%]

-------------------------------------------------------------------------------------------------------------------------

 

进入 cd   /home/taobao/datax/bin  目录

运行命令 ： ./datax.py -e true

----------------------------------------------出现空指针错误---------------------------------------------------------

[root@hadoop3 bin]# ./datax.py -e true

Taobao DataX V1.0 

Exception in thread "main" java.lang.NullPointerException

        at java.util.Arrays$ArrayList.<init>(Arrays.java:2842)

        at java.util.Arrays.asList(Arrays.java:2828)

        at com.taobao.datax.engine.tools.JobConfGenDriver.getPluginDirAsList(JobConfGenDriver.java:36)

        at com.taobao.datax.engine.tools.JobConfGenDriver.filterPluginType(JobConfGenDriver.java:43)

        at com.taobao.datax.engine.tools.JobConfGenDriver.produceXmlConf(JobConfGenDriver.java:166)

        at com.taobao.datax.engine.schedule.Engine.main(Engine.java:389)

 

-----------------------------------------------------------------------------------------------------------------------------

出现该问题的原因是因为 datax安装好以后，还没有安装各种读写插件(mysql，oracle等)

所以我们现在需要安装读写插件

 

首先安装 mysqlreader插件，还是和本文开头遇见的第一个错误一样，需要修改 t_dp_datax_mysqlreader.spec 文件

-------------------------------------------红色字体就是修改后的--------------------------------------------------------

[root@hadoop3 rpm]# vi t_dp_datax_mysqlreader.spec 

summary: DataX mysqlreader can read data from mysql

Name: t_dp_datax_mysqlreader

Version: 1.0.0

Release: 1

Group: System

License: GPL

AutoReqProv: no

BuildArch: noarch

Requires: t_dp_datax_engine

 

%define dataxpath %{buildroot}/home/taobao/datax

%define vdataxpath /home/taobao/datax                  (添加这一行)

%description

DataX mysqlreader can read data from mysql

 

 

%prep

cd ${OLDPWD}/../

export LANG=zh_CN.UTF-8

ant dist

 

%build

 

%install

mkdir -p %{dataxpath}/plugins/reader/mysqlreader

 

cp ${OLDPWD}/../src/com/taobao/datax/plugins/reader/mysqlreader/ParamKey.java %{dataxpath}/plugins/reader/mysqlreader

cp ${OLDPWD}/../build/plugins/mysqlreader-1.0.0.jar %{dataxpath}/plugins/reader/mysqlreader

cp ${OLDPWD}/../build/plugins/plugins-common-1.0.0.jar %{dataxpath}/plugins/reader/mysqlreader

cp -r ${OLDPWD}/../libs/mysql-connector-java-5.1.18-bin.jar %{dataxpath}/plugins/reader/mysqlreader

cp -r ${OLDPWD}/../libs/commons-dbcp-1.4.jar %{dataxpath}/plugins/reader/mysqlreader

cp -r ${OLDPWD}/../libs/commons-pool-1.5.4.jar %{dataxpath}/plugins/reader/mysqlreader

cp -r ${OLDPWD}/../libs/commons-logging-1.1.1.jar %{dataxpath}/plugins/reader/mysqlreader

 

%files

%defattr(0755,root,root)

%{vdataxpath}/plugins/reader/mysqlreader             （替换后的）

 

%changelog

* Fri Aug 20 2010 meining

- Version 1.0.0

-------------------------------------------------------------------------------------------------------------------

 接着再运行

rpmbuild --ba t_dp_datax_mysqlreader.spec 

-----------------------------------------------------------------

Processing files: t_dp_datax_mysqlreader-1.0.0-1.noarch

Wrote: /root/rpmbuild/SRPMS/t_dp_datax_mysqlreader-1.0.0-1.src.rpm

Wrote: /root/rpmbuild/RPMS/noarch/t_dp_datax_mysqlreader-1.0.0-1.noarch.rpm

Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.EhHiFf

+ umask 022

+ cd /root/rpmbuild/BUILD

+ /bin/rm -rf /root/rpmbuild/BUILDROOT/t_dp_datax_mysqlreader-1.0.0-1.x86_64

+ exit 0

-------------------------------成功生成rpm包---------------------------------------------

 

安装该插件

------------------------------------

[root@hadoop3 sa]# rpm -ivh t_dp_datax_mysqlreader-1.0.0-1.noarch.rpm 

Preparing...                ########################################### [100%]

   1:t_dp_datax_mysqlreader ########################################### [100%]

------------------------------------------------------------------------------------------------------------------------

 

接着mysqlwriter也是按照同样的方法进行安装即可。

 

datax必须同时存在 一个reader插件和一个writer插件 才能正常运行   bin/datax.py -e true  命令，不然都会报错空指针的异常。

----------------------------------------------------------------------------------------------------------

[root@hadoop3 datax]# bin/datax.py -e true

Taobao DataX V1.0 

Data Source List :

0       mysql

Please choose [0-0]: 0

Data Destination List :

0       mysql

Please choose [0-0]: 0

Generate /home/taobao/datax/jobs/mysqlreader_to_mysqlwriter_1426588439516.xml successfully .

 

------------------------------------------------成功生成job的xml配置文件----------------------------------------------------