新版本Hadoop安装过程

1、进程差异

新版本Hadoop
SecondaryNameNode  NodeManager   NameNode  DataNode  ResourceManager

旧版本Hadoop

SecondaryNamenode，JobTracker，NameNode，DataNode，TraskTracker五个进程

 

安装新版本Haddop注意事项，安装过程同hadoop低版本

loaded: ssh: Could not resolve hostname loaded: Name or service not known
It's: ssh: Could not resolve hostname It's: Name or service not known
try: ssh: Could not resolve hostname try: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
You: ssh: Could not resolve hostname You: Name or service not known
that: ssh: Could not resolve hostname that: Name or service not known
might: ssh: Could not resolve hostname might: Name or service not known
you: ssh: Could not resolve hostname you: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
to: ssh: Could not resolve hostname to: Name or service not known
highly: ssh: Could not resolve hostname highly: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known

解决方案：

export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/bigdata/hadoop272/lib/native
export HADOOP_OPTS="-Djava.library.path=/opt/bigdata/hadoop272/lib"

 

 

Maven依赖的包一起打包进jar包。使用maven-assembly打成jar后，在没有打成jar包的工程中，是可以正常运行的，通过长时间的观察和分析，发现与hadoop filesystem相关的包有两个，分别是：hadoop-hdfs-2.7.1.jar和hadoop-common-2.7.1.jar，这两个包 的META-INF中的services目录下都有,org.apache.hadoop.fs.FileSystem这个文件。使用Maven-assembly-plugin的时候，会将所有依赖的包unpack，然后在pack，这样就会出现，同样的文件被覆盖的情况

 

在函数中使用了hdfs://IP : port的schema，而在生成的最终jar包中，无法找到这个schema的实现。所以就抛出了

      java.io.IOException: No FileSystem for scheme: hdfs

      解决方案是，在设置hadoop的配置的时候，显示设置这个类："org.apache.hadoop.hdfs.DistributedFileSystem：

      configuration.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");