hadoop UnsupportedFileSystemException 问题

使用hdfs操作文件的时候遇到以下两个异常：

1.java.io.IOException: No FileSystem for scheme: hdfs

2.org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "file"

 

类似No FileSystem for scheme: hdfs （file）原因是jar包或打包问题。

依赖的jar包是hadoop-commons 和 hadoop-hdfs

 

Different JARs (hadoop-commons for LocalFileSystem, hadoop-hdfs for DistributedFileSystem) each contain a different file called org.apache.hadoop.fs.FileSystem in their META-INFO/servicesdirectory. This file lists the canonical classnames of the filesystem implementations they want to declare (This is called a Service Provider Interface implemented via java.util.ServiceLoader, see org.apache.hadoop.FileSystem line 2622).

When we use maven-assembly-plugin, it merges all our JARs into one, and all META-INFO/services/org.apache.hadoop.fs.FileSystem overwrite each-other. Only one of these files remains (the last one that was added). In this case, the FileSystem list from hadoop-commonsoverwrites the list from hadoop-hdfs, so DistributedFileSystem was no longer declared.

 

解决方法如下：

1.通过配置方式：

		// 这个解决hdfs问题
		hadoopConf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
		// 这个解决本地file问题
		hadoopConf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());

 2.通过打包插件方式

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>2.3</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <transformers>
          <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
        </transformers>
      </configuration>
    </execution>
  </executions>
</plugin>