Hadoop SequcenceFile 处理多个小文件

利用sequenceFile打包多个小文件，MapFile是sequenceFile的排序形式，程序如下：

 

public class testSequenceFile {

	public static void main(String[] args) throws IOException{
		Configuration conf = new Configuration();
		
		FileSystem fs = FileSystem.get(conf);
		FileStatus[] files = fs.listStatus(new Path(args[0]));
		
		Text key = new Text();
		Text value = new Text();
		SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf, new Path(args[1]),key.getClass() , value.getClass());
		InputStream in = null;
		byte[] buffer = null;
		
		for(int i=0;i<files.length;i++){
			key.set(files[i].getPath().getName());
			in = fs.open(files[i].getPath());
			buffer = new byte[(int) files[i].getLen()];
			IOUtils.readFully(in, buffer, 0, buffer.length);
			value.set(buffer);
			IOUtils.closeStream(in);
			System.out.println(key.toString()+"\n"+value.toString());
			writer.append(key, value);
		}	
		
		IOUtils.closeStream(writer);
	}
}

 

 

 

 这里需要注意的是sequenceFile是二进制文件，cat more less 之类的命令都不能以文本形式显示顺序文件的内容，需要用到fs命令的-text选项，该选项可以查看文件的代码，检测出文件的类型并适当的转化成文本，如下图“

 

KeXie@KeXie-PC ~/hadoop-0.20.2
$ hadoop fs -cat soutput
SEQorg.apache.hadoop.io.Textorg.apache.hadoop.io.Text*org.apache.hadoop.io.compress.DefaultCodec▒A▒=▒▒=U▒2▒,a.txtx▒▒▒L▒,*▒▒,,M▒▒▒▒<▒A#b.txtx▒▒L▒H▒▒▒y▒\▒▒▒y▒\@6n:c.txtx▒▒+*▒▒,,M▒▒▒▒%▒▒
KeXie@KeXie-PC ~/hadoop-0.20.2
$ hadoop fs -text soutput
a.txt   xie     chen     liang
quan
b.txt   chen    chen    wen
an
wen
c.txt   mo     an
an