【Mahout三】基于Mahout CBayes算法的20newsgroup流程分析

1.Mahout环境搭建

 1.下载Mahout

http://mirror.bit.edu.cn/apache/mahout/0.10.0/mahout-distribution-0.10.0.tar.gz

 

 2.解压Mahout

 3. 配置环境变量

vim /etc/profile

export HADOOP_HOME=/home/hadoop/software/hadoop-2.5.2
export PATH=$HADOOP_HOME/bin:$PATH

export MAHOUT_HOME=/home/hadoop/software/mahout-distribution-0.10.0
export PATH=$MAHOUT_HOME/bin:$PATH

 

4. Mahout依赖于Hadoop执行MapReduce作业，不同于一般的添加Hadoop依赖的方式，Mahout不需要修改它专有的配置文件进行Hadoop配置，只要在环境变量中添加HADOOP_HOME即可

 

2. Mahout Naive Bayes算法的执行流程

1. 针对样本文件创建Sequence File
2. 将Sequence File转换为tfidf向量文件
3. 根据tdidf向量文件创建training向量文件和测试向量文件
4. 训练Naive Bayes模型，输入为训练向量文件，输出为训练模型文件
5.1 基于训练向量文件进行测试
5.2 基于测试向量文件进行测试

 

3.具体流程

1.启动Mahout 20newsgroups程序

1. 运行脚本./examples/bin/classify-20newsgroups.sh

 

2.下载数据文件并上传到HDFS中

 

1. 数据文件（20news-bydate.tar.gz）下载到/tmp/mahout-work-hadoop目录下，并解压至本地/tmp/mahout-work-hadoop/20news-all目录下，目录下的数据文件是原始的文本内容.

2. 将数据文件上传到HDFS的/tmp/mahout-work-hadoop/20news-all目录下，执行如下HDFS命令

 

/home/hadoop/software/hadoop-2.5.2/bin/hdfs dfs -put /tmp/mahout-work-hadoop/20news-all /tmp/mahout-work-hadoop/

 此时，HDFS上包含如下路径/tmp/mahout-work-hadoop/20news-all，这个目录下包含了20newsgroup的原始文件

 

3. 根据原始数据创建Sequence Files

 

1.通过执行如下的命令，为原始的数据20newsgroups创建Sequence Files

 

./bin/mahout seqdirectory -i /tmp/mahout-work-hadoop/20news-all -o /tmp/mahout-work-hadoop/20news-seq -ow

 

通过上面的命令可以看到，输入输出文件目录都是在HDFS上，产出的Sequence Files放于HDFS的/tmp/mahout-work-hadoop/20news-seq目录下，如下所示

 

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-seq
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:13 /tmp/mahout-work-hadoop/20news-seq/_SUCCESS
-rw-r--r--   1 hadoop supergroup   19202391 2015-05-22 07:13 /tmp/mahout-work-hadoop/20news-seq/part-m-00000

 根据这个显示信息，mahout seqdirectory触发了Map任务，产生的part-m-00000文件的大小大约19M

 

4. 将Sequence Files转换为Vector

 

1.执行如下命令，将Sequence Files转换为向量，

 

./bin/mahout seq2sparse -i /tmp/mahout-work-hadoop/20news-seq -o /tmp/mahout-work-hadoop/20news-vectors -lnorm -nv -wt tfidf

 

通过上面的命令可知，生成的向量文件存放于HDFS的/tmp/mahout-work-hadoop/20news-vectors目录下

 

2.HDFS的/tmp/mahout-work-hadoop/20news-vectors目录下内容是

 

 

drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:18 /tmp/mahout-work-hadoop/20news-vectors/df-count
-rw-r--r--   1 hadoop supergroup    1937084 2015-05-22 07:15 /tmp/mahout-work-hadoop/20news-vectors/dictionary.file-0
-rw-r--r--   1 hadoop supergroup    1890053 2015-05-22 07:18 /tmp/mahout-work-hadoop/20news-vectors/frequency.file-0
drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:20 /tmp/mahout-work-hadoop/20news-vectors/tf-vectors
drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:21 /tmp/mahout-work-hadoop/20news-vectors/tfidf-vectors
drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:14 /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents
drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:15 /tmp/mahout-work-hadoop/20news-vectors/wordcount

 

20news-vectors目录下有两个文件(dictionary.file-0和frequency.file.0)以及五个目录

 

2.1 df-count目录

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/df-count
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:18 /tmp/mahout-work-hadoop/20news-vectors/df-count/_SUCCESS
-rw-r--r--   1 hadoop supergroup    1890073 2015-05-22 07:18 /tmp/mahout-work-hadoop/20news-vectors/df-count/part-r-00000

 

2.2 tf-vectors目录

 

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/tf-vectors
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:20 /tmp/mahout-work-hadoop/20news-vectors/tf-vectors/_SUCCESS
-rw-r--r--   1 hadoop supergroup   28689283 2015-05-22 07:20 /tmp/mahout-work-hadoop/20news-vectors/tf-vectors/part-r-00000

 

2.3 tfidf-vectors目录

 

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/tfidf-vectors
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:21 /tmp/mahout-work-hadoop/20news-vectors/tfidf-vectors/_SUCCESS
-rw-r--r--   1 hadoop supergroup   28689283 2015-05-22 07:21 /tmp/mahout-work-hadoop/20news-vectors/tfidf-vectors/part-r-00000

 

2.4 tokenized-documents

 

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:14 /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents/_SUCCESS
-rw-r--r--   1 hadoop supergroup   27503580 2015-05-22 07:14 /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents/part-m-00000

 

2.5 wordcount

 

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/wordcount
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:15 /tmp/mahout-work-hadoop/20news-vectors/wordcount/_SUCCESS
-rw-r--r--   1 hadoop supergroup    2315037 2015-05-22 07:15 /tmp/mahout-work-hadoop/20news-vectors/wordcount/part-r-00000

 

3. 以上2个文件和目录的生成过程

 

3.1 通过Map任务生成tokenized-documents

 

SparseVectorsFromSequenceFiles: Tokenizing documents in /tmp/mahout-work-hadoop/20news-seq

 

3.2 根据Tokenized-Documents生成Term Frequency Vectors

这是一个Map Reduce任务，生成的TFV保存在wordcount目录下

 

15/05/22 07:14:54 INFO SparseVectorsFromSequenceFiles: Creating Term Frequency Vectors
15/05/22 07:14:54 INFO DictionaryVectorizer: Creating dictionary from /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents and saving at /tmp/mahout-work-hadoop/20news-vectors/wordcount

 

 3.3 计算IDF

这是一个MapReduce任务，生成的文件放置于df-count目录下

15/05/22 07:17:34 INFO SparseVectorsFromSequenceFiles: Calculating IDF

 

 

3. 训练Bayes Model

通过如下命令生成Bayes Model

./bin/mahout trainnb -i /tmp/mahout-work-hadoop/20news-train-vectors -o /tmp/mahout-work-hadoop/model -li /tmp/mahout-work-hadoop/labelindex -ow -c

 

通过命令可以看出，为Bayes模型的训练提供的输入是/tmp/mahout-work-hadoop/20news-train-vectors,输出是/tmp/mahout-work-hadoop/model