hadoop安装与环境配置

美丽的小岛

浏览: 310651 次
性别:
来自: 大连

最近访客更多访客>>

loginboot

firstfish

ningpanyun

vctrane_007

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

hadoop

hadoop:

1.安装java,配置环境变量：export JAVA_HOME=///

export PATH=/////

注意：这个配置有效期不是很久的，要把这配置写到自己的配置文件(~/.bash_profile)

export PATH=$PATH:////////

2.下载hadoop-1.2.1, 解压：

>sudo tar -vxf hadoop-1.2.1.tar

>export HADOOP_HOME=///////

>export PATH=/////

>cd hadoop-1.2.1/conf

配置一个环境文件(hadoop-env.sh)：

把”JAVA_HOME=“注解打开，把java的路径加进去； 
export JAVA_HOME=$(/usr/libexec/java_home)

配置三个文件，hadoop-site.xml, hdfs-site.xml, mapred-site.xml;

其中：

注意fs.default.name这个字段，有可能要把主机名加入去的，mac就是这样，要不，后面的namenode运行不起来。

hadoop-site.xml,

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>

    <name>hadoop.tmp.dir</name>

    <value>~/hadoop_tmp</value>

    <description>A base for other temporary directories.</description>

  </property>

  <property>

    <name>fs.default.name</name>

    <value>hdfs://localhost(******.local):8020</value>

  </property>

</configuration>

hdfs-site.xml,

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->


<configuration>

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration> 

</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
        <name>mapred.job.tracker</name>
        <value>liangr.local:8021</value>
    </property>
    
    <property>
        <name>mapred.tasktracker.map.tasks.maximum</name>
        <value>2</value>
    </property>
    
    <property>
        <name>mapred.tasktracker.reduce.tasks.maximum</name>
        <value>2</value>
    </property>


</configuration>

配置完之后，格式化文件系统：

>hadoop namenode -format

显示为：

最后，运行：

>start-all.sh

查看：

>jps

显示：

1411 SecondaryNameNode

6740 Jps

1317 DataNode

1223 NameNode

1480 JobTracker

1576 TaskTracker

把开web浏览器：

http://localhost:50030/jobtracker.jsp

http://localhost:50070/dfshealth.jsp

NameNode '172-10-22-141.lightspeed.clmasc.sbcglobal.net:8020'

 Started: 	 Mon Sep 14 11:04:32 CST 2015
 Version: 	 1.2.1, r1503152
 Compiled: 	 Mon Jul 22 15:23:09 PDT 2013 by mattf
 Upgrades: 	 There are no upgrades in progress.

Browse the filesystem
Namenode Logs
Cluster Summary

9 files and directories, 2 blocks = 11 total. Heap Size is 123 MB / 889 MB (13%) 
  Configured Capacity	 :	 232.62 GB
 DFS Used	 :	 28 KB
 Non DFS Used	 :	 44.1 GB
 DFS Remaining	 :	 188.52 GB
 DFS Used%	 :	 0 %
 DFS Remaining%	 :	 81.04 %
 Live Nodes 	 :	 1
 Dead Nodes 	 :	 0
 Decommissioning Nodes 	 :	 0
 Number of Under-Replicated Blocks	 :	 0

liangr Hadoop Map/Reduce Administration

Quick Links State: RUNNING
Started: Mon Sep 14 11:04:33 CST 2015
Version: 1.2.1, r1503152
Compiled: Mon Jul 22 15:23:09 PDT 2013 by mattf
Identifier: 201509141104
SafeMode: OFF
Cluster Summary (Heap Size is 115 MB/889 MB)

Running Map Tasks	Running Reduce Tasks	Total Submissions	Nodes	Occupied Map Slots	Occupied Reduce Slots	Reserved Map Slots	Reserved Reduce Slots	Map Task Capacity	Reduce Task Capacity	Avg. Tasks/Node	Blacklisted Nodes	Graylisted Nodes	Excluded Nodes
0	0	0	1	0	0	0	0	2	2	4.00	0	0	0

Scheduling Information

Queue Name	State	Scheduling Information
default	running	N/A
Filter (Jobid, Priority, User, Name) 
 
Example: 'user:smith 3200' will filter by 'smith' only in the user field and '3200' in all fields
Running Jobs

none
Retired Jobs

none
Local Logs

Log directory, Job Tracker History
This is Apache Hadoop release 1.2.1