部署hadoop

大数据这两年也是大热，自己虽然在工作方面没什么交集，但也通过公司的一些资源对这方面进行了了解，包括hadoop、spark的部署及简单用例，以下部分内容都是通过公司服务器实现了操作，部署hadoop的大致流程有下面几个步骤：

1.安装系统（centos 7.0）

2.安装ssh无密钥登录服务，实现master控制其他slave无需口令

3.安装jdk

4.安装配置hadoop

5.设置防火墙端口

需要注意centos 7.0的命令变化比较大，主要用到下面几个命令变化

如服务的开始停止：

     systemctl start firewalld.service 关闭防火墙

     systemctl disable firewalld.service 开机不启动

     systemctl  start sshd.service 

具体流程：

adoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html

 

配置master的/etc/hosts

192.168.0.7 mesos-master hadoop-master

192.168.0.20 mesos-slave-1 hadoop-slave-1

192.168.0.8 mesos-slave-2 hadoop-slave-2

 

 

Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:

  $ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

  $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
  $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

接着在master上生成密钥并配置SSH无密码登录

具体步骤如下：

1、 进入.ssh文件夹

2、 ssh-keygen -t  rsa 之后一路回车（产生秘钥）

3、 把id_rsa.pub 追加到授权的 key 里面去（cat id_rsa.pub >> authorized_keys）

4、 重启 SSH 服务命令使其生效

3.3 将生成的authorized_keys文件拷贝到两台slave主机相同的文件夹下，命令如下：

 

scp authorized_keys slave1:~/.ssh/

scp authorized_keys slave2:~/.ssh/

 

Hadoop configuration is driven by two types of important configuration files:

• Read-only default configuration - core-default.xml, hdfs-default.xml, yarn-default.xml and mapred-default.xml.
• Site-specific configuration - conf/core-site.xml, conf/hdfs-site.xml, conf/yarn-site.xml andconf/mapred-site.xml.

 

Configuration

Use the following:

etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

etc/hadoop/hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
</configuration>

Hadoop Startup

To start a Hadoop cluster you will need to start both the HDFS and YARN cluster.

Format a new distributed filesystem:

$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>

Start the HDFS with the following command, run on the designated NameNode:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode

Run a script to start DataNodes on all slaves:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode

Start the YARN with the following command, run on the designated ResourceManager:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

Run a script to start NodeManagers on all slaves:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager

Start a standalone WebAppProxy server. If multiple servers are used with load balancing it should be run on each of them:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start proxyserver --config $HADOOP_CONF_DIR

Start the MapReduce JobHistory Server with the following command, run on the designated server:

$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR

Hadoop Shutdown

Stop the NameNode with the following command, run on the designated NameNode:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode

Run a script to stop DataNodes on all slaves:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode

Stop the ResourceManager with the following command, run on the designated ResourceManager:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager

Run a script to stop NodeManagers on all slaves:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager

Stop the WebAppProxy server. If multiple servers are used with load balancing it should be run on each of them:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh stop proxyserver --config $HADOOP_CONF_DIR

Stop the MapReduce JobHistory Server with the following command, run on the designated server:

$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR

Operating the Hadoop Cluster

Once all the necessary configuration is complete, distribute the files to the HADOOP_CONF_DIR directory on all the machines.

This section also describes the various Unix users who should be starting the various components and uses the same Unix accounts and groups used previously:

 

集群方式启动HDFS：

hadoop-2.2.0$ sbin/start-dfs.sh

hadoop-2.2.0$ sbin/stop-dfs.sh

 

 

单独启动和停止datanode：

 

./sbin/hadoop-daemon.sh start datanode

 

./sbin/hadoop-daemon.sh stop datanode

 

 

Hadoop Startup

To start a Hadoop cluster you will need to start both the HDFS and YARN cluster.

Format a new distributed filesystem as hdfs:

[hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>

Start the HDFS with the following command, run on the designated NameNode as hdfs:

[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode

Run a script to start DataNodes on all slaves as root with a special environment variable HADOOP_SECURE_DN_USER set to hdfs:

[root]$ HADOOP_SECURE_DN_USER=hdfs $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode

Start the YARN with the following command, run on the designated ResourceManager as yarn:

[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

Run a script to start NodeManagers on all slaves as yarn:

[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager

Start a standalone WebAppProxy server. Run on the WebAppProxy server as yarn. If multiple servers are used with load balancing it should be run on each of them:

[yarn]$ $HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIR

Start the MapReduce JobHistory Server with the following command, run on the designated server as mapred:

[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR

Hadoop Shutdown

Stop the NameNode with the following command, run on the designated NameNode as hdfs:

[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode

Run a script to stop DataNodes on all slaves as root:

[root]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode

Stop the ResourceManager with the following command, run on the designated ResourceManager as yarn:

[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager

Run a script to stop NodeManagers on all slaves as yarn:

[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager

Stop the WebAppProxy server. Run on the WebAppProxy server as yarn. If multiple servers are used with load balancing it should be run on each of them:

[yarn]$ $HADOOP_YARN_HOME/bin/yarn stop proxyserver --config $HADOOP_CONF_DIR

Stop the MapReduce JobHistory Server with the following command, run on the designated server as mapred:

[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR

Web Interfaces

Once the Hadoop cluster is up and running check the web-ui of the components as described below:

Daemon Web Interface Notes

NameNode	http://nn_host:port/	Default HTTP port is 50070.
ResourceManager	http://rm_host:port/	Default HTTP port is 8088.
MapReduce JobHistory Server	http://jhs_host:port/	Default HTTP port is 19888.

将hadoop.tmp.dir所指定的目录删除。

（3）重新执行命令：hadoop namenode -format

参考内容：

http://hadoop.apache.org/docs/r2.4.1/ 

http://www.cnblogs.com/xia520pi/category/346943.html