- 浏览: 2188708 次
- 性别:
- 来自: 北京
文章分类
- 全部博客 (682)
- 软件思想 (7)
- Lucene(修真篇) (17)
- Lucene(仙界篇) (20)
- Lucene(神界篇) (11)
- Solr (48)
- Hadoop (77)
- Spark (38)
- Hbase (26)
- Hive (19)
- Pig (25)
- ELK (64)
- Zookeeper (12)
- JAVA (119)
- Linux (59)
- 多线程 (8)
- Nutch (5)
- JAVA EE (21)
- Oracle (7)
- Python (32)
- Xml (5)
- Gson (1)
- Cygwin (1)
- JavaScript (4)
- MySQL (9)
- Lucene/Solr(转) (5)
- 缓存 (2)
- Github/Git (1)
- 开源爬虫 (1)
- Hadoop运维 (7)
- shell命令 (9)
- 生活感悟 (42)
- shell编程 (23)
- Scala (11)
- MongoDB (3)
- docker (2)
- Nodejs (3)
- Neo4j (5)
- storm (3)
- opencv (1)
最新评论
-
qindongliang1922:
粟谷_sugu 写道不太理解“分词字段存储docvalue是没 ...
浅谈Lucene中的DocValues -
粟谷_sugu:
不太理解“分词字段存储docvalue是没有意义的”,这句话, ...
浅谈Lucene中的DocValues -
yin_bp:
高性能elasticsearch ORM开发库使用文档http ...
为什么说Elasticsearch搜索是近实时的? -
hackWang:
请问博主,有用solr做电商的搜索项目?
Solr中Group和Facet的用法 -
章司nana:
遇到的问题同楼上 为什么会返回null
Lucene4.3开发之第八步之渡劫初期(八)
之前散仙也用过eclipse直接向hadoop提交MR作业,也提交成功过,这次换了集群环境,提交作业时发现几个异常,特此整理一下,以防后面再出现类似问题。
主要的问题的有2个:
第一个问题,在win7上的eclipse向hadoop提交作业时,没有权限,异常信息如下:
第二个问题是提交上的MR作业,久久不能开始执行,但是如果随机提交到master上执行,那么可以正常执行,如果提交到slave机器上,那么就会一直处于阻塞状态,日志信息如下:
下面开始详细解说这两个问题的原因:
第一个问题由于没有权限的问题,异常信息提示很明确,举个例子,如果你在linux上使用的hadoop账户装的hadoop(关于hadoop的用户名可以在core-site.xml里面配置)
ok,是个hadoop账户,如果你在win7上提交,默认使用的用户名是你机器的名字,就如本例一样,散仙的机器名叫qindongliang,所以在提交任务时,hadoop权限认证就发现,有别的用户向这里提交作业,然后再MR还没跑起来时,就直接拒绝验证通过了,所以就出现了文章开始前的那个错误,好了,知道原因了,我们就该思考一下如何解决这个问题:
方法主要有6种:
(1),更改linux上hadoop集群的名字为qindongliang
(2),更改hadoop的hdfs所在的目录的权限为hadoop fs -chmod 777 /user/hadoop
(3),关闭HDFS的权限认证机制,将dfs.permissions修改为False(经测试,无效)
(4),更改Windows7的系统用户名为hadoop
(5),在Win7上的环境变量中加入HADOOP_USER_NAME并配置在linux上对应的用户名即可
(6),在提交程序里通过代码临时设置指定HADOOP_USER_NAME的名字和linux上的一致
分析上面的方法,发现,前两种是操作linux改变,相当于操作服务端,后面3种,是操作的客户端windows7,抱着能不改变服务端的原则,就不改变,推荐在客户端更改,散仙用的是最后一种方法在程序指定用户名,如果大家觉得麻烦,可以直接在环境变量里,更改,不过更改后需要重启eclipse,当然你就可以永久使用这个名字,作为hadoop的提交名了。
散仙在程序指定hadoop的用户名比较灵活,代码如下:
上面的这段代码加在main方法的第一行即可
下面看下第二个问题,具体的描述如下:
当 MR ApplicationMaster在master机器上启动时,MR程序跑得很好。
当 MR ApplicationMaster在slave机器上启动时,MR程序僵住。
不会显示任何的MapReduce执行进度,而且查看各个log信息,没有错误的提示,有的只是一直打印,如下的info信息:
上面的这个信息是由于host解析导致的,解决办法如下,在提交的代码里,加上如下代码:
如果没有注入调度地址,NodeManager会默认为0.0.0.0:8030。如果MR ApplicationMaster在 master机器上启动,0.0.0.0:8030 对应的调度器地址 恰好在本机;否则,在slave机器上0.0.0.0:8030 是找不到调度器的,因为调度器必须在master机器上。
知道了,这个原因,我们在代码里加上调度器的连接地址,即可!
主要的问题的有2个:
第一个问题,在win7上的eclipse向hadoop提交作业时,没有权限,异常信息如下:
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=qindongliang, access=EXECUTE, inode="/tmp":search:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:187) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:150) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5185) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5167) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:5123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermissionInt(FSNamesystem.java:1338) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1317) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:528) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:348) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59576) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at $Proxy9.setPermission(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at $Proxy9.setPermission(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setPermission(ClientNamenodeProtocolTranslatorPB.java:277) at org.apache.hadoop.hdfs.DFSClient.setPermission(DFSClient.java:2045) ... 16 more
第二个问题是提交上的MR作业,久久不能开始执行,但是如果随机提交到master上执行,那么可以正常执行,如果提交到slave机器上,那么就会一直处于阻塞状态,日志信息如下:
2014-10-31 17:48:08,453 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1414748532081_0002 2014-10-31 17:48:08,457 INFO [Socket Reader #1 for port 37494] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 37494 2014-10-31 17:48:08,465 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2014-10-31 17:48:08,468 INFO [IPC Server listener on 37494] org.apache.hadoop.ipc.Server: IPC Server listener on 37494: starting 2014-10-31 17:48:08,504 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2014-10-31 17:48:08,504 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2014-10-31 17:48:08,504 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2014-10-31 17:48:08,560 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030 2014-10-31 17:48:14,580 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:15,583 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:16,587 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:17,590 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:18,592 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:19,595 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:20,597 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:21,602 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:22,606 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:23,608 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:54,621 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:55,624 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:56,626 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:57,628 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:58,631 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:48:59,633 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:00,635 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:01,638 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:02,641 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:03,643 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:34,653 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:35,655 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:36,657 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is Re
下面开始详细解说这两个问题的原因:
第一个问题由于没有权限的问题,异常信息提示很明确,举个例子,如果你在linux上使用的hadoop账户装的hadoop(关于hadoop的用户名可以在core-site.xml里面配置)
<property> <name>hadoop.http.staticuser.user</name> <value>hadoop</value> </property>
ok,是个hadoop账户,如果你在win7上提交,默认使用的用户名是你机器的名字,就如本例一样,散仙的机器名叫qindongliang,所以在提交任务时,hadoop权限认证就发现,有别的用户向这里提交作业,然后再MR还没跑起来时,就直接拒绝验证通过了,所以就出现了文章开始前的那个错误,好了,知道原因了,我们就该思考一下如何解决这个问题:
方法主要有6种:
(1),更改linux上hadoop集群的名字为qindongliang
(2),更改hadoop的hdfs所在的目录的权限为hadoop fs -chmod 777 /user/hadoop
(3),关闭HDFS的权限认证机制,将dfs.permissions修改为False(经测试,无效)
(4),更改Windows7的系统用户名为hadoop
(5),在Win7上的环境变量中加入HADOOP_USER_NAME并配置在linux上对应的用户名即可
(6),在提交程序里通过代码临时设置指定HADOOP_USER_NAME的名字和linux上的一致
分析上面的方法,发现,前两种是操作linux改变,相当于操作服务端,后面3种,是操作的客户端windows7,抱着能不改变服务端的原则,就不改变,推荐在客户端更改,散仙用的是最后一种方法在程序指定用户名,如果大家觉得麻烦,可以直接在环境变量里,更改,不过更改后需要重启eclipse,当然你就可以永久使用这个名字,作为hadoop的提交名了。
散仙在程序指定hadoop的用户名比较灵活,代码如下:
System.setProperty("HADOOP_USER_NAME", "hadoop");
上面的这段代码加在main方法的第一行即可
下面看下第二个问题,具体的描述如下:
当 MR ApplicationMaster在master机器上启动时,MR程序跑得很好。
当 MR ApplicationMaster在slave机器上启动时,MR程序僵住。
不会显示任何的MapReduce执行进度,而且查看各个log信息,没有错误的提示,有的只是一直打印,如下的info信息:
2014-10-31 17:49:38,661 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:39,663 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:40,665 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:41,668 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:42,671 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:49:43,673 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:14,684 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:15,687 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:16,689 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:17,691 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:18,692 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:19,695 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:20,699 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:21,702 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:22,705 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:23,707 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:54,717 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:55,719 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:56,721 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2014-10-31 17:50:57,723 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 3 time(s); retry polic
上面的这个信息是由于host解析导致的,解决办法如下,在提交的代码里,加上如下代码:
conf.set("yarn.resourcemanager.scheduler.address", "192.168.223.163:8030");
如果没有注入调度地址,NodeManager会默认为0.0.0.0:8030。如果MR ApplicationMaster在 master机器上启动,0.0.0.0:8030 对应的调度器地址 恰好在本机;否则,在slave机器上0.0.0.0:8030 是找不到调度器的,因为调度器必须在master机器上。
知道了,这个原因,我们在代码里加上调度器的连接地址,即可!
发表评论
-
Apache Flink在阿里的使用(译)
2019-02-21 21:18 1209Flink是未来大数据实时 ... -
计算机图形处理的一些知识
2018-04-25 17:46 1234最近在搞opencv来做一些 ... -
如何在kylin中构建一个cube
2017-07-11 19:06 1282前面的文章介绍了Apache Kylin的安装及数据仓 ... -
Apache Kylin的入门安装
2017-06-27 21:27 2148Apache Kylin™是一个开源的分布式分析引擎,提供 ... -
ES-Hadoop插件介绍
2017-04-27 18:07 1996上篇文章,写了使用spark集成es框架,并向es写入数据,虽 ... -
如何在Scala中读取Hadoop集群上的gz压缩文件
2017-04-05 18:51 2138存在Hadoop集群上的文件,大部分都会经过压缩,如果是压缩 ... -
如何收集项目日志统一发送到kafka中?
2017-02-07 19:07 2798上一篇(http://qindongliang.iteye. ... -
Hue+Hive临时目录权限不够解决方案
2016-06-14 10:40 4725安装Hue后,可能会分配多个账户给一些业务部门操作hive,虽 ... -
Hadoop的8088页面失效问题
2016-03-31 11:21 4458前两天重启了测试的hadoop集群,今天访问集群的8088任 ... -
Hadoop+Hbase集群数据迁移问题
2016-03-23 21:00 2528数据迁移或备份是任何 ... -
如何监控你的Hadoop+Hbase集群?
2016-03-21 16:10 4924前言 监控hadoop的框架 ... -
Logstash与Kafka集成
2016-02-24 18:44 11650在ELKK的架构中,各个框架的角色分工如下: Elastic ... -
Kakfa集群搭建
2016-02-23 15:36 2653先来整体熟悉下Kafka的一些概念和架构 (一)什么是Ka ... -
大数据日志收集框架之Flume入门
2016-02-02 14:25 4193Flume是Cloudrea公司开源的一款优秀的日志收集框架 ... -
Apache Tez0.7编译笔记
2016-01-15 16:33 2536目前最新的Tez版本是0.8,但还不是稳定版,所以大家还 ... -
Bug死磕之hue集成的oozie+pig出现资源任务死锁问题
2016-01-14 15:52 3844这两天,打算给现有的 ... -
Hadoop2.7.1和Hbase0.98添加LZO压缩
2016-01-04 17:46 26091,执行命令安装一些依赖组件 yum install -y ... -
Hadoop2.7.1配置NameNode+ResourceManager高可用原理分析
2015-11-11 19:51 3184关于NameNode高可靠需要配置的文件有core-site ... -
设置Hadoop+Hbase集群pid文件存储位置
2015-10-20 13:40 2865有时候,我们对运行几 ... -
Hadoop+Maven项目打包异常
2015-08-11 19:36 1594先简单说下业务:有一个单独的模块,可以在远程下载Hadoop上 ...
相关推荐
Hadoop 2.2 是一个重要的版本,它在Hadoop生态系统中引入了多项改进和优化,使得大数据处理变得更加高效和可靠。在这个版本中,Hadoop增强了其分布式存储系统HDFS(Hadoop Distributed File System)以及分布式计算...
hadoop2.2 64位 (下) centos6.4 64位编译 这是下半部分
标题中的“hadoop2.2 eclipse插件编译”意味着我们要讨论的是如何在Eclipse中编译适用于Hadoop 2.2版本的插件。这个过程通常涉及到下载源代码、配置构建环境以及执行编译命令。 描述中提到的“hadoop 2.x插件编译所...
hadoop 2.2 eclipse plugins 插件 拷贝至plugins即可 留给自己的,当做备份用
Hadoop2.2+Zookeeper3.4.5+HBase0.96集群环境搭建 Hadoop2.2+Zookeeper3.4.5+HBase0.96集群环境搭建是大数据处理和存储的重要组件,本文档将指导用户从零开始搭建一个完整的Hadoop2.2+Zookeeper3.4.5+HBase0.96集群...
《Hadoop 2.2 安装手册》是针对大数据处理框架Hadoop 2.2的详尽安装指南,特别适合初学者和有经验的IT专业人员参考。这份手册不仅包含了文字描述,还配有完整的截图,使安装过程更加直观易懂。在本文中,我们将深入...
hadoop2.2集群环境搭建,按照本文档操作,可以搭建hadoop2.2的环境,从而进行大数据学习
在本主题中,我们将深入探讨如何使用Eclipse IDE(版本4.2,也称为Juno)来编译Hadoop 1.0.3项目。Eclipse是一款强大的Java开发工具,而Hadoop则是一个分布式计算框架,广泛应用于大数据处理。通过集成Eclipse的插件...
标题中的"hadoop2.2-64-native包(redhat6.3x64)"指的是一个针对Red Hat Enterprise Linux 6.3 64位系统的Hadoop 2.2版本的本地库(native libraries)包。这个包是用户从源代码编译而来的,通常包含了Hadoop运行时所需...
hadoop2.2 安装 工具 hive hbase快速安装工具
在本文中,我们将深入探讨如何在Hadoop 2.2环境下安装Hive。Hive是Apache软件基金会开发的一个数据仓库工具,它允许用户通过SQL-like查询语言(HQL)来处理存储在Hadoop分布式文件系统(HDFS)中的大数据集。在...
首先,我们关注的标题是“hadoop2.2在window7 sp1 32位系统中运行所需要的文件”。这意味着我们要在32位Windows环境下搭建Hadoop 2.2环境。通常,在Windows上安装Hadoop会遇到一些挑战,因为官方主要支持Linux平台。...
这个插件提供了与Hadoop集群交互的功能,例如创建、提交和监控MapReduce作业。它简化了开发过程,允许开发者在Eclipse环境中直接管理Hadoop项目,浏览HDFS文件系统,甚至直接在IDE内编写、测试和调试作业。 安装...
《Hadoop2.2部署指南》 在当前的数字化时代,大数据处理已成为企业的重要需求,而Hadoop作为开源的大数据处理框架,因其高效、可扩展的特性,深受业界青睐。本指南将详细介绍如何在Red Hat Enterprise Linux Server...
在本文中,我们将深入探讨Hadoop 2.2的编译和安装过程,这是一个广泛用于大数据处理和存储的开源框架。Hadoop的核心组件包括HDFS(Hadoop分布式文件系统)和MapReduce,它们共同构建了一个可扩展、容错性强的大数据...
在Hadoop 2.2中,Ganglia监控可以提供丰富的性能指标,如CPU利用率、内存使用、磁盘I/O、网络流量以及Hadoop特有的指标,如作业执行时间、任务进度等。这些数据对于优化Hadoop集群的性能、预防故障和规划扩展都至关...
开发者在使用Hadoop 2.2 API时,需要了解如何配置Hadoop环境,编写Map和Reduce函数,设置输入输出格式,以及如何提交和监控作业。此外,理解YARN的资源管理和调度策略也是高效开发的关键。 总之,Hadoop 2.2 API是...
hadoop2.2伪分布式集群搭建 #查看防火墙状态 service iptables status #关闭防火墙 service iptables stop #查看防火墙开机启动状态 chkconfig iptables --list #关闭防火墙开机启动 chkconfig iptables off
hadoop2.2集群搭建遇到的各种问题。
hadoop2.2 64位 centos6.4 64位编译 这是上半部分