在 hadoop2集群 master1上, 执行 HADOOP自带的例子, wordcount 包,命令如下
cd /usr/local/hadoop/hadoop-2.0.2-alpha/share/hadoop/mapreduce
hadoop jar hadoop-mapreduce-examples-2.0.2-alpha.jar wordcount hdfs://192.168.1.33:9000/input hdfs://192.168.1.33:9000/output
出现错误:
13/06/24 05:48:06 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:83)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:76)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1188)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1184)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1183)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
但是我通过start-all.sh启动之后,jps查看 master机器上:ResourceManager和NameNode都已启动,slave机器上NodeManager和DataNode也已经启动了,查看启动日志均无报错,但是验证map/reduce时就报这个错误,请指点迷津,不胜感谢
分享到:
相关推荐
12/04/24 15:32:44 ERROR security.UserGroupInformation: PriviledgedActionException as:Administrator cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\...
在Windows平台上进行Hadoop的Map/Reduce开发可能会比在Linux环境下多一些挑战,但通过详细的步骤和理解Map/Reduce的工作机制,开发者可以有效地克服这些困难。以下是对标题和描述中涉及知识点的详细说明: **Hadoop...
hadoop 2.7.3 的本地接口库,snappy 使用的是1.1.3版本。使用时直接把包解压后放到$HADOOP_HOME/lib/native目录下即可。 [root@pseudo ~]# hadoop checknative -a 17/09/11 14:30:49 WARN bzip2.Bzip2Factory: ...
解决办法:该问题是由于reduce预处理阶段shuffle时获取已完成的map输出失败次数超过上限造成的,上限默认为5。解决办法是修改/etc/security/limits.conf文件,增加nofile参数的值。具体来说,需要添加两行:`* soft ...
MapReduce的设计理念源于Google的同名论文,它通过将大规模数据处理任务分解为两个阶段:Map(映射)和Reduce(化简),使得海量数据能够在多台计算机上并行处理,极大地提高了数据处理效率。 Map阶段是数据处理的...
当从本地上传文件到HDFS中...org.apache.hadoop.fs.ChecksumException: Checksum error: file:/hyk/data/hyk.txt [root@node01 data]# hadoop fs -put hyk.txt /hyk/test 20/02/18 12:54:39 INFO fs.FSInputChecker: Fo
ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: \tmp\hadoop-admin \mapred\local\ttprivate to 0700 at org.apache...
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) at org.apache.hadoop.mapreduce.Job....
18/05/25 19:51:49 INFO mapreduce.Job: map 0% reduce 0% 18/05/25 19:52:20 INFO mapreduce.Job: map 100% reduce 0% 18/05/25 19:52:29 INFO mapreduce.Job: map 100% reduce 100% 18/05/25 19:52:31 INFO ...
util.Shell (Shell.java:(694)) - Did not find ...java.io.FileNotFoundException: Could not locate Hadoop executable: E:\hadoop-3.0.2\bin\winutils.exe -see https://wiki.apache.org/hadoop/WindowsProblems
例如,`org.apache.hadoop.mapred.MapTask`和`org.apache.hadoop.mapreduce.ReduceTask`分别对应Map和Reduce任务的实现,开发者可以通过阅读这些源码了解任务执行的详细流程。 7. **工具集成**:有许多开源工具可以...
export CLASSPATH="$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:${HADOOP_HOME}/lib/commons-logging-1.0.4.jar:${HADOOP_HOME}/lib/guava-r09-jarjar.jar:${HADOOP_HOME}/hadoop-core-0.20.2-cdh3u3.jar:/usr/...
解决windows下远程调试hadoop权限问题( ERROR security.UserGroupInformation: PriviledgedActionException as:Administrator cause:java.io.IOException: Failed to set permissions of path:)重新编译了hadoop-...
Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=L.MOON, access=WRITE, inode=/user/lsy/result1/_temporary/0:lsy:supergroup:drwxr-xr-x Caused by: org....
如果出现如下bug:“Could not locate executable null\bin\winutils.exe in the Hadoop binaries”,则下载该文件,放入hadoop的bin文件夹下,并设置环境变量HADOOP_HOME:F:\hadoop2.2.0即可。
* hadoop.security.group.mapping:组内用户的列表的类设定,default value 是 org.apache.hadoop.security.ShellBasedUnixGroupsMapping。 * hadoop.security.authorization:authorization,default value 是 ...
hadoop安装与配置 hadoop安装与配置 Hadoop的安装与配置可以分成几个主要步骤: 1. 安装Java 2. 下载Hadoop 3. 配置Hadoop 4. 格式化Hadoop文件系统 5. 启动Hadoop 以下是基于Linux系统的简化安装与配置步骤: 1. ...