cygwin下，hadoop的wordcount运行出错。调试发现是因为symlink，怎么解决?

0 0

cygwin下，hadoop的wordcount运行出错。调试发现是因为symlink，怎么解决?5

之前在ubuntu的多台机器的环境下使用，一切都很正常。最近因为有几台电脑必须是windows，才用了cygwin，没想到连pseudo模式的wordcount都跑不起来。

环境：windows7+cygwin 1.7.11-1+jdk1.6.0_31+hadoop 1.0.0
windows7是加入域的，但用本地新建的administrators的用户问题一样。

目前的问题：
1. 目前standalone模式运行正常，pseudo和cluster下运行wordcount，都出同样问题了。同样设置，在ubuntu下运行都很正常。
2. 另外，我的cygwin安装在d:/cygwin下。因为默认设置中，hadoop.tmp.dir = /tmp/hadoop-${user.name}，我的user.name是timwu，所以本应该新建tmp目录在cygwin的/tmp/hadoop-timwu下，但我发现实际确实建立在d:/tmp/hadoop-timwu，对于cygwin，也就是/cygdriver/d/tmp/hadoop-timwu。这个正常吗?

3. 问：hadoop运行在cygwin下时，如果要操作文件，比如new File(...)时，路径是按照cygwin下的规则(/home/user/....), 还是按照windows下的规则(d:/cygwin/home/user/....)。我这个跟踪代码发现是windows的规则。所以想知道，如果正确情况，应该是那种？另外，我想看看正确的情况下，cygwin运行hadoop，classpath是用win的还是cygwin的，你可以贴出你的情况下tasktracker的classpath目录吗？

echo exec "$JAVA" -Dproc_$COMMAND $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
exec "$JAVA" -Dproc_$COMMAND $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@

比如修改bin/hadoop文件，在倒数第二句前加个echo，比如如下：

我这的信息如下：

exec /cygdrive/c/jdk1.6.0_31//bin/java -Dproc_tasktracker -Xmx1000m -Dhadoop.log.dir=D:\cygwin\home\timwu\hadoop-1.0.0\logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=D:\cygwin\home\timwu\hadoop-1.0.0\ -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Dhadoop.security.logger=INFO,NullAppender -Djava.library.path=/home/timwu/hadoop-1.0.0/lib/native/Windows_XP-x86-32 -Dhadoop.policy.file=hadoop-policy.xml -classpath D:\cygwin\home\timwu\hadoop-1.0.0\conf.pseudo;C:\jdk1.6.0_31\lib\tools.jar;D:\cygwin\home\timwu\hadoop-1.0.0\;... org.apache.hadoop.mapred.TaskTracker

能贴个你的给我看看么

4. 在pseudo下，运行wordcount，tasktracker出错, 具体任务下的log目录是空的。而tasktracker的log输出的错误信息如下。

12/03/28 14:35:13 INFO mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201203280212_0005_m_-1386636958
12/03/28 14:35:13 INFO mapred.JvmManager: JVM Runner jvm_201203280212_0005_m_-1386636958 spawned.
12/03/28 14:35:17 INFO mapred.JvmManager: JVM Not killed jvm_201203280212_0005_m_-1386636958 but just removed
12/03/28 14:35:17 INFO mapred.JvmManager: JVM : jvm_201203280212_0005_m_-1386636958 exited with exit code -1. Number of tasks it ran: 0
12/03/28 14:35:17 WARN mapred.TaskRunner: attempt_201203280212_0005_m_000002_0 : Child Error
java.io.IOException: Task process exit with nonzero status of -1.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
12/03/28 14:35:21 INFO mapred.TaskTracker: addFreeSlot : current free slots : 2
12/03/28 14:35:24 INFO mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201203280212_0005_m_000002_1 task's state:UNASSIGNED
12/03/28 14:35:24 INFO mapred.TaskTracker: Trying to launch : attempt_201203280212_0005_m_000002_1 which needs 1 slots
12/03/28 14:35:24 INFO mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201203280212_0005_m_000002_1 which needs 1 slots
12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stdout log for task: attempt_201203280212_0005_m_000002_0
java.io.FileNotFoundException: D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_0\log.index (The system cannot find the path specified)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:120)
        at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
        at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188)
        at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:423)
        at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
        at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stderr log for task: attempt_201203280212_0005_m_000002_0
java.io.FileNotFoundException: D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_0\log.index (The system cannot find the path specified)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:120)
        at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
        at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188)
        at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:423)
        at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
        at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

我用远程调试跟踪代码，发现是tasktracker在新建一个jvm线程运行任务前，需要先建立该任务的log目录。跟踪并查看一下代码：

org.apache.hadoop.mapredTaskLog.createTaskAttemptLogDir(TaskAttemptID, boolean, String[]) line: 97:
public static void createTaskAttemptLogDir(TaskAttemptID taskID,
      boolean isCleanup, String[] localDirs) throws IOException{
    String cleanupSuffix = isCleanup ? ".cleanup" : "";
    String strAttemptLogDir = getTaskAttemptLogDir(taskID, 
        cleanupSuffix, localDirs);
    File attemptLogDir = new File(strAttemptLogDir);
    if (!attemptLogDir.mkdirs()) {
      throw new IOException("Creation of " + attemptLogDir + " failed.");
    }
    String strLinkAttemptLogDir = 
        getJobDir(taskID.getJobID()).getAbsolutePath() + File.separatorChar +
        taskID.toString() + cleanupSuffix;
    if (FileUtil.symLink(strAttemptLogDir, strLinkAttemptLogDir) != 0) {
      throw new IOException("Creation of symlink from " + 
                            strLinkAttemptLogDir + " to " + yestrAttemptLogDir +
                            " failed.");
    }
    //Set permissions for target attempt log dir 
    FsPermission userOnly = new FsPermission((short) 0777); //FsPermission userOnly = new FsPermission((short) 0700);
    FileUtil.setPermission(attemptLogDir, userOnly);
  }

其中symLink()函数的代码如下：

public static int symLink(String target, String linkname) throws IOException{
    String cmd = "ln -s " + target + " " + linkname;
    Process p = Runtime.getRuntime().exec(cmd, null);
    int returnVal = -1;
    try{
      returnVal = p.waitFor();
    } catch(InterruptedException e){
      //do nothing as of yet
    }
    if (returnVal != 0) {
      LOG.warn("Command '" + cmd + "' failed " + returnVal + 
               " with: " + copyStderr(p));
    }
    return returnVal;
  }

也就是说，此时实际是在strAttemptLogDir建了个目录，而后调用"ln -s"在strAttemptLogDir建立个它的symlink。具体的一次任务时：

strLinkAttemptLogDir = D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1
strAttemptLogDir=/tmp/hadoop-timwu/mapred/local\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1

而后tasktracker会新建一个thread(jvm)去运行这个任务，org.apache.hadoop.mapredTaskRunner.java
MapTaskRunner(TaskRunner).run() line: 249: launchJvmAndWait(setupCmds, vargs, stdout, stderr, logSize, workDir);
跟踪子thread，可以看到出错相关的代码是：

org.apache.hadoop.mapred.java
DefaultTaskController.launchTask(String, String, String, List<String>, List<String>, File, String, String) line: 107        
     ...........
      //mkdir the loglocation
      String logLocation = TaskLog.getAttemptDir(jobId, attemptId).toString();
      if (!localFs.mkdirs(new Path(logLocation))) {
        throw new IOException("Mkdirs failed to create " 
                   + logLocation);
      }
     ..........

此时mkdirs()会出错，因为logLocation是个symlink文件。具体说，
logLocation=D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1, 也就是上面说的strLinkAttemptLogDir的值，实际上只是cygwin的一个symlink，在windows下看，会发现只是一个二进制文件，内容大概类似
<symlink>/tmp/hadoop-timwu/mapred/local\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1

另外，fs.mkdir()内容如下。

public boolean mkdirs(Path f) throws IOException {
    Path parent = f.getParent();
    File p2f = pathToFile(f);
    return (parent == null || mkdirs(parent)) &&
      (p2f.mkdir() || p2f.isDirectory());
  }

跟踪发现,p2f.isDirectory()返回的是false，而p2f.isFile()返回的是true，也就是说对于java来说，这个symlink是个文件，所以mkdir()就出错，抛出
IOException(“Mkdirs failed to create D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000001_1”) ，

而后子进程return -1。所以父进程就报最开始说的异常。而子进程因为log文件没建立，所以对应的userlogs/job_201203280212_0005\attempt_201203280212_0005_m_000001_1/ 下肯定是空的。

5. 有没有什么参数能关闭了symlink，让hadoop直接建一个目录，而不是建了目录又建link?我尝试在mapred-site.xml文件添加

    <property>
         <name>mapred.create.symlink</name>
         <value>false</value>
     </property>

不过没用，应该不是这个参数。我又不想随便改代码，相信这个不是错误的本质。

===================== 细节补充
1. 这个symlink也很怪，对于cygwin, 因为我的第二个问题，文件是建立在/cygdriver/d/tmp/hadoop-timwu/... 下，而这个symlink是只向/tmp/hadoop-timwu/.....,所以在cygwin下，这个symlink is not file or directory。而对于windows，这个symlink就是一个普通的文件，只是它的内。两种情况，java默认都无法识别。
2. 我在多台电脑上都试过，同样错误。
3. 另外，之前我修改过两个地方，一个就是上面说过的public static void createTaskAttemptLogDir(TaskAttemptID taskID,
boolean isCleanup, String[] localDirs)的倒数第二行，把之前的FsPermission userOnly = new FsPermission((short) 0700); 改为FsPermission userOnly = new FsPermission((short) 0777);，否则会报错说在win下无法修改为0700
另一处是

org.apache.hadoop.fs.RawLocalFileSystem.java, line: 507
public void setPermission(Path p, FsPermission permission
                            ) throws IOException {
    FileUtil.setPermission(pathToFile(p), permission);    
  }

改成:

private FsPermission permission777 = FsPermission.valueOf("-rwxrwxrwx");

public void setPermission(Path p, FsPermission permission
                            ) throws IOException {
    //FileUtil.setPermission(pathToFile(p), permission);
    FileUtil.setPermission(pathToFile(p), permission777);
  }

同样是如果不改，会出现0777的错误，如

java.io.IOException: Failed to set permissions of path:
file:/tmp/hadoop-iwonabb/mapred/staging/iwonabb-1931875024/.staging to 0700
                at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:526)
                at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:500)
                at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:310)
                at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
                at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
                at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:799)

...................

2012年3月28日 21:03