cdh3u0的jetty导致Error Reading IndexFile

 

在36个机器上面跑一个大作业，8千多个map，2w多个reduce，跑reduce的时候经常会出现如下问题：

 

Map output lost, rescheduling: getMapOutput(attempt_201204130934_0012_m_000022_0,5416) failed :
java.io.IOException: Error Reading IndexFile
   at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113)
   at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66)
   at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3651)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
   at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:824)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
   at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
   at org.mortbay.jetty.Server.handle(Server.java:326)
   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
   at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
   at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
   at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.EOFException
   at java.io.DataInputStream.readFully(DataInputStream.java:180)
   at java.io.DataInputStream.readLong(DataInputStream.java:399)
   at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:75)
   at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:55)
   at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109)
   ... 23 more

 

reduce读取不到map的输出结果，然后map重新跑，reduce重新读取数据，整个job运行时间就变长了。。

在官方issue MAPREDUCE-2389 Spurious EOFExceptions reading SpillRecord index files中有介绍。

https://issues.apache.org/jira/browse/MAPREDUCE-2389

https://issues.apache.org/jira/browse/MAPREDUCE-2980

解决方案：

（1）升级到成3u3（没有测试，但是应该能解决）

（2）把3u0的lib目录下面jetty相关包替换成3u3使用的就行（经过测试没有问题）

jetty-6.1.26.cloudera.1.jar 、jetty-servlet-tester-6.1.26.cloudera.1.jar 、jetty-util-6.1.26.cloudera.1.jar

（3）官方issue中有提到换成jetty-6.1.14（没有测试）

评论

1 楼 tsbian 2012-06-08  

我也遇到了类似的问题。

这个错误好像是由于Web服务器Jetty造成的。


参考：https://issues.apache.org/jira/browse/MAPREDUCE-2389

Map output lost, rescheduling: getMapOutput(attempt_201205161733_6589_m_000313_0,40) failed :
java.io.IOException: Illegal seek
at java.io.FileInputStream.skip(Native Method)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3841)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)