奇怪的错误!!HdInsight Hadoop MapReduce失败，代码为255

本文关键字：失败 代码 MapReduce Hadoop 错误 HdInsight | 更新日期: 2023-09-27 18:05:50

我使用Microsoft Azure的HdInsight与1头节点&1个数据节点

如果我使用一个小的数据集(85 MB)和我写的MapReduce程序，一切都很好，我在我的容器/blob中得到了想要的输出。较大的文件失败，错误如下。

我读到一些文章提到设置mapreduce.map.memory.mb应该设置为"1024"，这样映射器就有更多的内存。我不明白这是如何扩展的，考虑到我有190 GB的文件要处理，而集群的机器都没有接近这个数量的RAM。

我确信有一些小的我错过了，但是有人知道我应该如何1)解决这个问题2)使它，所以我可以扩展mapReduce过程到大的输入文件没有这些错误?

如果我使用的输入文件是8gb，那么我得到下面的错误:

15/08/01 05:43:17 INFO mapreduce.Job:  map 56% reduce 0%
15/08/01 05:43:23 INFO mapreduce.Job:  map 57% reduce 0%
15/08/01 05:43:29 INFO mapreduce.Job:  map 58% reduce 0%
15/08/01 05:43:30 INFO mapreduce.Job: Task Id : attempt_1438405138600_0006_m_0
010_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
iled with code 255
        at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed
ava:320)
        at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.ja
:533)
        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInfor
tion.java:1594)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
15/08/01 05:43:31 INFO mapreduce.Job:  map 57% reduce 0%
15/08/01 05:43:36 INFO mapreduce.Job:  map 58% reduce 0%
15/08/01 05:43:43 INFO mapreduce.Job:  map 59% reduce 0%
15/08/01 05:43:47 INFO mapreduce.Job:  map 60% reduce 0%
15/08/01 05:43:53 INFO mapreduce.Job:  map 61% reduce 0%
15/08/01 05:43:59 INFO mapreduce.Job:  map 62% reduce 0%
15/08/01 05:44:05 INFO mapreduce.Job:  map 63% reduce 0%
15/08/01 05:44:05 INFO mapreduce.Job: Task Id : attempt_1438405138600_0006_m_0
010_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
iled with code 255
        at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed
ava:320)
        at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.ja
:533)
        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInfor
tion.java:1594)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
15/08/01 05:44:06 INFO mapreduce.Job:  map 61% reduce 0%
15/08/01 05:44:07 INFO mapreduce.Job:  map 62% reduce 0%
15/08/01 05:44:16 INFO mapreduce.Job:  map 63% reduce 0%
15/08/01 05:44:20 INFO mapreduce.Job:  map 64% reduce 0%
15/08/01 05:44:27 INFO mapreduce.Job:  map 65% reduce 0%
15/08/01 05:44:35 INFO mapreduce.Job:  map 66% reduce 0%
15/08/01 05:44:41 INFO mapreduce.Job:  map 69% reduce 0%
15/08/01 05:44:48 INFO mapreduce.Job:  map 70% reduce 0%
15/08/01 05:44:48 INFO mapreduce.Job: Task Id : attempt_1438405138600_0006_m_0
010_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
iled with code 255
        at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed
ava:320)
        at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.ja
:533)
        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInfor
tion.java:1594)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
15/08/01 05:44:49 INFO mapreduce.Job:  map 68% reduce 0%
15/08/01 05:44:50 INFO mapreduce.Job:  map 71% reduce 0%
15/08/01 05:45:02 INFO mapreduce.Job:  map 72% reduce 0%
15/08/01 05:45:05 INFO mapreduce.Job:  map 75% reduce 0%
15/08/01 05:45:13 INFO mapreduce.Job:  map 76% reduce 0%
15/08/01 05:45:18 INFO mapreduce.Job:  map 77% reduce 0%
15/08/01 05:45:21 INFO mapreduce.Job:  map 78% reduce 0%
15/08/01 05:45:23 INFO mapreduce.Job:  map 100% reduce 100%
15/08/01 05:45:28 INFO mapreduce.Job: Job job_1438405138600_0006 failed with s
te FAILED due to: Task failed task_1438405138600_0006_m_000010
Job failed as tasks failed. failedMaps:1 failedReduces:0
15/08/01 05:45:28 INFO mapreduce.Job: Counters: 35
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=605590393
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                WASB: Number of bytes read=5915160278
                WASB: Number of bytes written=0
                WASB: Number of read operations=0
                WASB: Number of large read operations=0
                WASB: Number of write operations=0
        Job Counters
                Failed map tasks=4
                Killed map tasks=3
                Launched map tasks=18
                Other local map tasks=3
                Rack-local map tasks=15
                Total time spent by all maps in occupied slots (ms)=1441061
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=1441061
                Total vcore-seconds taken by all map tasks=1441061
                Total megabyte-seconds taken by all map tasks=1475646464
        Map-Reduce Framework
                Map input records=16375
                Map output records=319985
                Map output bytes=5051193751
                Map output materialized bytes=604451353
                Input split bytes=1210
                Combine input records=0
                Spilled Records=319985
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=2836
                CPU time spent (ms)=1003820
                Physical memory (bytes) snapshot=9417678848
                Virtual memory (bytes) snapshot=13221376000
                Total committed heap usage (bytes)=11603542016
        File Input Format Counters
                Bytes Read=5915148288
15/08/01 05:45:28 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!

奇怪的错误!!HdInsight Hadoop MapReduce失败，代码为255

第一:1024是1GB内存。A1通常是人们在测试时使用的标准大小，它是1.75 GB，然后您将使用A0，它是768mb。然而，这读起来不像内存问题。我希望出现类似这样的错误:

"容器[pid=container_1406552545451_0009_01_000002,containerID=container_234132_0001_01_000001]运行超出物理内存限制。当前使用:569.1 MB的512 MB物理内存使用;使用了1.0 GB的970.1 MB虚拟内存。造成容器。"

这个错误输出读起来像一个作业配置问题。您是否确保您的构建目标是x64并且优选32位未选中?请参阅msdn: https://social.msdn.microsoft.com/Forums/en-US/d79befb1-be5d-4c5a-bb05-30ea9fccc475/hdinsight-mapreduce-fails-with-pipemapredwaitoutputthreads-subprocess-failed-with-code-255?forum=hdinsight

相关文章：