java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
winutils
Windows binaries for Hadoop versions (built from the git commit ID used for the ASF relase)
项目地址:https://gitcode.com/gh_mirrors/wi/winutils

·
问题说明:
一般来说,我编写Spark,MapReduce程序都是会在本地IDEA中进行,开发。本地跑通了,才会把代码放到集群上去跑。
当我在运行一个简单的Spark Job 的时候,控制台出现如下的错误:
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
问题解决:
一看不能加载,我就感觉我的机器(Windows 10)可能是因为缺少这个winutils的文件。
我把错误放到google上一搜,方案真的多
我就找了一篇比较不错的,然后完美解决 我的问题。
具体如下:
-
Download winutils.exe from http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe.
Also, be sure to download the correct winutils.exe based on the version of hadoop that spark is compiled for (so, not necessarily the link above) -
create a folder c:\hadoop\bin
-
add Download winutils.exe to the c:\hadoop\bin.
-
Set environment variable HADOOP_HOME to C:\hadoop.warning!set HADOOP_HOME to hadoop folder instead of the bin folder.
OR,System.setProperty(“hadoop.home.dir”, “C:\hadoop\”) in your code.




Windows binaries for Hadoop versions (built from the git commit ID used for the ASF relase)
最近提交(Master分支:24 天前 )
e8089ecf - 2 年前
d4f71517
point people at cdarlint/winutils for binaries and call out the fact that we could remove the need for this entirely just to run spark on windows 6 年前
更多推荐
所有评论(0)