解决spark运行中ERROR Shell:Failed to locate the winutils binary in the hadoop binary path的问题
winutils
Windows binaries for Hadoop versions (built from the git commit ID used for the ASF relase)
项目地址：https://gitcode.com/gh_mirrors/wi/winutils
免费下载资源
·
17/09/03 21:27:13 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:362)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$30.apply(SparkContext.scala:1014)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$30.apply(SparkContext.scala:1014)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:179)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:179)在代码中设置hadoop.home.dir
System.setProperty("hadoop.home.dir", "D:\\hadoop-2.5.0\\hadoop-2.5.0")
GitHub 加速计划 / wi / winutils
5
1
下载
Windows binaries for Hadoop versions (built from the git commit ID used for the ASF relase)
最近提交(Master分支：2 个月前 )
e8089ecf - 2 年前
d4f71517
point people at cdarlint/winutils for binaries and call out the fact that we could remove the need for this entirely just to run spark on windows 6 年前
新一代开源开发者平台 GitCode，通过集成代码托管服务、代码仓库以及可信赖的开源组件库，让开发者可以在云端进行代码托管和开发。旨在为数千万中国开发者提供一个无缝且高效的云端环境，以支持学习、使用和贡献开源项目。
更多推荐
0
0- 0
已为社区贡献2条内容
所有评论(0)