java 解决Could not locate executable null\bin\winutils.exe in the Hadoop bin
winutils
Windows binaries for Hadoop versions (built from the git commit ID used for the ASF relase)
项目地址:https://gitcode.com/gh_mirrors/wi/winutils
免费下载资源
·
问题描述
在windows环境下运行连接hadoop服务的程序报告以下错误信息:
og4j:ERROR Could not find value for key log4j.appender.logRollingFile
log4j:ERROR Could not instantiate appender named "logRollingFile".
[ERROR] 2018-09-17 17:40:02,151 method:org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:374)
Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2807)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:160)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
at com.yue.modular.hdfsClient.main(hdfsClient.java:31)
[WARN ] 2018-09-17 17:40:03,432 method:org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
原因分析
根据错误信息进入hadoop源码发现:
static {
WINDOWS = osType == Shell.OSType.OS_TYPE_WIN;
SOLARIS = osType == Shell.OSType.OS_TYPE_SOLARIS;
MAC = osType == Shell.OSType.OS_TYPE_MAC;
FREEBSD = osType == Shell.OSType.OS_TYPE_FREEBSD;
LINUX = osType == Shell.OSType.OS_TYPE_LINUX;
OTHER = osType == Shell.OSType.OS_TYPE_OTHER;
PPC_64 = System.getProperties().getProperty("os.arch").contains("ppc64");
//注意这行
HADOOP_HOME_DIR = checkHadoopHome();
WINUTILS = getWinUtilsPath();
isSetsidAvailable = isSetsidSupported();
TOKEN_SEPARATOR_REGEX = WINDOWS ? "[|\n\r]" : "[ \t\n\r\f]";
}
private static String checkHadoopHome() {
//高能高能~~~~~~~~~
String home = System.getProperty("hadoop.home.dir");
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
try {
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
}
if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length() - 1);
}
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir + " does not exist, is not a directory, or is not an absolute path.");
}
home = homedir.getCanonicalPath();
} catch (IOException var2) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", var2);
}
home = null;
}
return home;
}
public static final String getQualifiedBinPath(String executable) throws IOException {
String fullExeName = HADOOP_HOME_DIR + File.separator + "bin" + File.separator + executable;
File exeFile = new File(fullExeName);
if (!exeFile.exists()) {
throw new IOException("Could not locate executable " + fullExeName + " in the Hadoop binaries.");
} else {
return exeFile.getCanonicalPath();
}
}
问题解决
分析发现是hadoop.home.dir 为null 和获取HADOOP_HOME 变量没找到目录没找到,
解决办法2种:
第一种方法:添加环境变量HADOOP_HOME
第二种方法:直接java设置hadoop.home.dir 指向 winutil.exe 的目录
主要说一下设置hadoop.home.dir 目录指向:
System.setProperty("hadoop.home.dir", "C:\\Program Files\\Hadoop\\hadoop2.7.2-bin");
//构造一个访问指定hdfs系统的客户端对象
FileSystem fs = FileSystem.get(URI.create("hdfs://192.168.119.129:9000"), new Configuration(), "root");
winutils.exe下载地址:
winutils.exe下载地址
GitHub 加速计划 / wi / winutils
2.51 K
3 K
下载
Windows binaries for Hadoop versions (built from the git commit ID used for the ASF relase)
最近提交(Master分支:2 个月前 )
e8089ecf - 1 年前
d4f71517
point people at cdarlint/winutils for binaries and call out the fact that we could remove the need for this entirely just to run spark on windows 5 年前
更多推荐
已为社区贡献1条内容
所有评论(0)