远程Java客户端上传文件到HDFS
linux-dash
A beautiful web dashboard for Linux
项目地址:https://gitcode.com/gh_mirrors/li/linux-dash
免费下载资源
·
Hadoop集群环境:
三台机器:namenode0, datanode1, datanode2
操作系统:Ubuntu 11.04 Server version
Haddop版本: hadoop-0.20.2-cdh3u1
HBase版本:hbase-0.90.4-cdh3u2
Java版本:jdk-6u29-linux-x64
客户端机器:
注意点:
- conf需添加core-site.xml配置文件,且其内容为正确。
- 权限的问题:集群里的机器是以hadoop账户运行的,我本地机器是以lotus账户运行的,如果上传到hdfs://namenode0:54310/usr/目录下会失败,抛出Permission denied的异常。因为/usr目录的owner是hadoop,而上传到hdfs://namenode0:54310/usr/lotus/目录就成功,是因为lotus目录的owner被设置为lotus账户了。
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://namenode0:54310</value>
</property>
</configuration>
- 代码
import java.io.File;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class HDFSUpLoad {
//HDFSUpLoad srcFilePath, dstFilePath
public static void main(String[] args) throws Exception
{
if(args.length < 2)
{
System.out.print("command line: HDFSUpLoad srcFilePath dstFilePath");
return;
}
String srcFilePath = args[0];
String dstFilePath = args[1];
Configuration conf = new Configuration();
conf.addResource(new Path("/home/lotus/work/hadoop-0.20.2-cdh3u1/conf/core-site.xml"));
FileSystem hdfs = FileSystem.get(conf);
Path src = new Path(srcFilePath);
Path dst = new Path(dstFilePath);
System.out.print("start upload");
long startTime = System.currentTimeMillis();
hdfs.copyFromLocalFile(src, dst);
long endTime = System.currentTimeMillis();
System.out.print("finish upload, cost time: " + String.valueOf(endTime - startTime));
}
}
- 性能测试:
HDFSUpload /media/PC-Book/DD/win2000.dd hdfs://namenode0:54310/usr/lotus/win2000.dd
上传一个3.9G的dd文件,花了428秒,第二次上传花了377秒,性能问题以后研究。
GitHub 加速计划 / li / linux-dash
10.39 K
1.2 K
下载
A beautiful web dashboard for Linux
最近提交(Master分支:1 个月前 )
186a802e
added ecosystem file for PM2 4 年前
5def40a3
Add host customization support for the NodeJS version 4 年前
更多推荐
已为社区贡献1条内容
所有评论(0)