初次接触hadoop，安装了一个测试环境，安装的过程参考了网上很多文章。对有些hadoop东西还不是很理解，以后再慢慢学习。

基本信息

IP	192.168.112.114
Hadoop版本	2.2
机器	Oracle Linux 6.3 64位

 

目录配置

/home/hadoop	用户目录
/app/hadoop/hadoop-2.2.0	软件home
/app/hadoop/dfs/name	数据和编辑文件
/app/hadoop/dfs/data	数据和编辑文件
/app/hadoop/mapred/local	存放数据
/app/hadoop/mapred/system	存放数据

 

安装JAVA

从oracle官网下载，然后上传至机器

[root@hadoop01 soft]# rpm -ivhjdk-7u45-linux-x64.rpm

添加用户

[root@OEL63 ~]# useradd hadoop

[root@OEL63 ~]# passwd hadoop

SSH配置

Hadoop用户

[hadoop@hadoop01 ~]$ ssh-keygen -t dsa -P'' -f ~/.ssh/id_dsa

[hadoop@hadoop01 ~]$ cat ~/.ssh/id_dsa.pub>> ~/.ssh/authorized_keys

 

Root用户

[root@hadoop01 ~]# chmod go-w /home/hadoop/home/hadoop/.ssh

[root@hadoop01 ~]# chmod 600/home/hadoop/.ssh/authorized_keys

 

Hadoop用户

[hadoop@hadoop01 ~]$ ssh localhost

环境变量

$ vi .bash_profile

export JAVA_HOME=/usr/java/jdk1.7.0_45

export HADOOP_HOME=/app/hadoop/hadoop-2.2.0

export JRE_HOME=$JAVA_HOME/jre

exportPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

exportCLASSPATH=./:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

 

下载安装hadoop：

http://apache.fayea.com/apache-mirror/hadoop/common/hadoop-2.2.0/

 

上传解压

[hadoop@hadoop01 soft]$ gzip -dhadoop-2.2.0.tar.gz

[hadoop@hadoop01 soft]$ tar -xvfhadoop-2.2.0.tar

[root@hadoop01 ~]# mkdir -p /app/hadoop

[root@hadoop01 ~]# chown -R hadoop:root/app/hadoop/

移动解压软件到软件目录

/app/hadoop/hadoop-2.2.0

 

修改hadoop参数文件

安装参考：

http://wenku.baidu.com/view/1681511a52d380eb62946df6.html

 

以下所有配置文件的修改均在下面目录完成

[hadoop@hadoop01 hadoop]$ pwd

/app/hadoop/hadoop-2.2.0/etc/hadoop

Core-site.xml

[hadoop@hadoop01 hadoop]$ vi core-site.xml

<configuration>

<property>

 <name>fs.default.name</name>

 <value>hdfs://localhost:8020</value>

 <description>The name of the defaultfile system. Either the literal string "local" or a host:port forNDFS.

 </description>

 <final>true</final>

</property>

</configuration>

Hdfs-site.xml

[hadoop@hadoop01 hadoop]$ vi hdfs-site.xml

 

 

<configuration>

         <property>

                   <name>dfs.namenode.name.dir</name>

                   <value>file:/app/hadoop/dfs/name</value>

                   <description>Determineswhere on the local filesystem the DFS name node should store the name table. Ifthis is a comma-delimited list of directories then the name table is replicatedin all of the directories, for redundancy. </description>

                   <final>true</final>

         </property>

         <property>

                   <name>dfs.datanode.data.dir</name>

                   <value>file:/app/hadoop/dfs/data</value>

                   <description>Determineswhere on the local filesystem an DFS data node should store its blocks. If thisis a comma-delimited list of directories, then data will be stored in all nameddirectories, typically on different devices.Directories that do not exist areignored.

                   </description>

                   <final>true</final>

         </property>

         <property>

                   <name>dfs.replication</name>

                   <value>1</value>

         </property>

         <property>

         <name>dfs.permissions</name>

         <value>false</value>

         </property>

          

</configuration>

Mapred-site.xml

[hadoop@hadoop01 hadoop]$ vi mapred-site.xml

 

<configuration>

         <property>

         <name>mapreduce.framework.name</name>

         <value>yarn</value>

         </property>

         <property>

                   <name>mapred.system.dir</name>

                   <value>file:/app/hadoop/mapred/system</value>

                   <final>true</final>

         </property>

         <property>

         <name>mapred.local.dir</name>

         <value>file:/app/hadoop/mapred/local</value>

         <final>true</final>

         </property>

</configuration>

Yarn-site.xml

[hadoop@hadoop01 hadoop]$ vi yarn-site.xml

<configuration>

<property>

<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>

<value>mapreduce_shuffle</value>

 <description>shuffle service that needsto be set for Map Reduce to run </description>

 </property> 

</configuration>

 

Hadoop-env.sh

[hadoop@hadoop01 hadoop]$ vi hadoop-env.sh

增加

export JAVA_HOME=/usr/java/jdk1.7.0_45

 

说明：该文件中原来有一条export JAVA_HOME=${JAVA_HOME}，环境变量中也设置了JAVA_HOME，但是后面会报错，应该是没有识别出来，所以就又手工添加了上面一行，写到了原来这一行的后面。（什么原因导致没认出来，没想明白，也有可能找的是root中的JAVA_HOME，下一次安装再进行验证）

 

启动hadoop

格式化namenode

[hadoop@hadoop01 ~]$ hdfs namenode –format

开启守护进程

[hadoop@hadoop01 ~]$ hadoop-daemon.sh start namenode

 [hadoop@hadoop01~]$ hadoop-daemon.sh start datanode

开启yarn守护进程

[hadoop@hadoop01 ~]$ yarn-daemon.sh start resourcemanager

[hadoop@hadoop01 ~]$ yarn-daemon.sh start nodemanager

[hadoop@hadoop01 ~]$ start-yarn.sh

检查进程是否启动

[hadoop@hadoop01 ~]$ jps

2912 NameNode

5499 ResourceManager

2981 DataNode

6671 Jps

6641 NodeManager

6473 SecondaryNameNode

 

有以上内容说明已经启动

查看hadoop资源管理页面

http://localhost:8088