CentOS7安装部署CDH6.1.0
CentOS7安装部署CDH6.1.0
- 1、下载相关安装包
- 2、安装三台服务器
- 3、修改主机名及映射主机名(所有节点/重启)
- 4、关闭防火墙和 selinux(所有节点)
- 5、时钟同步(所有节点)
- 6、配置 ssh 免密登录(所有节点)
- 7、禁用透明大页面压缩(所有节点)
- 8、优化交换分区(所有节点)
- 9、安装jdk(所有节点)
- 10、安装 mysql(master)
- 11、安装 mysql JDBC 驱动(master)
- 12、为下表中每个服务创建 utf8 字符集的数据库:
- 13、安装 cloudera manager 包
- 14、修改 agent 的服务器地址(所有节点)
- 15、启动 agent 服务(所有节点)
- 16、准备 Cloudera Manager 数据库(master)
- 17、准备 parcel 包(master)
- 18、启动 Cloudera Manager Server(master)
- 19、查看启动日志:
- 20、启用 Auto-TLS 以自动创建证书(master)
- 21、登录 web 页面(忽略许可授权,选择免费版)
直接进入正题
1、下载相关安装包
Cloudera Manager 下载地址:
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
jdk 下载地址:
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
CDH 安装包下载地址:https://archive.cloudera.com/cdh6/6.1.0/parcels/
需要下载以下文件:
CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
manifest.json
2、安装三台服务器
IP地址 | 主机名 |
---|---|
192.168.1.211 | cdh1-master |
192.168.1.212 | cdh2-slave |
192.168.1.213 | cdh3-slave |
3、修改主机名及映射主机名(所有节点/重启)
修改主机名:hostnamectl set-hostname
设置映射主机名,将以下内容添加到 /etc/hosts 文件:
192.168.1.211 cdh1-master
192.168.1.212 cdh2-slave
192.168.1.213 cdh3-slave
4、关闭防火墙和 selinux(所有节点)
查看防火墙:systemctl status firewalld.service
执行关闭命令:systemctl stop firewalld.service
执行开机禁用防火墙命令:systemctl disable firewalld.service
关闭selinux:vi /etc/sysconfig/selinux 将 SELINUX=enforcing 改为 SELINUX=disabled
5、时钟同步(所有节点)
安装 NTP
yum -y install ntp
编辑 /etc/ntp.conf 文件添加 NTP 服务器
server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org
启动 NTP
systemctl enable ntpd
systemctl start ntpd
手工同步网络时间
ntpdate -u 0.cn.pool.ntp.org
同步系统时钟
hwclock --systohc
6、配置 ssh 免密登录(所有节点)
在每台机上产生新的rsa公钥私钥文件,并统一拷贝到一个authorized_keys文件中
- 登录cdh1-master,在.ssh目录下输入命令:
ssh-keygen -t rsa
三次回车后,该目录下将会产生id_rsa,id_rsa.pub文件。其他主机也使用该方式产生密钥文件。
- 登录cdh1-master,将id_rsa.pub公钥内容拷贝到authorized_keys文件中 输入命令:
cat id_rsa.pub >> authorized_keys
赋予权限:
chmod 600 authorized_keys
- 登录其他主机,将其他主机的公钥文件内容都拷贝到cdh1-master主机上的authorized_keys文件中,命令如下:
ssh-copy-id -i cdh1-master #登录cdh2-slave,将公钥拷贝到cdh1-master的authorized_keys中
ssh-copy-id -i cdh1-master #登录cdh3-slave,将公钥拷贝到cdh1-master的authorized_keys中
登录cdh1-master,将授权文件拷贝到cdh2-slave、cdh3-slave…,命令如下:
scp /root/.ssh/authorized_keys cdh2-slave:/root/.ssh/ #拷贝到cdh2-slave上
scp /root/.ssh/authorized_keys cdh3-slave:/root/.ssh/ #拷贝到cdh3-slave上
7、禁用透明大页面压缩(所有节点)
打开编辑 /etc/rc.local 文件:
vi /etc/rc.local
添加以下内容:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
8、优化交换分区(所有节点)
RHEL6
编辑 /etc/sysctl.conf 并添加 vm.swappiness=0
vi /etc/sysctl.conf
RHEL7
tuned 服务会动态调整系统参数,查找 tuned 中配置,直接将配置修改为 vm.swappiness=0
cd /usr/lib/tuned/
grep "vm.swappiness" * -R
9、安装jdk(所有节点)
yum -y install oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
/etc/profile 末尾添加以下内容
#set jdk environment
export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
10、安装 mysql(master)
wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql-community-server
systemctl start mysqld
systemctl status mysqld
grep "password" /var/log/mysqld.log
mysql -u root -p
登录后修改密码(尽量复杂一点,太简单了验证不通过,也可以查看如何关闭mysql校验规则)
ALTER USER 'root'@'localhost' IDENTIFIED BY 'Abc123!@#';
修改mysql外网访问:
增加允许远程访问的用户或者允许现有用户的远程访问。
给root授予在任意主机(%)访问任意数据库的所有权限。
use mysql;
update user set host='%' where user='root' and host='localhost';
exit
systemctl restart mysqld
启动服务并设置开机启动:
systemctl enable mysqld
11、安装 mysql JDBC 驱动(master)
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
12、为下表中每个服务创建 utf8 字符集的数据库:
Service | database | user |
---|---|---|
Cloudera Manager Server | scm | scm |
Activity Monitor | amon | amon |
Reports Manager | rman | rman |
Hue | hue | hue |
Hive Metastore Server | metastore | hive |
Sentry Server | sentry | sentry |
Cloudera Navigator Audit Server | nav | nav |
Cloudera Navigator Metadata Server | navms | navms |
Oozie | oozie | oozie |
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Abc123!@#';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Abc123!@#';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'Abc123!@#';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Abc123!@#';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'Abc123!@#';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'Abc123!@#';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'Abc123!@#';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'Abc123!@#';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Abc123!@#';
flush privileges;
13、安装 cloudera manager 包
master 节点:
yum -y install cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
slave 节点:
yum -y install cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
14、修改 agent 的服务器地址(所有节点)
打开并编辑 /etc/cloudera-scm-agent/config.ini 文件:
/etc/cloudera-scm-agent/config.ini
将 server_host=localhost 改为 master 节点主机名 cdh1-master
15、启动 agent 服务(所有节点)
systemctl start cloudera-scm-agent
16、准备 Cloudera Manager 数据库(master)
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
输入scm的密码
若已存在,则执行:
rm /etc/cloudera-scm-server/db.mgmt.properties
17、准备 parcel 包(master)
将已下载的 CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel 和 manifest.json 拷贝至 /opt/cloudera/parcel-repo/ 目录,并且生成 parcel 包的 sha 值:
sha1sum /opt/cloudera/parcel-repo/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
将 sha 值填入到 CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha 文件中
vi /opt/cloudera/parcel-repo/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha
18、启动 Cloudera Manager Server(master)
启动:
systemctl start cloudera-scm-server
19、查看启动日志:
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
看到下面的信息,说明启动完成
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
20、启用 Auto-TLS 以自动创建证书(master)
慎用:我用过之后agent连接不了server,导致我装了好多遍,跟证书配置有关系,熟悉可以试一下
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services
21、登录 web 页面(忽略许可授权,选择免费版)
http://192.168.1.211:7180
账号/密码:admin / admin
大功告成
更多推荐
所有评论(0)