【Sqoop】CDH版Hadoop、Hive及Sqoop的伪分布式环境搭建

BigData 专栏收录该内容
84 篇文章 8 订阅

1、CDH版本介绍

CDH版本由Cloudera公司的Doug Cutting组队开发,Cloudera公司提供认证:Hadoop开发工程师认证和Hadoop管理员认证。
CDH版本的特点:兼容性好,解决了各个框架之间的兼容协调问题,比Apache版本稳定。
CDH版本自动化的大数据管理平台:Cloudera Manager。
HDP版本自动化的大数据管理平台:Ambari。

2、CDH版Hadoop的伪分布式环境搭建

(1)因为/opt的所有者是root,切换到root用户,在/opt下新建目录cdh-5.3.6:opt]# mkdir cdh-5.3.6
更改cdh-5.3.6的用户和所属组为beifeng:opt]# chown -R beifeng:beifeng /opt/cdh-5.3.6/
退出root用户。
(2)在/opt/software/目录下新建cdh目录:software]$ mkdir cdh
向/opt/software/cdh目录上传下载好的CDH版本的Hadoop、Hive及Sqoop:
hadoop-2.5.0-cdh5.3.6.tar.gz
hive-0.13.1-cdh5.3.6.tar.gz
sqoop-1.4.5-cdh5.3.6.tar.gz
向/opt/software/cdh目录中的文件添加执行权限:cdh]$ chmod u+x ./*
(3)解压hadoop:cdh]$ tar -zxf hadoop-2.5.0-cdh5.3.6.tar.gz -C /opt/cdh-5.3.6/
则hadoop的目录为:/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6
(4)创建目录:$ mkdir -p /opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/data/tmp
(5)查看JAVA_HOME环境变量:
$ echo ${JAVA_HOME}
/opt/modules/jdk1.7.0_67
(6)配置/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

(7)配置/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/yarn-env.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

(8)配置/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/mapred-env.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

(9)配置/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hadoop-senior.ibeifeng.com:8020</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/data/tmp</value>
        </property>
        <property>
        		<name>hadoop.http.staticuser.user</name>
        		<value>beifeng</value>
        </property>
</configuration>

(10)配置/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/hdfs-site.xml

<configuration>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>hadoop-senior.ibeifeng.com:50090</value>
        </property>
        <property>
                <name>dfs.namenode.http-address</name>
                <value>hadoop-senior.ibeifeng.com:50070</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.permissions.enabled</name>
                <value>false</value>
        </property>
</configuration>

(11)配置/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/slaves

hadoop-senior.ibeifeng.com

(12)配置/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/yarn-site.xml

<configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>hadoop-senior.ibeifeng.com</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.log-aggregation-enable</name>
                <value>true</value>
        </property>
        <property>
                <name>yarn.log-aggregation.retain-seconds</name>
                <value>640800</value>
        </property>
</configuration>

(13) hadoop]$ cp mapred-site.xml.template mapred-site.xml
配置/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>hadoop-senior.ibeifeng.com:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>hadoop-senior.ibeifeng.com:19888</value>
        </property>
</configuration>

(14)可以替换一下native本地库,注意使用CDH版本的native本地库,把lib下的native替换过来。
(15)查看$ more /etc/profile中没有配置HADOOP_HOME等环境变量,否则需要改成当前CDH版的hadoop目录。
(16)删除/tmp目录下的所有文件:tmp]$ rm -rf /tmp/*
(17)格式化namenode
hadoop-2.5.0-cdh5.3.6]$ bin/hdfs namenode -format
(18)启动namenode、datanode、resourcemanager、nodemanager、historyserver。
hadoop-2.5.0-cdh5.3.6]$ sbin/hadoop-daemon.sh start namenode
hadoop-2.5.0-cdh5.3.6]$ sbin/hadoop-daemon.sh start datanode
hadoop-2.5.0-cdh5.3.6]$ sbin/yarn-daemon.sh start resourcemanager
hadoop-2.5.0-cdh5.3.6]$ sbin/yarn-daemon.sh start nodemanager
hadoop-2.5.0-cdh5.3.6]$ sbin/mr-jobhistory-daemon.sh start historyserver

3、CDH版Hive的伪分布式环境搭建

(1)进入/opt/software/cdh目录,解压hive安装文件:
cdh]$ tar -zxf hive-0.13.1-cdh5.3.6.tar.gz -C /opt/cdh-5.3.6/
(2)进入/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/conf目录,
conf]$ cp hive-env.sh.template hive-env.sh
配置/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/conf/hive-env.sh

HADOOP_HOME=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6
export HIVE_CONF_DIR=/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/conf

(3) conf]$ cp hive-log4j.properties.template hive-log4j.properties
配置/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/conf/hive-log4j.properties

hive.log.dir=/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/logs

(4) $ cp /opt/modules/hive-0.13.1/conf/hive-site.xml /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/conf/
修改hive-site.xml文件中的下列属性如下:

<configuration>
        <property>
               <name>javax.jdo.option.ConnectionURL</name>
               <value>jdbc:mysql://bigdata-senior.ibeifeng.com:3306/metadata?createDatabaseIfNotExist=true</value>
     	</property>
        <property>
               <name>javax.jdo.option.ConnectionDriverName</name>
               <value>com.mysql.jdbc.Driver</value>
        </property>
        <property>
               <name>javax.jdo.option.ConnectionUserName</name>
               <value>root</value>
        </property>
        <property>
               <name>javax.jdo.option.ConnectionPassword</name>
               <value>123456</value>
        </property>
        <property>
               <name>hive.metastore.warehouse.dir</name>
               <value>/user/hive/warehouse</value>
        </property>
        <property>
               <name>hive.cli.print.header</name>
               <value>true</value>
        </property>
        <property>
               <name>hive.cli.print.current.db</name>
               <value>true</value>
        </property>
</configuration>

(5)添加驱动类:$ cp /opt/modules/hive-0.13.1/lib/mysql-connector-java-5.1.27-bin.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib
(6)查看hdfs文件系统中有无/tmp目录:hadoop-2.5.0-cdh5.3.6]$ bin/hdfs dfs -ls -R /
(7)新建目录/user/hive/warehouse:hadoop-2.5.0-cdh5.3.6]$ bin/hdfs dfs -mkdir -p /user/hive/warehouse
(8)为目录/tmp和/user/hive/warehouse赋予写权限:
hadoop-2.5.0-cdh5.3.6]$ bin/hdfs dfs -chmod g+w /tmp
hadoop-2.5.0-cdh5.3.6]$ bin/hdfs dfs -chmod g+w /user/hive/warehouse
(9)检查hive是否安装成功:hive-0.13.1-cdh5.3.6]$ bin/hive

4、CDH版Sqoop的伪分布式环境搭建

(1)进入/opt/software/cdh目录中,解压sqoop安装文件:
cdh]$ tar -zxf sqoop-1.4.5-cdh5.3.6.tar.gz -C /opt/cdh-5.3.6/
(2)进入/opt/cdh-5.3.6/sqoop-1.4.5-cdh5.3.6/conf目录,
conf]$ cp sqoop-env-template.sh sqoop-env.sh
(3)配置/opt/cdh-5.3.6/sqoop-1.4.5-cdh5.3.6/conf/sqoop-env.sh:

export HADOOP_COMMON_HOME=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6
export HADOOP_MAPRED_HOME=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6
export HIVE_HOME=/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6

(4)导入连接mysql的jar包:
~]$ cd /opt/cdh-5.3.6/sqoop-1.4.5-cdh5.3.6/lib
lib]$ cp /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/mysql-connector-java-5.1.27-bin.jar
(5)输入sqoop-1.4.5-cdh5.3.6]$ bin/sqoop version,出现如下信息,说明配置成功:

18/07/02 17:27:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.6
Sqoop 1.4.5-cdh5.3.6
git commit id 
Compiled by jenkins on Tue Jul 28 15:08:32 PDT 2015

(6)用sqoop列出mysql中的数据库:

sqoop-1.4.5-cdh5.3.6]$ bin/sqoop list-databases \
--connect jdbc:mysql://hadoop-senior.ibeifeng.com:3306 \
--username root \
--password 123456
18/07/02 17:48:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.6
18/07/02 17:48:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/07/02 17:48:23 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
metadata
metastore
mysql
performance_schema
test

(7)用sqoop列出metastore数据库下的表:

bin/sqoop \
list-tables \
--connect jdbc:mysql://bigdata-senior01.ibeifeng.com:3306/metastore \
--username root \
--password 123456
  • 0
    点赞
  • 0
    评论
  • 0
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

打赏
文章很值,打赏犒劳作者一下
相关推荐
©️2020 CSDN 皮肤主题: 书香水墨 设计师:CSDN官方博客 返回首页

打赏

魏晓蕾

你的鼓励将是我创作的最大动力

¥2 ¥4 ¥6 ¥10 ¥20
输入1-500的整数
余额支付 (余额:-- )
扫码支付
扫码支付:¥2
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值