安装单机Hadoop

安装单机hadoop

SSH 名密钥登录

ssh-keygen -t rsa
cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
1
2

下载文件

wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz

编辑配置文件

vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh 
export JAVA_HOME=/app/jre1.8.0_131
1
2

(此步骤不可省略)

vim $HADOOP_HOME/etc/hadoop/core-site.xml
1

<configuration>
    <property>
        <name>hadoop.tmp.dirname>
        <value>~/Documents/tmpvalue>
        <description>A base for other temporary directories.description>
    property>
    <property>
        <name>fs.default.namename>
        <value>hdfs://whoami.local:9000value>
    property>
configuration>
1
2
3
4
5
6
7
8
9
10
11

vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replicationname>
        <value>1value>
    property>
configuration>
1
2
3
4
5
6

格式化namenode

cd $HADOOP_HOME/bin/
./hdfs namenode -format
1
2

启动hdfs

cd ../sbin
./start-dfs.sh
1
2

访问页面

http://127.0.0.1:50070

Yarn 配置

cd $HADOOP_HOME/etc/hadoop
cp mapred-site.xml.template mapred-site.xml

vim mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.namename>
                <value>yarnvalue>
        property>
configuration>

vim yarn-site.xml

<configuration>
        <property>
                <name>yarn.nodemanager.aux-servicesname>
                <value>mapreduce_shufflevalue>
        property>
configuration>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

启动yarn服务

$HADOOP_HOME/sbin/start-yarn.sh 
1