搭了一个hadoop3.3.6的hadoop完全分布式测试集群,在测试阶段发现namenode和yarn都能正常启动,它们的web页面访问也正常,但是当我运行bin目录下的命令时运行线程会把配置文件里面namenode的服务组ID当做一个DNS的域名去解析,这一点该如何调整呢?
[root@hdp4 wy]# hadoop fs -get /
2024-06-16 17:49:31,873 WARN fs.FileSystem: Failed to initialize filesystem hdfs://hdp: java.lang.IllegalArgumentException: java.net.UnknownHostException: hdp
-get: java.net.UnknownHostException: hdp
Usage: hadoop fs [generic options]
[-appendToFile [-n] ... ]
[-cat [-ignoreCrc] ...]
[-checksum [-v] ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] [,MODE] ... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-concat ...]
[-copyFromLocal [-f] [-p] [-l] [-d] [-t ] [-q ] ... ]
[-copyToLocal [-f] [-p] [-crc] [-ignoreCrc] [-t ] [-q ] ... ]
[-count [-q] [-h] [-v] [-t [] ] [-u] [-x] [-e] [-s] ...]
[-cp [-f] [-p | -p[topax]] [-d] [-t ] [-q ] ... ]
[-createSnapshot [] ]
[-deleteSnapshot ]
[-df [-h] [ ...] ]
[-du [-s] [-h] [-v] [-x] ...]
[-expunge [-immediate] [-fs ] ]
[-find ... ...]
[-get [-f] [-p] [-crc] [-ignoreCrc] [-t ] [-q ] ... ]
[-getfacl [-R] ]
[-getfattr [-R] {-n name | -d} [-e en] ]
[-getmerge [-nl] [-skip-empty-file] ]
[-head ]
[-help [cmd ...]]
[-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [ ...] ]
[-mkdir [-p] ...]
[-moveFromLocal [-f] [-p] [-l] [-d] ... ]
[-moveToLocal ]
[-mv ... ]
[-put [-f] [-p] [-l] [-d] [-t ] [-q ] ... ]
[-renameSnapshot ]
[-rm [-f] [-r|-R] [-skipTrash] [-safely] ...]
[-rmdir [--ignore-fail-on-non-empty] ...]
[-setfacl [-R] [{-b|-k} {-m|-x } ] |[--set ] ]
[-setfattr {-n name [-v value] | -x name} ]
[-setrep [-R] [-w] ...]
[-stat [format] ...]
[-tail [-f] [-s ] ]
[-test -[defswrz] ]
[-text [-ignoreCrc] ...]
[-touch [-a] [-m] [-t TIMESTAMP (yyyyMMdd:HHmmss) ] [-c] ...]
[-touchz ...]
[-truncate [-w] ...]
[-usage [cmd ...]]
Generic options supported are:
-conf specify an application configuration file
-D define a value for a given property
-fs specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt specify a ResourceManager
-files specify a comma-separated list of files to be copied to the map reduce cluster
-libjars specify a comma-separated list of jar files to be included in the classpath
-archives specify a comma-separated list of archives to be unarchived on the compute machines
The general command line syntax is:
command [genericOptions] [commandOptions]
Usage: hadoop fs [generic options] -get [-f] [-p] [-crc] [-ignoreCrc] [-t ] [-q ] ...
问题就出现在2024-06-16 17:49:31,873 WARN fs.FileSystem: Failed to initialize filesystem hdfs://hdp: java.lang.IllegalArgumentException: java.net.UnknownHostException: hdp -get: java.net.UnknownHostException: hdp这一块。
hdp是我在配置文件中高可用的namenode服务组ID,在我的core-site.xml中存在如下配置
<configuration>
<property>
<name>fs.defaultFSname>
<value>hdfs://hdpvalue>
property>
<property>
<name>hadoop.tmp.dirname>
<value>/opt/hadoop-3.3.6/hdpData/tmpvalue>
property>
<property>
<name>hadoop.http.staticuser.username>
<value>rootvalue>
property>
<property>
<name>ha.zookeeper.quorumname>
<value>hdp4:2181,hdp5:2181,hdp6:2181value>
property>
<property>
<name>ipc.client.connect.max.retriesname>
<value>100value>
property>
<property>
<name>ipc.client.connect.retry.intervalname>
<value>10000value>
property>
configuration>
在hdfs-site.xml中也存在如下配置
<property>
<name>dfs.nameservicesname>
<value>hdpvalue>
property>
我尝试了在环境变量中明确指定hadoop配置文件路径但是没有影响这个问题的发生。
