大数据安全

一、Kerberos

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html

http://web.mit.edu/kerberos
http://web.mit.edu/kerberos/krb5-current/doc/admin/admin_commands/kadmin_local.html#commands
在这里插入图片描述
介绍
https://www.cnblogs.com/wuyongyin/p/15624452.html
https://godatadriven.com/blog/kerberos-basics-and-installing-a-kdc/
https://www.ibm.com/docs/zh/storage-scale/4.2.0?topic=security-kerberos-mode

安装部署
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/managing_smart_cards/configuring_a_kerberos_5_server

西北偏北UP
https://www.cnblogs.com/niceshot/p/13199203.html
https://www.cnblogs.com/niceshot/p/13216455.html
https://www.cnblogs.com/niceshot/p/14906696.html
https://mp.weixin.qq.com/s?__biz=MzI4OTY3MTUyNg==&mid=2247484735&idx=1&sn=b021eb28562d566b5d3c97f3d4024905
https://blog.csdn.net/u011026329/article/details/79167884

https://www.cnblogs.com/yinzhengjie2020/p/13616881.html
https://zhuanlan.zhihu.com/p/392506380

https://docs.oracle.com/cd/E19253-01/819-7061/seamtm-1/index.html

云厂商
新华三H3C
https://www.h3c.com/cn/d_202305/1843598_30005_0.htm
https://www.h3c.com/cn/pub/Document_Center/2021/01/H3C_DataEngine_SJGC_E5103P02-5W102_WebHelp/help/creatDatasource.html
UCloud
https://github.com/UCloudDoc-Team/USDP/blob/master/developer/ranger/ranger_hive.md
华为云 Kerberos
https://support.huaweicloud.com/mrs_faq/mrs_03_1167.html
腾讯云 Kerberos
https://cloud.tencent.com/document/product/589/44251
https://github.com/tencentyun/qcloud-documents/tree/master/product/%E5%A4%A7%E6%95%B0%E6%8D%AE%E4%B8%8EAI/%E5%BC%B9%E6%80%A7MapReduce/EMR%20%E5%BC%80%E5%8F%91%E6%95%99%E7%A8%8B/Kerberos%E4%BD%BF%E7%94%A8%E6%8C%87%E5%8D%97
阿里云 Kerberos
https://help.aliyun.com/zh/emr/emr-on-ecs/user-guide/connect-to-an-external-kdc
CDP集群高安全Kerberos+Ranger使用
https://help.aliyun.com/zh/cdp/user-guide/use-kerberos-and-ranger-in-a-cdp-ha-cluster

Kerberos 身份验证在 ChunJun 中的落地实践（袋鼠云）
https://developer.aliyun.com/article/1115235
https://developer.aliyun.com/article/1125173
https://developer.aliyun.com/article/1254761
https://developer.aliyun.com/article/1276369
https://developer.aliyun.com/article/25636

https://blog.51cto.com/zhangxueliang/2967427
http://support.supermap.com.cn/DataWarehouse/WebDocHelp/iServer/Server_Service_Management/Spark_cluster/yarn_kerberose_using.htm

尚硅谷
《尚硅谷大数据项目之尚品汇7用户认证KerberosV4.1.docx》
《尚硅谷大数据项目之尚品汇8安全环境实战V4.0.docx》
https://www.itjc8.com/thread-11765-1-1.html
280 尚硅谷用户认证 Kerberos概述
https://www.youtube.com/watch?v=GVnj52WGs_Q

黑马程序员
https://www.bilibili.com/video/BV1pV411k7ut

https://kmgy.top/doc/323
https://www.cnblogs.com/30go/p/16376826.html
https://www.iizhi.cn/resource/detail/e2b81e11363049e6808ad86fd4dda90c

https://blog.csdn.net/h952520296/article/details/130869070
https://blog.csdn.net/h952520296/article/details/127404776
https://cloud.tencent.com/developer/article/1496451

Kerberos 部署后端口和进程列表：
在这里插入图片描述

Kerberos 命令

认证
kinit -kt /etc/security/keytab/nn.service.keytab nn/100.realtime.hadoop.fql.com

查看认证信息
klist -e -k -t /etc/security/keytab/dn.service.keytab
1
2
3
4
5

二、Ranger

《尚硅谷大数据项目之尚品汇9权限管理RangerV4.0.doc》
https://xie.infoq.cn/article/7b79cbafa5eed708a402f2f90

Ranger整合HDFS
https://mp.weixin.qq.com/s/WUR0Py1MTokA-IwlLjr6eA
Ranger整合Hive
https://mp.weixin.qq.com/s?__biz=MzIyMTE1Nzk0OA==&mid=2247489679&idx=1&sn=2f25f13c0607c7af7b5c86dd6bc37416

在这里插入图片描述

编译

mvn clean compile package assembly:assembly install
mvn clean compile package assembly:assembly install -DskipTests -Dspotbugs.skip=true -Dcheckstyle.skip=true -Drat.skip=true
编译指定hive-agent模块
mvn -U -pl hive-agent clean package -DskipTests -Dspotbugs.skip=true -Dcheckstyle.skip=true

在这里插入图片描述

https://www.cnblogs.com/yjt1993/p/11837398.html
https://www.cnblogs.com/zhenxiLi-2017/p/11798725.html

https://cloud.tencent.com/developer/article/1746603
https://blog.csdn.net/mnasd/article/details/80617999
https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+Installation
https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5±+User+Guide

https://www.jianshu.com/p/d9941b8687b7
https://www.slideshare.net/Hadoop_Summit/securing-hadoop-with-apache-ranger
https://www.slideshare.net/HadoopSummit/security-and-data-governance-using-apache-ranger-and-apache-atlas

原创-在kerberos+HA环境下的ranger编译安装
https://xiuechen.github.io/2017/04/13/%E5%9C%A8kerberos-HA%E7%8E%AF%E5%A2%83%E4%B8%8B%E7%9A%84ranger%E7%BC%96%E8%AF%91%E5%AE%89%E8%A3%85/

spark远程调试
https://xiuechen.github.io/2018/03/02/spark%E8%BF%9C%E7%A8%8B%E8%B0%83%E8%AF%95/

https://cloud.tencent.com/document/product/589/55236

https://cwiki.apache.org/confluence/display/RANGER/Row-level+filtering+and+column-masking+using+Apache+Ranger+policies+in+Apache+Hive
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868896

Ranger 部署后进程列表：
在这里插入图片描述

三、其他

HDFS 中的 POSIX 权限模型实现机制
https://edu.51cto.com/lesson/838534.html
https://www.cnblogs.com/niceshot/p/12901539.html

https://patents.google.com/patent/CN106375323A/zh

Hadoop官网

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html
https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-common/SecureMode.html
https://hadoop.apache.org/docs/r2.7.7/hadoop-project-dist/hadoop-common/SecureMode.html

https://hadoop.apache.org/docs/r2.7.7/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

四、问题解决

【已解决】1、问题1，ranger整合Hive时，ranger日志中有异常信息 javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
解决，
cd /usr/local/fqlhadoop/ranger/ranger-0.5.4-SNAPSHOT-admin
vim ews/ranger-admin-services.sh

新增 -Djavax.security.auth.useSubjectCredsOnly=false

start() {
        java -Djavax.security.auth.useSubjectCredsOnly=false -Dproc_rangeradmin ${JAVA_OPTS} -Dlogdir=${XAPOLICYMGR_EWS_DIR}/logs/ -Dcatalina.base=${XAPOLICYMGR_EWS_DIR} -cp "${XAPOLICYMGR_EWS_DIR}/webapp/WEB-INF/classes/conf:${XAPOLICYMGR_EWS_DIR}/lib/*:${RANGER_JAAS_LIB_DIR}/*:${RANGER_JAAS_CONF_DIR}:${JAVA_HOME}/lib/*:$CLASSPATH" org.apache.ranger.server.tomcat.EmbeddedServer > logs/catalina.out 2>&1 &
        echo "Apache Ranger Admin has started."
}
1
2
3
4
5
6

参考
https://blog.csdn.net/qq_21383435/article/details/124326190
https://www.cnblogs.com/slankka/p/10217038.html
https://developer.aliyun.com/article/1115235
https://stackoverflow.com/questions/33829017/gssexception-no-valid-credentials-provided-mechanism-level-failed-to-find-any

【未解决】2、问题2
windows 环境下通过 DataGrip JDBC方式连接Kerberos Hive
https://intellij-support.jetbrains.com/hc/en-us/community/posts/4409692344082-Hive-driver-class-not-found
https://querysurge.zendesk.com/hc/en-us/articles/115001218863-Setting-Up-a-Hive-Connection-with-Kerberos-using-Apache-JDBC-Drivers-Windows

【已解决】3、问题3，NameNode连接JournalNode有异常信息

2023-11-12 21:11:33,739 ERROR org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception initializing https://1.common2.hadoop.fql.com:8481/getJournal?jid=common2&segmentTxId=1&storageInfo=-63%3A2120035820%3A1699144549183%3ACID-32487aa5-1b0e-4000-a712-784b0116dd33
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:347)
        at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:218)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1938)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:514)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:508)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:158)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:209)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:267)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:190)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:190)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:222)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:167)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:757)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:335)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1073)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:695)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:736)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:961)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:940)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1714)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1782)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
        ... 44 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
        ... 50 more        
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

参考
hadoop集群安装HTTPS服务（包括生成CA证书）
https://blog.csdn.net/weixin_40496191/article/details/128522371

https://www.cnblogs.com/swordfall/p/13301097.html
https://blog.csdn.net/hncscwc/article/details/126964637

相关阅读:
阿里云技术专家杨泽强：弹性计算云上可观测能力构建
 postgresql
【操作系统】BIOS篇
 三菱Q系列PLC远程调试并实现4G/5G数据通讯？
设计模式学习（十二）：享元模式
 Linux文件/目录管理
 校正叠加（calibrated stacking）方法—技术
 PYTHON链家租房数据分析：岭回归、LASSO、随机森林、XGBOOST、KERAS神经网络、KMEANS聚类、地理可视化...
使用Go env命令设置Go的环境
 alibaba.fastjson的使用（三）-- Map、List ==》JSON字符串
原文地址：https://blog.csdn.net/wl101yjx/article/details/133700844