S3 client向ceph上传文件注意事项

背景：

众所知周，AWS S3 client功能强大，能与各类符合OSS标准的存储进行工作。本项目中刚好用到ceph，就是直接使用S3 java client上传文件。在根据ceph官网的的一些文档操作时遇到一些问题，这里记录下来，仅供遇到类似问题的网友参考。

问题与解决方法

按照ceph https://docs.ceph.com/en/quincy/radosgw/s3/java/ 官网的示例，在本人环境中无法遍历bucket或者upload文件， 在多方搜索对比测试后发现，需要添加几个参数才行，目前不确定是否是因为本地ceph的设置导致的问题， 这里记录下来仅供参考。

 //  disable SSL cert check
  String accessKey = "insert your access key here!";
  String secretKey = "insert your secret key here!";
  //具体的域名， 每个项目或者公司都不一样
  String cephHost = "http://cepha.b.com";
  System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY,"true");
    if (SDKGlobalConfiguration.isCertCheckingDisabled())
    {
      System.out.println("Cert checking is disabled");
    }

    // S3 Client configuration
    ClientConfiguration config = new ClientConfiguration();
    // Not the standard "AWS3SignerType", maar expliciet signerTypeV2
    config.setSignerOverride("S3SignerType");
    // or else Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch will happen. 这个也很重要，不然给putRequest 添加一些mata时会报SignatureDoesNotMatch的错误
    config.setSignerOverride("AWSS3V4SignerType");
    config.setProtocol(Protocol.HTTP);

    BasicAWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);

    AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.
      EndpointConfiguration(cephHost, "local-ceph");
// 重点就是这个 "local-ceph"， 不设置它，在本人项目中，可以list buckt但是不能upload file

    AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
      .withClientConfiguration(config)
      .withCredentials(new AWSStaticCredentialsProvider(credentials))
      .withEndpointConfiguration(endpointConfiguration)
      .enablePathStyleAccess()
      .disableChunkedEncoding()
      .build();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

对比一下ceph官网的文档

String accessKey = "insert your access key here!";
String secretKey = "insert your secret key here!";

AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 s3Client = new AmazonS3Client(credentials);
s3Client.setEndpoint("objects.dreamhost.com");
1
2
3
4
5
6

其他

在创建了AmazonS3 对象后记可以操作bucket中内容

 List<Bucket> buckets = s3Client.listBuckets();
1

上传文件，并带有meta信息，

try {
      ObjectMetadata metaData = new ObjectMetadata();
      Map<String, String> userMetaData = new HashMap<>();
      userMetaData.put("key1", "val1");
      userMetaData.put("key2", "val2");
      userMetaData.put("key3", "val3");
      metaData.setUserMetadata(userMetaData);

      PutObjectRequest putObjReq = new PutObjectRequest(bucketName, fileKey, file);
      putObjReq.setMetadata(metaData);

      PutObjectResult result =  s3Client.putObject(putObjReq);
      log.info("upload result {}", result);
    } catch (Exception ex) {
      log.error("fail to  upload to fileKey {}, exMsg:{}",
        fileKey, ex.getMessage(), ex);
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

如果上传的文件较大，使用TransferManager， 参看S3官方文档
https://docs.aws.amazon.com/zh_cn/sdk-for-java/v1/developer-guide/examples-s3-transfermanager.html

      PutObjectRequest putObjReq = new PutObjectRequest(bucketName, fileKey, file);
      putObjReq.setMetadata(metaData);

      TransferManager tm = TransferManagerBuilder.standard()
        .withS3Client(s3Client)
        .build();

      Upload upload = tm.upload(putObjReq, progressListener);
      log.info("start to upload file '{}' to fileKey:{},",
        file.getAbsolutePath(), fileKey);

      // wait for the upload to finish before continuing.
      upload.waitForCompletion();
1
2
3
4
5
6
7
8
9
10
11
12
13