• PF-Net基于深度学习的点云补全网络

    目录

    1. 论文和代码

    2. 论文阅读笔记

    2.1 目的和框架

    2.2 IFPS 下采样

    3. 源码解读

    3.1 载入数据

    3.1.1 归一化操作

    3.2 数据前处理

    3.3 网络输入输出

    3.3.1 判别器训练

    3.3.2 生成器训练

    3.4 判别器模型

    3.5 生成器模型

    3.5.1 CMLP

    3.5.2 Final Feature Vector V

    3.5.3 生成器主代码 

    3.6 测试效果

    1. 论文和代码

    论文：Point Fractal Network for 3D Point Cloud Completionhttps://openaccess.thecvf.com/content_CVPR_2020/papers/Huang_PF-Net_Point_Fractal_Network_for_3D_Point_Cloud_Completion_CVPR_2020_paper.pdficon-default.png?t=N7T8https://openaccess.thecvf.com/content_CVPR_2020/papers/Huang_PF-Net_Point_Fractal_Network_for_3D_Point_Cloud_Completion_CVPR_2020_paper.pdf

    作者来自上海交通大学和上汤科技的大佬，发表在2020CVPR。 

    代码： 

    https://github.com/zztianzz/PF-Net-Point-Fractal-Networkicon-default.png?t=N7T8https://github.com/zztianzz/PF-Net-Point-Fractal-Network

    2. 论文阅读笔记

    2.1 目的和框架

            该PF-Net要做的是点云补全，即将有残缺的点云数据（比如上图飞机少了机头，或者凳子少了腿），通过一些技术补全为完整的点云数据。

     简单来讲，PF-Net输入残缺后点云（飞机的机身），输出残缺的部分点云（飞机的机尾），端对端训练，作为生成器网络，生成残缺点云，再接一个判别器网络。

            该网络的特点：不改变原始的数据，只生成残缺部分的点云数据。即机身的点云数据不变，直接生成机头部分的点云。

           算法步骤：

    （1）原始的黄色点云输入数据，经过了两次IFPS下采样，获得三种尺度的点云输入数据，其中N是原始的点云中点的个数，k是下采样倍数；

    （2）再经过CMLP全链接网络，获得Latent vector F；

    （3）再将各个latent vector拼接起来获得Final Laten Map M；

    （4）接一个MLP和Linear全链接网络，再使用FPN特征金字塔作为解码网络，获取三种尺度下的残缺点云数据；

    （5）对原始尺度下的残缺点云预测加一个判别器网络，使其生成的残缺数据更真实。

    下面对各个部件，从输入到输出一个一个梳理。

    2.2 IFPS 下采样

            Iterative farthest point sampling (IFPS)，迭代最远点采样（技术来自Pointnet++），采集点云数据中骨架点点集合，通俗的将不破坏点云整体结构的情况下，就是只保留一些点。用该技术进行才采样比CNNs更快。

    上图，原始台灯有 2048个点，即使下采样到128个点（保留了6.25%），依然很好的保留了台灯的基本骨架。

    实现参考iterative farthest point sample (IFPS or FPS)_Mr.Q的博客-CSDN博客迭代最远距离采样，在点云论文PointNet++和PF-Net中用于对点云数据下采样。https://blog.csdn.net/jizhidexiaoming/article/details/128198099?spm=1001.2014.3001.5501

    3. 源码解读

    3.1 载入数据

    shapenet_part_loader.py

    1. # from __future__ import print_function
    2. import torch.utils.data as data
    3. import os
    4. import os.path
    5. import torch
    6. import json
    7. import numpy as np
    8. import sys
    9.  
    10. BASE_DIR = os.path.dirname(os.path.abspath(__file__))
    11. dataset_path = os.path.abspath(
    12.     os.path.join(BASE_DIR, '../dataset/shapenet_part/shapenetcore_partanno_segmentation_benchmark_v0/'))
    13.  
    14.  
    15. class PartDataset(data.Dataset):
    16.     def __init__(self, root=dataset_path, npoints=2500, classification=False, class_choice=None, split='train',
    17.                  normalize=True):
    18.         """
    20.         Parameters
    21.         ----------
    22.         root: str. 数据集完整路径
    23.         npoints: 2048. the point number of a sample. 输入到网络中点云的点个数。
    24.         classification: bool. True. "Airplane" or "Mug" or something else.
    25.         class_choice: list. None. 训练指定的类别。
    26.         split: str. train/test
    27.         normalize: bool. 是否归一化
    28.         """
    29.         self.npoints = npoints
    30.         self.root = root
    31.         self.catfile = os.path.join(self.root, 'synsetoffset2category.txt')  # 映射表格
    32.         self.cat = {}  # 存放映射字典， {airplane: 11231414, ...}
    33.         self.classification = classification
    34.         self.normalize = normalize
    35.  
    36.         with open(self.catfile, 'r') as f:
    37.             for line in f:
    38.                 ls = line.strip().split()
    39.                 self.cat[ls[0]] = ls[1]
    40.         # print(self.cat)
    41.         if not class_choice is None:
    42.             self.cat = {k: v for k, v in self.cat.items() if k in class_choice}
    43.             print(self.cat)
    44.         self.meta = {}
    45.         with open(os.path.join(self.root, 'train_test_split', 'shuffled_train_file_list.json'), 'r') as f:
    46.             train_ids = set([str(d.split('/')[2]) for d in json.load(f)])  # 点云文件名称
    47.         with open(os.path.join(self.root, 'train_test_split', 'shuffled_val_file_list.json'), 'r') as f:
    48.             val_ids = set([str(d.split('/')[2]) for d in json.load(f)])
    49.         with open(os.path.join(self.root, 'train_test_split', 'shuffled_test_file_list.json'), 'r') as f:
    50.             test_ids = set([str(d.split('/')[2]) for d in json.load(f)])
    51.  
    52.         # 获取datapath list [("Airplane", 点云文件路径，分割文件路径，点云文件夹id，点云文件名称), ...]
    53.         for item in self.cat:
    54.             # print('category', item)
    55.             self.meta[item] = []  # {"Airplane": [(点云文件路径,分割文件路径,点云类别id,点云文件名称), ...],
    56.                                   #  "": [], ...}
    57.             dir_point = os.path.join(self.root, self.cat[item], 'points')  # 当前类别的点云文件夹路径
    58.             dir_seg = os.path.join(self.root, self.cat[item], 'points_label')  # 当前类别的分割文件夹路径
    59.             # print(dir_point, dir_seg)
    60.             fns = sorted(os.listdir(dir_point))  # 当前类别的所有点云文件名
    61.             if split == 'trainval':
    62.                 fns = [fn for fn in fns if ((fn[0:-4] in train_ids) or (fn[0:-4] in val_ids))]
    63.             elif split == 'train':
    64.                 fns = [fn for fn in fns if fn[0:-4] in train_ids]  # 获取所有属于训练集的点云文件名称
    65.             elif split == 'val':
    66.                 fns = [fn for fn in fns if fn[0:-4] in val_ids]
    67.             elif split == 'test':
    68.                 fns = [fn for fn in fns if fn[0:-4] in test_ids]
    69.             else:
    70.                 print('Unknown split: %s. Exiting..' % (split))
    71.                 sys.exit(-1)
    72.  
    73.             for fn in fns:  #
    74.                 token = (os.path.splitext(os.path.basename(fn))[0])  # 获取点云文件名称
    75.                 self.meta[item].append((os.path.join(dir_point, token + '.pts'), os.path.join(dir_seg, token + '.seg'),
    76.                                         self.cat[item], token))  # {"Airplane": [(点云文件路径,分割文件路径,点云文件夹id,点云文件名称), ...]}
    77.         self.datapath = []  # [("Airplane", 点云文件路径，分割文件路径，点云文件夹id，点云文件名称), ...]
    78.         for item in self.cat:
    79.             for fn in self.meta[item]:
    80.                 self.datapath.append((item, fn[0], fn[1], fn[2], fn[3]))
    81.         # ["cls_name": cls_id, ...]
    82.         self.classes = dict(zip(sorted(self.cat), range(len(self.cat))))  # {"Airplane": 0, "", 1, ...} 按首字母排序。
    83.         print(self.classes)
    84.         self.num_seg_classes = 0
    85.         if not self.classification:
    86.             for i in range(len(self.datapath) // 50):
    87.                 l = len(np.unique(np.loadtxt(self.datapath[i][2]).astype(np.uint8)))
    88.                 if l > self.num_seg_classes:
    89.                     self.num_seg_classes = l
    90.         # print(self.num_seg_classes)
    91.         self.cache = {}  # from index to (point_set, cls, seg) tuple
    92.         self.cache_size = 18000  # 加载一次后，不会重复加载
    93.  
    94.     def __getitem__(self, index):
    95.         if index in self.cache:  # 加载一次后，不会重复加载，所以如果在缓存中，直接取出来即可。
    96.             #            point_set, seg, cls= self.cache[index]
    97.             point_set, seg, cls, foldername, filename = self.cache[index]
    98.         else:
    99.             fn = self.datapath[index]
    100.             # 1. cls. "Mug"类别id是11
    101.             cls = self.classes[self.datapath[index][0]]
    102.             # 2. point_set
    103.             point_set = np.loadtxt(fn[1]).astype(np.float32)  # (2817, 3). 载入点云，并转成float32类型
    104.             if self.normalize:
    105.                 point_set = self.pc_normalize(point_set)
    106.             # 3. seg
    107.             seg = np.loadtxt(fn[2]).astype(np.int64) - 1  # 分割类别id
    108.             # 4. foldername 点云文件夹
    109.             foldername = fn[3]
    110.             # 5. filename 点云文件名称
    111.             filename = fn[4]
    112.             if len(self.cache) < self.cache_size:  # 载入缓存，以便下次迭代时使用
    113.                 self.cache[index] = (point_set, seg, cls, foldername, filename)
    114.  
    115.         # 随机选择npoints个点参与训练
    116.         choice_idx = np.random.choice(len(seg), self.npoints, replace=True)  # 其实可以不用seg文件来随机
    117.         # resample
    118.         point_set = point_set[choice_idx, :]
    119.         seg = seg[choice_idx]
    120.  
    121.         # To Pytorch
    122.         point_set = torch.from_numpy(point_set)  # (2048,3)
    123.         seg = torch.from_numpy(seg)  # (2048,)
    124.         cls = torch.from_numpy(np.array([cls]).astype(np.int64))  # (1,)
    125.         if self.classification:
    126.             return point_set, cls
    127.         else:
    128.             return point_set, seg, cls
    129.  
    130.     def __len__(self):
    131.         return len(self.datapath)
    132.  
    133.     def pc_normalize(self, pc):
    134.         """ pc: NxC, return NxC """
    135.         # l = pc.shape[0]
    136.         centroid = np.mean(pc, axis=0)  # [-0.00400733  0.14655513  0.0053034 ]
    137.         pc = pc - centroid  # 所有的值减去均值
    138.         m = np.max(np.sqrt(np.sum(pc ** 2, axis=1)))  # sqrt(x1^2+y1^2+z1^2) + sqrt(x2^2+y2^2+z2^2)+...  0.55
    139.         pc = pc / m
    140.         return pc
    141.  
    142.  
    143. if __name__ == '__main__':
    144.     dset = PartDataset(root='./dataset/shapenetcore_partanno_segmentation_benchmark_v0/', classification=True,
    145.                        class_choice=None, npoints=4096, split='train')
    146.     #    d = PartDataset( root='./dataset/shapenetcore_partanno_segmentation_benchmark_v0/',classification=False, class_choice=None, npoints=4096, split='test')
    147.     print(len(dset))
    148.     ps, cls = dset[10000]
    149.     print(cls)
    150. #    print(ps.size(), ps.type(), cls.size(), cls.type())
    151. #    print(ps)
    152. #    ps = ps.numpy()
    153. #    np.savetxt('ps'+'.txt', ps, fmt = "%f %f %f")

    3.1.1 归一化操作

    （1）坐标值减去各自坐标值的均值；

    （2）sqrt(x1^2+y1^2+z1^2) + sqrt(x2^2+y2^2+z2^2)+...  == 0.55

    （3）坐标值 / 0.55

    3.2 数据前处理

    Trian_PFNet.py

    1. dset = shapenet_part_loader.PartDataset(
    2.     root='/home/zxq/code/python/PF-Net-Point-Fractal-Network/dataset/shapenetcore_partanno_segmentation_benchmark_v0/',
    3.     classification=True, 
    4.     class_choice=None, 
    5.     npoints=opt.pnum, 
    6.     split='train')
    7. assert dset
    8. dataloader = torch.utils.data.DataLoader(dset, batch_size=opt.batchSize, shuffle=True, num_workers=int(opt.workers))
    9.  
    10. real_label = 1
    11. fake_label = 0
    12.  
    13. for i, data in enumerate(dataloader, 0):
    14.  
    15.     real_point, target = data  # 点云坐标(b,2048,3). 点云类别(b,1) (Airplane or Mug).
    16.  
    17.     batch_size = real_point.size()[0]
    18.     real_center = torch.FloatTensor(batch_size, 1, opt.crop_point_num, 3)  # (b,1,512,3). # 保存裁剪点的坐标
    19.     input_cropped1 = torch.FloatTensor(batch_size, opt.pnum, 3)  # (b,2048,3). 原始点云数据的坐标，后面将裁剪掉crop_point_num个点
    20.     input_cropped1 = input_cropped1.data.copy_(real_point)  # input_cropped1的地址指向没变，只是重新赋值。
    21.     real_point = torch.unsqueeze(real_point, 1)  # (b,2048,3) -> (b,1,2048,3)
    22.     input_cropped1 = torch.unsqueeze(input_cropped1, 1)  # (b,2048,3) -> (b,1,2048,3)
    23.     p_origin = [0, 0, 0]
    24.  
    25.     # 计算点云和各自视点之间的距离，并从小到大排序；裁剪点云
    26.     # input_cropped1被裁剪后的点云，real_center是被裁剪下来的点云
    27.     # Set viewpoints
    28.     vp_choice_list = [torch.Tensor([1, 0, 0]), torch.Tensor([0, 0, 1]), torch.Tensor([1, 0, 1]),
    29.                       torch.Tensor([-1, 0, 0]), torch.Tensor([-1, 1, 0])]
    30.     for m in range(batch_size):  # 计算batch中所有点云距离vp
    31.         cur_vp_index = random.sample(vp_choice_list, 1)  # Random choose one of the viewpoint
    32.         p_center = cur_vp_index[0]  # eg. [1,0,0]
    33.         distance_list = []  # 点和各自vp之间的距离
    34.         for n in range(opt.pnum):  # 点云中第n个点
    35.             distance_list.append(distance_squre(real_point[m, 0, n], p_center))  # 当前点和vp之间的距离
    36.         distance_order = sorted(enumerate(distance_list), key=lambda x: x[1])  # enumerate使其变成2维，x[1]第二维度
    37.         # 裁剪掉距离视点最近的前crop_point_num个点
    38.         for sp in range(opt.crop_point_num):  # distance_order[sp] == (point_idx, dist_val)
    39.             input_cropped1.data[m, 0, distance_order[sp][0]] = torch.FloatTensor([0, 0, 0])  # 坐标置为0
    40.             real_center.data[m, 0, sp] = real_point[m, 0, distance_order[sp][0]]  # 保存裁剪点的坐标
    41.  
    42.     label.resize_([batch_size, 1]).fill_(real_label)  # (b,) -> (b,1).  填充1
    43.  
    44.     # to cuda
    45.     real_point = real_point.to(device)  # (b,1,2048,3) 原始完整点云坐标数据
    46.     real_center = real_center.to(device)  # (b,1,512,3) 被裁剪下来的点云
    47.     input_cropped1 = input_cropped1.to(device)  # (b,1,2048,3) 被裁剪后的点云
    48.     label = label.to(device)  # (2,1) 1是真实，0是生成
    49.  
    50.     ############################
    51.     # (1) data prepare
    52.     ###########################
    53.     # 被裁剪下来的点云
    54.     # scale 0
    55.     real_center = Variable(real_center, requires_grad=True)
    56.     real_center = torch.squeeze(real_center, 1)  # (b,1,512,3) -> (b,512,3)
    57.     # scale 1
    58.     real_center_key1_idx = utils.farthest_point_sample(real_center, 64, RAN=False)  # 提取64个点作为骨架点
    59.     real_center_key1 = utils.index_points(real_center, real_center_key1_idx)
    60.     real_center_key1 = Variable(real_center_key1, requires_grad=True)
    61.     # scale 2
    62.     real_center_key2_idx = utils.farthest_point_sample(real_center, 128, RAN=True)  # 提取128个点作为骨架点
    63.     real_center_key2 = utils.index_points(real_center, real_center_key2_idx)  # 被裁剪下来的点云
    64.     real_center_key2 = Variable(real_center_key2, requires_grad=True)
    65.     # 被裁剪后的点云
    66.     # scale 0
    67.     input_cropped1 = torch.squeeze(input_cropped1, 1)  # (b,1,2048,3) -> (b,512,3)
    68.     # scale 1
    69.     input_cropped2_idx = utils.farthest_point_sample(input_cropped1, opt.point_scales_list[1], RAN=True)  # 1024
    70.     input_cropped2 = utils.index_points(input_cropped1, input_cropped2_idx)
    71.     # scale 2
    72.     input_cropped3_idx = utils.farthest_point_sample(input_cropped1, opt.point_scales_list[2], RAN=False)  # 512
    73.     input_cropped3 = utils.index_points(input_cropped1, input_cropped3_idx)
    74.  
    75.     input_cropped1 = Variable(input_cropped1, requires_grad=True)
    76.     input_cropped2 = Variable(input_cropped2, requires_grad=True)
    77.     input_cropped3 = Variable(input_cropped3, requires_grad=True)
    78.  
    79.     # to cuda
    80.     input_cropped2 = input_cropped2.to(device)
    81.     input_cropped3 = input_cropped3.to(device)
    82.     input_cropped = [input_cropped1, input_cropped2, input_cropped3]  # 被裁剪后的点云 from diff scales

     得到数据：

    real_center: (b,512,3).  被裁剪下来的点云

    input_cropped: list of tensor. (b,2048,3), (b,1024,3), (b,512,3) . 裁剪后的点云

    label_center: (b,1). 0/1是否是真是点云

    real_center_key1: (b,128,3). 被裁剪下来的点云（下次样）

    real_center_key2: (b,64,3). 被裁剪下来的点云（下次样）

    3.3 网络输入输出

    3.3.1 判别器训练

    （1）输入真实的被裁剪下来的点云，判别器进行判断，计算errD_real_loss；

    （2）利用被裁剪后的点云，生成假的被裁剪下来的点云，再经过判别器，计算errD_fake_loss；

    判别器的目标是：

    • 真的判定为真的，即图中real_center的预测值越接近1，损失越小； 
    • 假的判定为假的，即图中fake的预测值越接近0，损失越小。

     对应的代码

    1. point_netG = point_netG.train()
    2. point_netD = point_netD.train()
    3. ############################
    4. # (2) Update D network
    5. ###########################
    6. point_netD.zero_grad()
    7. real_center = torch.unsqueeze(real_center, 1)  # (b,512,3) -> (b,1,512,3)
    8. output = point_netD(real_center)  # (b,1,512,3). output: (b,1)
    9. # label: (b,1) fill with 1. 对于判别器来说，output值越大越好，损失值越小
    10. errD_real = criterion(output, label)
    11. errD_real.backward()
    12.  
    13. # input_cropped: (2,2048,3)/(2,1024,3)/(2,512,3). fake_1: (b,64,3), fake_2: (b,128,3), fake: (b,512,3).
    14. fake_center1, fake_center2, fake = point_netG(input_cropped)
    15. fake = torch.unsqueeze(fake, 1)  # (b,512,3) -> (b,1,512,3)
    16. label.data.fill_(fake_label)  # (b,1). label赋值为0
    17. output = point_netD(fake.detach())  # output: (b,1)
    18. # label: (b,1) fill with 0. 对于判别器来说，output值越小越好，损失值越小
    19. errD_fake = criterion(output, label)  #
    20. errD_fake.backward()
    21.  
    22. errD = errD_real + errD_fake  # errD 没有参与训练，只是用于打印，没啥其他用处。
    23.  
    24. optimizerD.step()

    3.3.2 生成器训练

    对图中生成的4个fake点云进行学习，降低损失函数。

    1. ############################
    2. # (3) Update G network: maximize log(D(G(z)))
    3. ###########################
    4. point_netG.zero_grad()
    5. label.data.fill_(real_label)  # (b,1). label赋值为1
    6. # fake: (b,1,512,3). output: (b,1)。利用更新后的判别器再次判断fake数据
    7. output = point_netD(fake)
    8. errG_D = criterion(output, label)  # tensor(0.5747)
    9.  
    10. # fake: (b,1,512,3) -> (b,512,3), real_center: (b,1,512,3) -> (b,512,3)
    11. CD_LOSS = criterion_PointLoss(torch.squeeze(fake, 1), torch.squeeze(real_center, 1))  # 只是打印，没有参与训练
    12.  
    13. # 生成不同尺度下数据的损失CD
    14. # fake and real_center: (b,1,512,3). 生成的假的被裁剪下来的点云、真的被裁剪下来的点云
    15. # fake_center1 and real_center_key1: (b,64,3)
    16. # fake_center2 and real_center_key2: (b,128,3)
    17. errG_l2 = criterion_PointLoss(torch.squeeze(fake, 1), torch.squeeze(real_center, 1)) \
    18.           + alpha1 * criterion_PointLoss(fake_center1, real_center_key1) \
    19.           + alpha2 * criterion_PointLoss(fake_center2, real_center_key2)
    20.  
    21. errG = (1 - opt.wtl2) * errG_D + opt.wtl2 * errG_l2  # 0.05*errG_D + 0.95*errG_l2
    22. errG.backward()
    23. optimizerG.step()

    3.4 判别器模型

     对应到论文中的框架图：

    其中CMLP等于上图的conv2d+maxpool+conc组合操作。

    （1） 输入生成的假的被裁剪下来的点云，四次卷积，缩小通道数，获得多尺度特征；

    （2）分别对最后三个多尺度卷积结果进行最大池化，4维度变2维度特征；

    （3）拼接多个尺度特征，再接4个全链接层。

    1. class _netlocalD(nn.Module):
    2.     def __init__(self, crop_point_num):
    3.         super(_netlocalD, self).__init__()
    4.         self.crop_point_num = crop_point_num
    5.         self.conv1 = torch.nn.Conv2d(in_channels=1, out_channels=64, kernel_size=(1, 3))
    6.         self.conv2 = torch.nn.Conv2d(64, 64, 1)
    7.         self.conv3 = torch.nn.Conv2d(64, 128, 1)
    8.         self.conv4 = torch.nn.Conv2d(128, 256, 1)
    9.  
    10.         self.maxpool = torch.nn.MaxPool2d(kernel_size=(self.crop_point_num, 1), stride=1)
    11.  
    12.         self.bn1 = nn.BatchNorm2d(64)
    13.         self.bn2 = nn.BatchNorm2d(64)
    14.         self.bn3 = nn.BatchNorm2d(128)
    15.         self.bn4 = nn.BatchNorm2d(256)
    16.  
    17.         self.fc1 = nn.Linear(448, 256)
    18.         self.fc2 = nn.Linear(256, 128)
    19.         self.fc3 = nn.Linear(128, 16)
    20.         self.fc4 = nn.Linear(16, 1)
    21.  
    22.         self.bn_1 = nn.BatchNorm1d(256)
    23.         self.bn_2 = nn.BatchNorm1d(128)
    24.         self.bn_3 = nn.BatchNorm1d(16)
    25.  
    26.     def forward(self, x):  # size: (2,1,512,3)
    27.         x = F.relu(self.bn1(self.conv1(x)))  # (b,1,512,3) -> (2,64,512,1). conv2d+bn2d+relu
    28.         x_64 = F.relu(self.bn2(self.conv2(x)))  # (b,64,512,1) -> (b,64,512,1)
    29.         x_128 = F.relu(self.bn3(self.conv3(x_64)))  # (b,64,512,1) -> (b,128,512,1)
    30.         x_256 = F.relu(self.bn4(self.conv4(x_128)))  # (b,128,512,1) -> (b,256,512,1)
    31.  
    32.         x_64 = torch.squeeze(self.maxpool(x_64))  # (b,64,512,1) -> (b,64,1,1)->(b,64)
    33.         x_128 = torch.squeeze(self.maxpool(x_128))  # (b,128,512,1) -> (b,128,1,1)->(b,128)
    34.         x_256 = torch.squeeze(self.maxpool(x_256))  # (b,256,512,1) -> (b,256,1,1)->(b,256)
    35.  
    36.         Layers = [x_256, x_128, x_64]  # (b,64), (b,128), (b,256)
    37.         x = torch.cat(Layers, 1)  # (b,448)
    38.         x = F.relu(self.bn_1(self.fc1(x)))  # (b,448) -> (b,256)
    39.         x = F.relu(self.bn_2(self.fc2(x)))  # (b,256) -> (b,128)
    40.         x = F.relu(self.bn_3(self.fc3(x)))  # (b,128) -> (b,16)
    41.         x = self.fc4(x)  # (b,1). real or fake
    42.         return x

    3.5 生成器模型

    3.5.1 CMLP

     框架图中的CMLP代码如下，输入size: (b,num_points,3)，输出size: (b,1024+512+256+128, 1).

    1. class Convlayer(nn.Module):
    2.     def __init__(self, point_scales):
    3.         """
    4.         CMLP: conv+max_pool+concat, 其中最大池化的核大小是动态的，使得最后输出的特征向量是固定大小
    5.         Parameters
    6.         ----------
    7.         point_scales: int. 2048/1024/512. 用于最大池化核算子大小，相当与自适应最大池化，把特征图池化到1x1大小
    8.         """
    9.         super(Convlayer, self).__init__()
    10.         self.point_scales = point_scales
    11.         self.conv1 = torch.nn.Conv2d(1, 64, (1, 3))
    12.         self.conv2 = torch.nn.Conv2d(64, 64, 1)
    13.         self.conv3 = torch.nn.Conv2d(64, 128, 1)
    14.         self.conv4 = torch.nn.Conv2d(128, 256, 1)
    15.         self.conv5 = torch.nn.Conv2d(256, 512, 1)
    16.         self.conv6 = torch.nn.Conv2d(512, 1024, 1)
    17.         self.maxpool = torch.nn.MaxPool2d((self.point_scales, 1), 1)
    18.         self.bn1 = nn.BatchNorm2d(64)
    19.         self.bn2 = nn.BatchNorm2d(64)
    20.         self.bn3 = nn.BatchNorm2d(128)
    21.         self.bn4 = nn.BatchNorm2d(256)
    22.         self.bn5 = nn.BatchNorm2d(512)
    23.         self.bn6 = nn.BatchNorm2d(1024)
    24.  
    25.     def forward(self, x):  # (b,num_point,3)
    26.         x = torch.unsqueeze(x, 1)  # (b,num_point,3) -> (b,1,num_point,3)
    27.         x = F.relu(self.bn1(self.conv1(x)))
    28.         x = F.relu(self.bn2(self.conv2(x)))
    29.         # 获取4个尺度的4维度特征
    30.         x_128 = F.relu(self.bn3(self.conv3(x)))
    31.         x_256 = F.relu(self.bn4(self.conv4(x_128)))
    32.         x_512 = F.relu(self.bn5(self.conv5(x_256)))
    33.         x_1024 = F.relu(self.bn6(self.conv6(x_512)))
    34.         # 4维度变2维度特征
    35.         x_128 = torch.squeeze(self.maxpool(x_128), 2)  # (b,c,num_point,1) -> (b,c,1)
    36.         x_256 = torch.squeeze(self.maxpool(x_256), 2)
    37.         x_512 = torch.squeeze(self.maxpool(x_512), 2)
    38.         x_1024 = torch.squeeze(self.maxpool(x_1024), 2)
    39.         # 拼接多尺度特征
    40.         L = [x_1024, x_512, x_256, x_128]  # (b,1024,1), (b,512,1),(b,256,1), (b,128,1)
    41.         x = torch.cat(L, 1)  # (b,1024+512+256+128, 1)
    42.         return x

    3.5.2 Final Feature Vector V

    如下是框架中的特征向量Final feature vector V求取代码.

    输入size: list. (b,2048,3)/(b,1024,3)/(b,512,3)，输出size: (b,1920).

    1. class Latentfeature(nn.Module):
    2.     def __init__(self, num_scales, each_scales_size, point_scales_list):
    3.         """
    5.         Parameters
    6.         ----------
    7.         num_scales: int. 3. number of scales.
    8.         each_scales_size: int. 1. each scales size. 即每个尺度的shape
    9.         point_scales_list: list. [2048, 1024, 512]. number of points in each scales.
    10.         """
    11.         super(Latentfeature, self).__init__()
    12.         self.num_scales = num_scales
    13.         self.each_scales_size = each_scales_size
    14.         self.point_scales_list = point_scales_list
    15.         self.Convlayers1 = nn.ModuleList(  # CMLP
    16.             [Convlayer(point_scales=self.point_scales_list[0]) for i in range(self.each_scales_size)])
    17.         self.Convlayers2 = nn.ModuleList(
    18.             [Convlayer(point_scales=self.point_scales_list[1]) for i in range(self.each_scales_size)])
    19.         self.Convlayers3 = nn.ModuleList(
    20.             [Convlayer(point_scales=self.point_scales_list[2]) for i in range(self.each_scales_size)])
    21.         self.conv1 = torch.nn.Conv1d(3, 1, 1)
    22.         self.bn1 = nn.BatchNorm1d(1)
    23.  
    24.     def forward(self, x):
    25.         """
    27.         Parameters
    28.         ----------
    29.         x: list. (b,2048,3)/(b,1024,3)/(b,512,3)
    31.         Returns. (b,1920)
    32.         -------
    34.         """
    35.         outs = []
    36.         # 1, CMLP. input (b,point_num,3), output latent vector.
    37.         for i in range(self.each_scales_size):
    38.             outs.append(self.Convlayers1[i](x[0]))  # CMLP: (2,2048,3) -> (b,1024+512+256+128,1)
    39.         for j in range(self.each_scales_size):
    40.             outs.append(self.Convlayers2[j](x[1]))  # CMLP: (2,1024,3) -> (b,1024+512+256+128,1)
    41.         for k in range(self.each_scales_size):
    42.             outs.append(self.Convlayers3[k](x[2]))  # CMLP: (2,512,3) ->  (b,1024+512+256+128,1)
    43.         # 2, CONCAT
    44.         latentfeature = torch.cat(outs, 2)  # (b,1920,3). final latent map M
    45.         # 3, MLP
    46.         latentfeature = latentfeature.transpose(1, 2)  # (b,1920,3) -> (b,3,1920)
    47.         latentfeature = F.relu(self.bn1(self.conv1(latentfeature)))  # (b,3,1920) -> (b,1,1920)
    48.         latentfeature = torch.squeeze(latentfeature, 1)  # (b,1,1920) -> (b,1920)
    49.         return latentfeature

    3.5.3 生成器主代码 

    1. class _netG(nn.Module):
    2.     def __init__(self, num_scales, each_scales_size, point_scales_list, crop_point_num):
    3.         """
    5.         Parameters
    6.         ----------
    7.         num_scales: int. 3. number of scales.
    8.         each_scales_size: int. 1. each scales size. 即每个尺度的shape
    9.         point_scales_list: list. [2048, 1024, 512]. number of points in each scale.
    10.         crop_point_num: int. 512. 裁剪多少个点下来
    11.         """
    12.         super(_netG, self).__init__()
    13.         self.crop_point_num = crop_point_num
    14.         self.latentfeature = Latentfeature(num_scales, each_scales_size, point_scales_list)
    15.         self.fc1 = nn.Linear(1920, 1024)
    16.         self.fc2 = nn.Linear(1024, 512)
    17.         self.fc3 = nn.Linear(512, 256)
    18.  
    19.         self.fc1_1 = nn.Linear(1024, 128 * 512)
    20.         self.fc2_1 = nn.Linear(512, 64 * 128)  # nn.Linear(512,64*256) !
    21.         self.fc3_1 = nn.Linear(256, 64 * 3)
    22.  
    23.         self.conv1_1 = torch.nn.Conv1d(512, 512, 1)  # torch.nn.Conv1d(256,256,1) !
    24.         self.conv1_2 = torch.nn.Conv1d(512, 256, 1)
    25.         self.conv1_3 = torch.nn.Conv1d(256, int((self.crop_point_num * 3) / 128), 1)
    26.         self.conv2_1 = torch.nn.Conv1d(128, 6, 1)  # torch.nn.Conv1d(256,12,1) !
    27.  
    28.     def forward(self, x):
    29.         """
    31.         Parameters
    32.         ----------
    33.         x: list. (b,2048,3)/(b,1024,3)/(b,512,3)
    35.         Returns (b,64,3), (b,128,3), (b,512,3).
    36.         -------
    38.         """
    39.         # final feature vector V
    40.         x = self.latentfeature(x)  # list -> (b,1920)
    41.         # FPN
    42.         # fc1, fc2, fc3
    43.         x_1 = F.relu(self.fc1(x))  # (b,1920) -> (b,1024)
    44.         x_2 = F.relu(self.fc2(x_1))  # (b,1024) -> (b,512)
    45.         x_3 = F.relu(self.fc3(x_2))  # (b,512) -> (b,256)
    46.         # x_3: fc+reshape. 少了论文中的一个conv
    47.         pc1_feat = self.fc3_1(x_3)  # (b,256) -> (b,192)
    48.         pc1_xyz = pc1_feat.reshape(-1, 64, 3)  # (b,192) -> (b,64,3). 64x3 center1. 64个点
    49.         # x_2: fc+reshape+conv1d
    50.         pc2_feat = F.relu(self.fc2_1(x_2))  # (b,192) -> (b,8192)
    51.         pc2_feat = pc2_feat.reshape(-1, 128, 64)  # (b,8192) -> (b,128,64)
    52.         pc2_xyz = self.conv2_1(pc2_feat)  # (b,128,64) -> (b,6,64). 6x64 center2
    53.         # x_1: fc_reshape+conv1d+conv1d+conv1d
    54.         pc3_feat = F.relu(self.fc1_1(x_1))  # (b,1024) -> (b,65536)
    55.         pc3_feat = pc3_feat.reshape(-1, 512, 128)  # (b,65536) -> (b,512,128)
    56.         pc3_feat = F.relu(self.conv1_1(pc3_feat))  # (b,512,128) -> (b,512,128)
    57.         pc3_feat = F.relu(self.conv1_2(pc3_feat))  # (b,512,128) -> (b,256,128)
    58.         pc3_xyz = self.conv1_3(pc3_feat)  # (b,256,128) -> (b,12,128). 12x128 fine
    59.  
    60.         # plus: scale 1 + scale 2
    61.         pc1_xyz_expand = torch.unsqueeze(pc1_xyz, 2)  # (b,64,3) -> (b,64,1,3)
    62.         pc2_xyz = pc2_xyz.transpose(1, 2)  # (b,6,64) -> (b,64,6)
    63.         pc2_xyz = pc2_xyz.reshape(-1, 64, 2, 3)  # (b,64,6) -> (b,64,2,3)
    64.         pc2_xyz = pc1_xyz_expand + pc2_xyz  # (b,64,1,3) + (b,64,2,3) = (b,64,2,3)
    65.         pc2_xyz = pc2_xyz.reshape(-1, 128, 3)  # (b,64,2,3) -> (b,128,3)
    66.         # plus: scale 2 + scale 3
    67.         pc2_xyz_expand = torch.unsqueeze(pc2_xyz, 2)  # (b,128,3) -> (b,128,1,3)
    68.         pc3_xyz = pc3_xyz.transpose(1, 2)  # (b,12,128) -> (b,12,128)
    69.         pc3_xyz = pc3_xyz.reshape(-1, 128, int(self.crop_point_num / 128), 3)  # (b,12,128) -> (b,128,4,3)
    70.         pc3_xyz = pc2_xyz_expand + pc3_xyz  # (b,128,1,3) + (b,128,4,3) = (b,128,4,3)
    71.         pc3_xyz = pc3_xyz.reshape(-1, self.crop_point_num, 3)  # (b,128,4,3) -> (b,512,3)
    72.  
    73.         return pc1_xyz, pc2_xyz, pc3_xyz  # (b,64,3), (b,128,3), (b,512,3). center1, center2, fine

    3.6 测试效果

     

    测试代码

    1. # 1. init model
    2. device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    3. point_netG = _netG(opt.num_scales, opt.each_scales_size, opt.point_scales_list, opt.crop_point_num)
    4. point_netG = torch.nn.DataParallel(point_netG)
    5. point_netG.to(device)
    6. point_netG.load_state_dict(torch.load(opt.netG, map_location=lambda storage, location: storage)['state_dict'])
    7. point_netG.eval()
    8.  
    9. # 2. load incomplete point cloud
    10. input_cropped1 = np.loadtxt(opt.infile, delimiter=',')  # (1536,3). csv文件
    11. input_cropped1 = torch.FloatTensor(input_cropped1)  # (1536,3)
    12. input_cropped1 = torch.unsqueeze(input_cropped1, 0)  # (1,1536,3)
    13.  
    14. Zeros = torch.zeros(1, 512, 3)  # (1,512,3)
    15. input_cropped1 = torch.cat((input_cropped1, Zeros), 1)  # (1,1536+512,3) = (1,2048,3)
    16.  
    17. # 2. preprocess
    18. # 获得多尺度输入： [input_cropped1, input_cropped2, input_cropped3]. (1,2048,3)/(1,1024,3)/(1,512,3)
    19. input_cropped2_idx = utils.farthest_point_sample(input_cropped1, opt.point_scales_list[1], RAN=True)
    20. input_cropped2 = utils.index_points(input_cropped1, input_cropped2_idx)  # (1,1024,3)
    21. input_cropped3_idx = utils.farthest_point_sample(input_cropped1, opt.point_scales_list[2], RAN=False)
    22. input_cropped3 = utils.index_points(input_cropped1, input_cropped3_idx)  # (1,512,3)
    23. # input_cropped4_idx = utils.farthest_point_sample(input_cropped1, 256, RAN=True)
    24. # input_cropped4 = utils.index_points(input_cropped1, input_cropped4_idx)  # (1,256,3). 没啥用
    25.  
    26. # to cuda
    27. input_cropped2 = input_cropped2.to(device)  # (1,1024,3)
    28. input_cropped3 = input_cropped3.to(device)  # (1,512,3)
    29. input_cropped = [input_cropped1, input_cropped2, input_cropped3]
    30. # 3. infer. fake.size: (1,512,3)
    31. fake_center1, fake_center2, fake = point_netG(input_cropped)
    32. # fake = fake.cuda()  # 返回的本来就在cuda设备上
    33. # fake_center1 = fake_center1.cuda()
    34. # fake_center2 = fake_center2.cuda()
    35.  
    36. # 4. post-process
    37. # input_cropped2 = input_cropped2.cpu()
    38. # input_cropped3 = input_cropped3.cpu()
    39. # input_cropped4 = input_cropped4.cpu()
    40.  
    41. # np_crop2 = input_cropped2[0].detach().numpy()
    42. # np_crop3 = input_cropped3[0].detach().numpy()
    43. # np_crop4 = input_cropped4[0].detach().numpy()
    44.  
    45. # # 真实被裁剪下来的点云，并生成多尺度真实点云
    46. # real = np.loadtxt(opt.infile_real, delimiter=',')
    47. # real = torch.FloatTensor(real)
    48. # real = torch.unsqueeze(real, 0)
    49. # real2_idx = utils.farthest_point_sample(real, 64, RAN=False)
    50. # real2 = utils.index_points(real, real2_idx)
    51. # real3_idx = utils.farthest_point_sample(real, 128, RAN=True)
    52. # real3 = utils.index_points(real, real3_idx)
    53. #
    54. # real2 = real2.cpu()
    55. # real3 = real3.cpu()
    56. #
    57. # np_real2 = real2[0].detach().numpy()
    58. # np_real3 = real3[0].detach().numpy()
    59.  
    60. fake = fake.cpu()
    61. # fake_center1 = fake_center1.cpu()
    62. # fake_center2 = fake_center2.cpu()
    63. np_fake = fake[0].detach().numpy()  # (1,512,3) -> (512,3)
    64. # np_fake1 = fake_center1[0].detach().numpy()
    65. # np_fake2 = fake_center2[0].detach().numpy()
    66. input_cropped1 = input_cropped1.cpu()
    67. np_crop = input_cropped1[0].numpy()  # (1,2048,3) -> (2048,3)
    68.  
    69. np.savetxt('test_one/crop_ours' + '.csv', np_crop, fmt="%f,%f,%f")
    70. np.savetxt('test_one/fake_ours' + '.csv', np_fake, fmt="%f,%f,%f")
    71. np.savetxt('test_one/crop_ours_txt' + '.txt', np_crop, fmt="%f,%f,%f")
    72. np.savetxt('test_one/fake_ours_txt' + '.txt', np_fake, fmt="%f,%f,%f")

            

  • 相关阅读:
    分布式消息队列RocketMQ概念详解
    Vulkan Cascade Shadow Map的故事
    创新洞见｜2023年B2B业务为何必须采用PLG增长策略
    单隐层神经网络在Matlab上实现及其简单应用
    【中间件篇-Redis缓存数据库08】Redis设计、实现、redisobject对象设计、多线程、缓存淘汰算法
    图09 --- 最小费用最大流问题
    PMP考试是如何提高项目管理能力的？
    我的十年编程路 尾声
    leecode959. 由斜杠划分区域
    络达开发----如何开启AGC功能
  • 原文地址：https://blog.csdn.net/jizhidexiaoming/article/details/128161796