• 深度学习基础：超分辨率网络整理之EDVR网络

    目录

    EDVR网络

    1.简介

    2.超分常见模块

    2.1.特征提取-feature extraction

    2.2.帧对齐-Alignment.

    2.3.融合-Fusion

    2.4.重建-reconstruction

    3.EDVR模块

    3.1整体架构

    3.2PCD-帧对齐

    3.3 TSA-时空融合

    4.结论

    5.EDVR模型代码

    EDVR网络

    1.简介

    超分辨率（super-resolution）、去模糊（deblurring）等视频恢复任务越来越受到计算机视觉界的关注。在NTIRE19挑战赛中发布了一个名为REDS的具有挑战性的基准测试。这个新的基准测试从两个方面挑战了现有的方法:

    (1)如何在给定大运动的情况下对齐多个帧

    (2)如何有效地融合不同运动和模糊的不同帧。

    在这项工作中，我们提出了一种新的视频恢复框架，增强可变形卷积，称为EDVR，以解决这些挑战。

    首先，为了处理大型运动，我们设计了一个金字塔、级联和可变形(PCD)对齐模块，在该模块中，帧对齐在特征级使用可变形卷积以从粗到细（coarse-to-fine）的方式完成。

    其次，我们提出了一个时空注意(TSA)融合模块，将注意力同时应用于时间和空间上，以强调重要特征，为后续恢复提供依据。

    多亏了这些模块，我们的EDVR在NTIRE19挑战赛中视频恢复和增强挑战的所有四个轨道中都获得了冠军，并以较大的优势超过了第二名。EDVR在视频超分辨率和去模糊方面的性能也优于最新发布的方法。

    2.超分常见模块

    2.1.特征提取-feature extraction

    2.2.帧对齐-Alignment.

    大多数现有的对准方法都是通过显式估计参考帧和相邻帧之间的光流场来实现的。根据估计的运动场对相邻帧进行变形处理。

    另一个研究分支是通过动态滤波或可变形卷积实现隐式运动补偿。reds数据集对现有的对齐算法提出了巨大的挑战。特别是，对于基于流量的方法来说，精确的运动估计和精确的运动补偿是具有挑战性和耗时的。在大运动的情况下，很难在单一分辨率尺度内显式或隐式地进行运动补偿。

    2.3.融合-Fusion

    融合来自对齐帧的特征是视频恢复任务的另一个关键步骤。现有的大多数方法要么使用卷积对所有帧[进行早期融合，要么采用循环网络逐步融合多个帧。Liu等人提出了一种时间自适应网络，可以动态融合不同时间尺度。这些现有的方法都没有考虑到每个帧上潜在的视觉信息量——不同的帧和位置对重建的信息量并不相同或有益，因为一些帧或区域会受到不完全对齐和模糊的影响。（这里是不是读出了注意力机制的味道？）

    2.4.重建-reconstruction

    3.EDVR模块

    3.1整体架构

     

    上图是EDVR框架。它是一个统一的框架，适用于各种视频恢复任务，如超分辨率和去模糊。对高空间分辨率的输入先进行下采样，以减少计算成本。给定模糊输入，在PCD对齐模块之前插入预模糊模块以提高对齐精度。我们使用三个输入帧作为示例。

    以视频SR为例，EDVR以2N+1个低分辨率帧作为输入，生成高分辨率输出。每个相邻帧由PCD对齐模块在特征级别上与参考帧对齐。TSA融合模块对不同帧的图像信息进行融合。然后，融合的特征通过一个重建模块，重建模块是EDVR中残留块的级联，可以很容易地被单幅图像SR中的任何其他高级模块替换。上采样操作在网络的末端执行，以增加空间大小。最后，将预测图像残差加入到直接上采样图像中，得到高分辨率帧Ot。对于其他具有高空间分辨率输入的任务，如视频去模糊，首先使用跨步卷积层对输入帧进行下采样。然后大部分的计算都是在低分辨率空间进行的，这大大节省了计算成本。在最后的上采样层将调整特征的大小回到原始的输入分辨率。在对准模块之前使用预模糊(PreDeblur)模块对模糊输入进行预处理，提高对准精度。虽然单一的EDVR模型可以达到最先进的性能，但我们采用两阶段策略来进一步提高NTIRE19比赛的性能。具体来说，我们用较浅的深度级联相同的EDVR网络，以细化第一阶段的输出帧。级联网络可以进一步消除之前模型无法处理的严重运动模糊。=

    3.2PCD-帧对齐

    3.3 TSA-时空融合

    提出的TSA是一个融合模块，有助于跨多个对齐的特征聚合信息。为了更好地考虑每一帧的视觉信息量，我们通过计算参考帧和每一相邻帧特征之间的元素相关性引入时间注意力。然后，相关系数在每个位置上对每个相邻特征进行加权，表明它对重构参考图像的信息量有多大。然后将所有帧的加权特征进行卷积并融合在一起。在与时间注意融合后，我们进一步应用空间注意为每个通道中的每个位置分配权重，以更有效地利用跨通道和空间信息。

    4.结论

    5.EDVR模型代码

    1. import torch
    2. from torch import nn as nn
    3. from torch.nn import functional as F
    4.  
    5. from basicsr.utils.registry import ARCH_REGISTRY
    6. from .arch_util import DCNv2Pack, ResidualBlockNoBN, make_layer
    7.  
    8. #对齐模块
    9. class PCDAlignment(nn.Module):
    10.     """Alignment module using Pyramid, Cascading and Deformable convolution
    11.     (PCD). It is used in EDVR.
    13.     Ref:
    14.         EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
    16.     Args:
    17.         num_feat (int): Channel number of middle features. Default: 64.
    18.         deformable_groups (int): Deformable groups. Defaults: 8.
    19.     """
    20.  
    21.     def __init__(self, num_feat=64, deformable_groups=8):
    22.         super(PCDAlignment, self).__init__()
    23.  
    24.         # Pyramid has three levels:
    25.         # L3: level 3, 1/4 spatial size
    26.         # L2: level 2, 1/2 spatial size
    27.         # L1: level 1, original spatial size
    28.         self.offset_conv1 = nn.ModuleDict()
    29.         self.offset_conv2 = nn.ModuleDict()
    30.         self.offset_conv3 = nn.ModuleDict()
    31.         self.dcn_pack = nn.ModuleDict()
    32.         self.feat_conv = nn.ModuleDict()
    33.  
    34.         # Pyramids
    35.         for i in range(3, 0, -1):
    36.             level = f'l{i}'
    37.             self.offset_conv1[level] = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
    38.             if i == 3:
    39.                 self.offset_conv2[level] = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    40.             else:
    41.                 self.offset_conv2[level] = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
    42.                 self.offset_conv3[level] = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    43.             self.dcn_pack[level] = DCNv2Pack(num_feat, num_feat, 3, padding=1, deformable_groups=deformable_groups)
    44.  
    45.             if i < 3:
    46.                 self.feat_conv[level] = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
    47.  
    48.         # Cascading dcn
    49.         self.cas_offset_conv1 = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
    50.         self.cas_offset_conv2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    51.         self.cas_dcnpack = DCNv2Pack(num_feat, num_feat, 3, padding=1, deformable_groups=deformable_groups)
    52.  
    53.         self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False)
    54.         self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
    55.  
    56.     def forward(self, nbr_feat_l, ref_feat_l):
    57.         """Align neighboring frame features to the reference frame features.
    59.         Args:
    60.             nbr_feat_l (list[Tensor]): Neighboring feature list. It
    61.                 contains three pyramid levels (L1, L2, L3),
    62.                 each with shape (b, c, h, w).
    63.             ref_feat_l (list[Tensor]): Reference feature list. It
    64.                 contains three pyramid levels (L1, L2, L3),
    65.                 each with shape (b, c, h, w).
    67.         Returns:
    68.             Tensor: Aligned features.
    69.         """
    70.         # Pyramids
    71.         upsampled_offset, upsampled_feat = None, None
    72.         for i in range(3, 0, -1):
    73.             level = f'l{i}'
    74.             offset = torch.cat([nbr_feat_l[i - 1], ref_feat_l[i - 1]], dim=1)
    75.             offset = self.lrelu(self.offset_conv1[level](offset))
    76.             if i == 3:
    77.                 offset = self.lrelu(self.offset_conv2[level](offset))
    78.             else:
    79.                 offset = self.lrelu(self.offset_conv2[level](torch.cat([offset, upsampled_offset], dim=1)))
    80.                 offset = self.lrelu(self.offset_conv3[level](offset))
    81.  
    82.             feat = self.dcn_pack[level](nbr_feat_l[i - 1], offset)
    83.             if i < 3:
    84.                 feat = self.feat_conv[level](torch.cat([feat, upsampled_feat], dim=1))
    85.             if i > 1:
    86.                 feat = self.lrelu(feat)
    87.  
    88.             if i > 1:  # upsample offset and features
    89.                 # x2: when we upsample the offset, we should also enlarge
    90.                 # the magnitude.
    91.                 upsampled_offset = self.upsample(offset) * 2
    92.                 upsampled_feat = self.upsample(feat)
    93.  
    94.         # Cascading
    95.         offset = torch.cat([feat, ref_feat_l[0]], dim=1)
    96.         offset = self.lrelu(self.cas_offset_conv2(self.lrelu(self.cas_offset_conv1(offset))))
    97.         feat = self.lrelu(self.cas_dcnpack(feat, offset))
    98.         return feat
    99. #融合模块
    100.  
    101. class TSAFusion(nn.Module):
    102.     """Temporal Spatial Attention (TSA) fusion module.
    104.     Temporal: Calculate the correlation between center frame and
    105.         neighboring frames;
    106.     Spatial: It has 3 pyramid levels, the attention is similar to SFT.
    107.         (SFT: Recovering realistic texture in image super-resolution by deep
    108.             spatial feature transform.)
    110.     Args:
    111.         num_feat (int): Channel number of middle features. Default: 64.
    112.         num_frame (int): Number of frames. Default: 5.
    113.         center_frame_idx (int): The index of center frame. Default: 2.
    114.     """
    115.  
    116.     def __init__(self, num_feat=64, num_frame=5, center_frame_idx=2):
    117.         super(TSAFusion, self).__init__()
    118.         self.center_frame_idx = center_frame_idx
    119.         # temporal attention (before fusion conv)
    120.         self.temporal_attn1 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    121.         self.temporal_attn2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    122.         self.feat_fusion = nn.Conv2d(num_frame * num_feat, num_feat, 1, 1)
    123.  
    124.         # spatial attention (after fusion conv)
    125.         self.max_pool = nn.MaxPool2d(3, stride=2, padding=1)
    126.         self.avg_pool = nn.AvgPool2d(3, stride=2, padding=1)
    127.         self.spatial_attn1 = nn.Conv2d(num_frame * num_feat, num_feat, 1)
    128.         self.spatial_attn2 = nn.Conv2d(num_feat * 2, num_feat, 1)
    129.         self.spatial_attn3 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    130.         self.spatial_attn4 = nn.Conv2d(num_feat, num_feat, 1)
    131.         self.spatial_attn5 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    132.         self.spatial_attn_l1 = nn.Conv2d(num_feat, num_feat, 1)
    133.         self.spatial_attn_l2 = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
    134.         self.spatial_attn_l3 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    135.         self.spatial_attn_add1 = nn.Conv2d(num_feat, num_feat, 1)
    136.         self.spatial_attn_add2 = nn.Conv2d(num_feat, num_feat, 1)
    137.  
    138.         self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
    139.         self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False)
    140.  
    141.     def forward(self, aligned_feat):
    142.         """
    143.         Args:
    144.             aligned_feat (Tensor): Aligned features with shape (b, t, c, h, w).
    146.         Returns:
    147.             Tensor: Features after TSA with the shape (b, c, h, w).
    148.         """
    149.         b, t, c, h, w = aligned_feat.size()
    150.         # temporal attention
    151.         embedding_ref = self.temporal_attn1(aligned_feat[:, self.center_frame_idx, :, :, :].clone())
    152.         embedding = self.temporal_attn2(aligned_feat.view(-1, c, h, w))
    153.         embedding = embedding.view(b, t, -1, h, w)  # (b, t, c, h, w)
    154.  
    155.         corr_l = []  # correlation list
    156.         for i in range(t):
    157.             emb_neighbor = embedding[:, i, :, :, :]
    158.             corr = torch.sum(emb_neighbor * embedding_ref, 1)  # (b, h, w)
    159.             corr_l.append(corr.unsqueeze(1))  # (b, 1, h, w)
    160.         corr_prob = torch.sigmoid(torch.cat(corr_l, dim=1))  # (b, t, h, w)
    161.         corr_prob = corr_prob.unsqueeze(2).expand(b, t, c, h, w)
    162.         corr_prob = corr_prob.contiguous().view(b, -1, h, w)  # (b, t*c, h, w)
    163.         aligned_feat = aligned_feat.view(b, -1, h, w) * corr_prob
    164.  
    165.         # fusion
    166.         feat = self.lrelu(self.feat_fusion(aligned_feat))
    167.  
    168.         # spatial attention
    169.         attn = self.lrelu(self.spatial_attn1(aligned_feat))
    170.         attn_max = self.max_pool(attn)
    171.         attn_avg = self.avg_pool(attn)
    172.         attn = self.lrelu(self.spatial_attn2(torch.cat([attn_max, attn_avg], dim=1)))
    173.         # pyramid levels
    174.         attn_level = self.lrelu(self.spatial_attn_l1(attn))
    175.         attn_max = self.max_pool(attn_level)
    176.         attn_avg = self.avg_pool(attn_level)
    177.         attn_level = self.lrelu(self.spatial_attn_l2(torch.cat([attn_max, attn_avg], dim=1)))
    178.         attn_level = self.lrelu(self.spatial_attn_l3(attn_level))
    179.         attn_level = self.upsample(attn_level)
    180.  
    181.         attn = self.lrelu(self.spatial_attn3(attn)) + attn_level
    182.         attn = self.lrelu(self.spatial_attn4(attn))
    183.         attn = self.upsample(attn)
    184.         attn = self.spatial_attn5(attn)
    185.         attn_add = self.spatial_attn_add2(self.lrelu(self.spatial_attn_add1(attn)))
    186.         attn = torch.sigmoid(attn)
    187.  
    188.         # after initialization, * 2 makes (attn * 2) to be close to 1.
    189.         feat = feat * attn * 2 + attn_add
    190.         return feat
    191.  
    192.  
    193. class PredeblurModule(nn.Module):
    194.     """Pre-dublur module.
    196.     Args:
    197.         num_in_ch (int): Channel number of input image. Default: 3.
    198.         num_feat (int): Channel number of intermediate features. Default: 64.
    199.         hr_in (bool): Whether the input has high resolution. Default: False.
    200.     """
    201.  
    202.     def __init__(self, num_in_ch=3, num_feat=64, hr_in=False):
    203.         super(PredeblurModule, self).__init__()
    204.         self.hr_in = hr_in
    205.  
    206.         self.conv_first = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1)
    207.         if self.hr_in:
    208.             # downsample x4 by stride conv
    209.             self.stride_conv_hr1 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
    210.             self.stride_conv_hr2 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
    211.  
    212.         # generate feature pyramid
    213.         self.stride_conv_l2 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
    214.         self.stride_conv_l3 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
    215.  
    216.         self.resblock_l3 = ResidualBlockNoBN(num_feat=num_feat)
    217.         self.resblock_l2_1 = ResidualBlockNoBN(num_feat=num_feat)
    218.         self.resblock_l2_2 = ResidualBlockNoBN(num_feat=num_feat)
    219.         self.resblock_l1 = nn.ModuleList([ResidualBlockNoBN(num_feat=num_feat) for i in range(5)])
    220.  
    221.         self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False)
    222.         self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
    223.  
    224.     def forward(self, x):
    225.         feat_l1 = self.lrelu(self.conv_first(x))
    226.         if self.hr_in:
    227.             feat_l1 = self.lrelu(self.stride_conv_hr1(feat_l1))
    228.             feat_l1 = self.lrelu(self.stride_conv_hr2(feat_l1))
    229.  
    230.         # generate feature pyramid
    231.         feat_l2 = self.lrelu(self.stride_conv_l2(feat_l1))
    232.         feat_l3 = self.lrelu(self.stride_conv_l3(feat_l2))
    233.  
    234.         feat_l3 = self.upsample(self.resblock_l3(feat_l3))
    235.         feat_l2 = self.resblock_l2_1(feat_l2) + feat_l3
    236.         feat_l2 = self.upsample(self.resblock_l2_2(feat_l2))
    237.  
    238.         for i in range(2):
    239.             feat_l1 = self.resblock_l1[i](feat_l1)
    240.         feat_l1 = feat_l1 + feat_l2
    241.         for i in range(2, 5):
    242.             feat_l1 = self.resblock_l1[i](feat_l1)
    243.         return feat_l1
    244.  
    245.  
    246. @ARCH_REGISTRY.register()
    247. class EDVR(nn.Module):
    248.     """EDVR network structure for video super-resolution.
    250.     Now only support X4 upsampling factor.
    251.     Paper:
    252.         EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
    254.     Args:
    255.         num_in_ch (int): Channel number of input image. Default: 3.
    256.         num_out_ch (int): Channel number of output image. Default: 3.
    257.         num_feat (int): Channel number of intermediate features. Default: 64.
    258.         num_frame (int): Number of input frames. Default: 5.
    259.         deformable_groups (int): Deformable groups. Defaults: 8.
    260.         num_extract_block (int): Number of blocks for feature extraction.
    261.             Default: 5.
    262.         num_reconstruct_block (int): Number of blocks for reconstruction.
    263.             Default: 10.
    264.         center_frame_idx (int): The index of center frame. Frame counting from
    265.             0. Default: Middle of input frames.
    266.         hr_in (bool): Whether the input has high resolution. Default: False.
    267.         with_predeblur (bool): Whether has predeblur module.
    268.             Default: False.
    269.         with_tsa (bool): Whether has TSA module. Default: True.
    270.     """
    271.  
    272.     def __init__(self,
    273.                  num_in_ch=3,
    274.                  num_out_ch=3,
    275.                  num_feat=64,
    276.                  num_frame=5,
    277.                  deformable_groups=8,
    278.                  num_extract_block=5,
    279.                  num_reconstruct_block=10,
    280.                  center_frame_idx=None,
    281.                  hr_in=False,
    282.                  with_predeblur=False,
    283.                  with_tsa=True):
    284.         super(EDVR, self).__init__()
    285.         if center_frame_idx is None:
    286.             self.center_frame_idx = num_frame // 2
    287.         else:
    288.             self.center_frame_idx = center_frame_idx
    289.         self.hr_in = hr_in
    290.         self.with_predeblur = with_predeblur
    291.         self.with_tsa = with_tsa
    292.  
    293.         # extract features for each frame
    294.         if self.with_predeblur:
    295.             self.predeblur = PredeblurModule(num_feat=num_feat, hr_in=self.hr_in)
    296.             self.conv_1x1 = nn.Conv2d(num_feat, num_feat, 1, 1)
    297.         else:
    298.             self.conv_first = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1)
    299.  
    300.         # extract pyramid features
    301.         self.feature_extraction = make_layer(ResidualBlockNoBN, num_extract_block, num_feat=num_feat)
    302.         self.conv_l2_1 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
    303.         self.conv_l2_2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    304.         self.conv_l3_1 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
    305.         self.conv_l3_2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
    306.  
    307.         # pcd and tsa module
    308.         self.pcd_align = PCDAlignment(num_feat=num_feat, deformable_groups=deformable_groups)
    309.         if self.with_tsa:
    310.             self.fusion = TSAFusion(num_feat=num_feat, num_frame=num_frame, center_frame_idx=self.center_frame_idx)
    311.         else:
    312.             self.fusion = nn.Conv2d(num_frame * num_feat, num_feat, 1, 1)
    313.  
    314.         # reconstruction
    315.         self.reconstruction = make_layer(ResidualBlockNoBN, num_reconstruct_block, num_feat=num_feat)
    316.         # upsample
    317.         self.upconv1 = nn.Conv2d(num_feat, num_feat * 4, 3, 1, 1)
    318.         self.upconv2 = nn.Conv2d(num_feat, 64 * 4, 3, 1, 1)
    319.         self.pixel_shuffle = nn.PixelShuffle(2)
    320.         self.conv_hr = nn.Conv2d(64, 64, 3, 1, 1)
    321.         self.conv_last = nn.Conv2d(64, 3, 3, 1, 1)
    322.  
    323.         # activation function
    324.         self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
    325.  
    326.     def forward(self, x):
    327.         b, t, c, h, w = x.size()
    328.         if self.hr_in:
    329.             assert h % 16 == 0 and w % 16 == 0, ('The height and width must be multiple of 16.')
    330.         else:
    331.             assert h % 4 == 0 and w % 4 == 0, ('The height and width must be multiple of 4.')
    332.  
    333.         x_center = x[:, self.center_frame_idx, :, :, :].contiguous()
    334.  
    335.         # extract features for each frame
    336.         # L1
    337.         if self.with_predeblur:
    338.             feat_l1 = self.conv_1x1(self.predeblur(x.view(-1, c, h, w)))
    339.             if self.hr_in:
    340.                 h, w = h // 4, w // 4
    341.         else:
    342.             feat_l1 = self.lrelu(self.conv_first(x.view(-1, c, h, w)))
    343.  
    344.         feat_l1 = self.feature_extraction(feat_l1)
    345.         # L2
    346.         feat_l2 = self.lrelu(self.conv_l2_1(feat_l1))
    347.         feat_l2 = self.lrelu(self.conv_l2_2(feat_l2))
    348.         # L3
    349.         feat_l3 = self.lrelu(self.conv_l3_1(feat_l2))
    350.         feat_l3 = self.lrelu(self.conv_l3_2(feat_l3))
    351.  
    352.         feat_l1 = feat_l1.view(b, t, -1, h, w)
    353.         feat_l2 = feat_l2.view(b, t, -1, h // 2, w // 2)
    354.         feat_l3 = feat_l3.view(b, t, -1, h // 4, w // 4)
    355.  
    356.         # PCD alignment
    357.         ref_feat_l = [  # reference feature list
    358.             feat_l1[:, self.center_frame_idx, :, :, :].clone(), feat_l2[:, self.center_frame_idx, :, :, :].clone(),
    359.             feat_l3[:, self.center_frame_idx, :, :, :].clone()
    360.         ]
    361.         aligned_feat = []
    362.         for i in range(t):
    363.             nbr_feat_l = [  # neighboring feature list
    364.                 feat_l1[:, i, :, :, :].clone(), feat_l2[:, i, :, :, :].clone(), feat_l3[:, i, :, :, :].clone()
    365.             ]
    366.             aligned_feat.append(self.pcd_align(nbr_feat_l, ref_feat_l))
    367.         aligned_feat = torch.stack(aligned_feat, dim=1)  # (b, t, c, h, w)
    368.  
    369.         if not self.with_tsa:
    370.             aligned_feat = aligned_feat.view(b, -1, h, w)
    371.         feat = self.fusion(aligned_feat)
    372.  
    373.         out = self.reconstruction(feat)
    374.         out = self.lrelu(self.pixel_shuffle(self.upconv1(out)))
    375.         out = self.lrelu(self.pixel_shuffle(self.upconv2(out)))
    376.         out = self.lrelu(self.conv_hr(out))
    377.         out = self.conv_last(out)
    378.         if self.hr_in:
    379.             base = x_center
    380.         else:
    381.             base = F.interpolate(x_center, scale_factor=4, mode='bilinear', align_corners=False)
    382.         out += base
    383.         return out

  • 相关阅读:
    iOS 关于单例常见使用方法
    企业如何才能保证自身可持续发展
    在SSL中进行交叉熵学习的步骤
    PyTorch 入门
    springboot社区人员管理系统的设计与实现毕业设计源码260839
    【云原生之Docker实战】使用docker部署PicUploader图床工具
    项目质量管理全部精华看这篇就够了
    Redis 的持久化
    还在到处找图片和封面？是时候了解下这些网站了
    运行 `npm install` 时的常见问题与解决方案
  • 原文地址：https://blog.csdn.net/weixin_43507744/article/details/126874809