• fasterrcnn tensorrt部署实践

    目录

    环境安装：

    pycuda安装：

    开源项目信息

    fasterRCNN的训练

    faster rcnn onnx实践

    第3步测试结果：

    第4步测试结果：

    单张图片测试代码：

    多张图片预测代码：

    第5步做了修改：

    第7步测试：

    第8步测试

    onnx转trt操作

    c++ 转换trt：

    Onnx转trt代码及操作：

    c++ tensorrt推理部分

    c++onnx转tensorrt

    打印模型输入输出参数：

    环境安装：

    pip install onnxsim

    pytorch安装：

    cuda版本是11.0，没有cuda11.0对应的torchvision，所以安装了cpu版：

    pip install torch==1.10.0+cpu torchvision==0.11.0+cpu torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html

    pip install torch==1.10.0+cu113 torchvision==0.11.0+cu113 -f https://download.pytorch.org/whl/torch_stable.html

    tensorrt安装：

    onnx转tensorrt 实战干货总结_AI视觉网奇的博客-CSDN博客_onnx转tensorrt

    pycuda安装：

    win10安装pycuda2022_AI视觉网奇的博客-CSDN博客

    开源项目信息

    根据开源项目进行部署测试：

    GitHub - thb1314/tensorrt-onnx-fasterrcnn-fpn-roialign

    本部分基于repo

    GitHub - shouxieai/tensorRT_Pro: C++ library based on tensorrt integration

    安装部分请看该项目的readme部分，在本项目文件下tensorrt_code下

    如果可以给该项目点个star的话，麻烦顺手给俺也点一个吧，谢谢。

    fasterRCNN的训练

    demo: 使用pytorch训练自己的Faster-RCNN目标检测模型 - 野生鹅鹅 - 博客园

    其他git code还请自行查找

    导出onnx的时候，需要加载加载自己训练的权重。

    faster rcnn onnx实践

    重要的环节是第3步和 第5步，

    作者提供了8个步骤：
     x01export_FasterRCNN_onnx.py
     x02test_FasterRCNN_onnx.py
     x03extract_RPN.py
     x04testRPNonnx.py
     x05extract_ROIHeader.py
     x06reduceRpnOnnx.py
     x07reduce_header_onnx.py
     x08test_header_onnx.py

    第一个步骤转onnx警告和解决方法：

    that if the size of dimension 1 of the input is not 1, the ONNX model will return an error_AI视觉网奇的博客-CSDN博客

    第3步测试结果：

    输出特征名：

    ['rpn_boxes', 'feature_0', 'feature_1', 'feature_2', 'feature_3', 'feature_pool']

    输出特征维度：

    1. list 1000*5

    2. list，长度4

    0： 1 256 200 264

    1： 1 256 100 132

    2： 1 256 50 66

    3： 1 256 25 33

    pool： 256 13 17

    3是图片

    4是维度

    原本代码batch_size为1时可以正确预测，但是batch_size为2时，预测结果都是第一张图片，

    稍微修改了代码：

    1. import torch
    2. import os
    3. import sys
    4. sys.path.insert(0, os.path.abspath('..'))
    5. from model import fasterrpn_resnet50_fpn
    6. import glob
    7. from torchvision import transforms
    8. import cv2
    9.  
    10.  
    11. if __name__ == '__main__':
    12.     model = fasterrpn_resnet50_fpn(pretrained=True)
    13.     model.eval()
    14.  
    15.     img_tensor_list = list()
    16.  
    17.     transform_func = transforms.Compose([
    18.         transforms.ToTensor(),
    19.         # transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    20.     ])
    21.     input_height, input_width = (600 + 31) // 32 * 32, (800 + 31) // 32 * 32
    22.  
    23.     image_list = list()
    24.     for item in glob.glob("./*.jpg"):
    25.         image_list.append(cv2.resize(cv2.imread(item), dsize=(input_width, input_height)))
    26.         img_tensor_list.append(
    27.             transform_func(cv2.cvtColor(cv2.resize(cv2.imread(item), dsize=(input_width, input_height)), cv2.COLOR_BGR2RGB)))
    28.  
    29.     with torch.no_grad():
    30.         results = model(img_tensor_list, is_show=True)
    31.     result = results[0]
    32.     for i, item in enumerate(result):
    33.         image = image_list[i].copy()
    34.         for score_box in item:
    35.             box = score_box[1:]
    36.             box = box.numpy()
    37.             cv2.rectangle(image, tuple(map(int, box[0:2])), tuple(map(int, box[2:4])), (0, 255, 0))
    38.         cv2.imshow("win", image)
    39.         cv2.waitKey()
    40.         cv2.destroyWindow("win")
    41.  
    42.     output_names = ["rpn_boxes", *tuple(['feature_'+item for item in results[1].keys()])]
    43.  
    44.     print(output_names)
    45.     dynamic_axes = {'input':{0: "N"},'rpn_boxes': {0: "N"},'feature_0': {0: "N"}, 'feature_1': {0: "N"}, 'feature_2': {0: "N"}, 'feature_3': {0: "N"}, 'feature_pool': {0: "N"}}
    46.     onnx_save_path = 'rpn_backbone_resnet50.onnx'
    47.     torch.onnx.export(model, torch.rand(2, 3, input_height, input_width), onnx_save_path, verbose=False,
    48.                       do_constant_folding=True,
    49.                       input_names=["input"], output_names=output_names,
    50.                       dynamic_axes=dynamic_axes,
    51.                       opset_version=11)
    52.  
    53.     import onnxsim
    54.     import onnx
    55.     model = onnx.load(onnx_save_path)
    56.     # convert model
    57.     model_simp, check = onnxsim.simplify(model, check_n=0,input_shapes={'input':[-1,3,input_height,input_width]},
    58.                                 dynamic_input_shape=True)
    59.                                 # dynamic_input_shape=False)
    60.     with open(onnx_save_path,'wb') as f:
    61.     # with open(onnx_save_path.replace(".onnx","_simp.onnx"),'wb') as f:
    62.         onnx.save(model_simp, f)
    63.

    多图片可以批量预测了，但是预测结果都不对。

    第4步测试结果：

    单张图片测试代码：

    1. import onnxruntime as rt
    2. import numpy as  np
    3. import torch
    4. import torchvision
    5. import cv2
    6. from torchvision import transforms
    7.  
    8. def get_classes(filepath):
    9.     with open(filepath, 'r', encoding='gbk') as f:
    10.         return [item.strip() for item in f.readlines()]
    11.  
    12.  
    13. if __name__ == '__main__':
    14.     onnx_save_path = "rpn_backbone_resnet50.onnx"
    15.     img = cv2.imread('./car.jpg')
    16.     img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    17.     img = cv2.resize(img, dsize=(800, 608))
    18.  
    19.     normalize = transforms.Compose([
    20.         transforms.ToTensor(),
    21.         transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    22.     ])
    23.     img_tensor = normalize(img).unsqueeze(dim=0)
    24.     img_input = img_tensor.numpy().astype(np.float32)
    25.  
    26.     sess = rt.InferenceSession(onnx_save_path)
    27.     input_name = sess.get_inputs()[0].name
    28.     label_names = [sess.get_outputs()[i].name for i in range(1)]
    29.  
    30.     print("input_name",input_name)
    31.  
    32.     pred_onnx = sess.run(label_names, {input_name:img_input})
    33.  
    34.     print("label_names", label_names,"pred size",pred_onnx[0].shape)
    35.     # output without nms
    36.     pred_onnx = dict(zip(label_names, pred_onnx))
    37.  
    38.     image = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
    39.     for box in pred_onnx['rpn_boxes'][0]:
    40.         box = box[1:]
    41.         cv2.rectangle(image, tuple(map(int,box[0:2])), tuple(map(int,box[2:4])), (0,255,0))
    42.     cv2.imshow("img", img)
    43.     cv2.imshow("win", image)
    44.     cv2.waitKey()
    45.     cv2.destroyWindow("win")
    46.  
    47.

    步骤4测试可视化结果，与3是不一样的， 4390*6，步骤3的结果是1000*6

    第四步backbone输入输出维度：

    input_name input
    label_names ['rpn_boxes'] pred size (1, 4390, 6)

    多张图片预测代码：

    1. import onnxruntime as rt
    2. import numpy as  np
    3. import torch
    4. import torchvision
    5. import cv2
    6. from torchvision import transforms
    7.  
    8. def get_classes(filepath):
    9.     with open(filepath, 'r', encoding='gbk') as f:
    10.         return [item.strip() for item in f.readlines()]
    11.  
    12.  
    13. if __name__ == '__main__':
    14.     onnx_save_path = "rpn_backbone_resnet50.onnx"
    15.     img = cv2.imread('./car.jpg')
    16.     img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    17.     img = cv2.resize(img, dsize=(800, 608))
    18.     print(img.shape)
    19.     normalize = transforms.Compose([
    20.         transforms.ToTensor(),
    21.         # transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    22.     ])
    23.     img_tensor = normalize(img)#.unsqueeze(dim=0)
    24.  
    25.     img_input = img_tensor.numpy().astype(np.float32)
    26.  
    27.     img_input=np.array([img_input,img_input])
    28.  
    29.     sess = rt.InferenceSession(onnx_save_path)
    30.     input_name = sess.get_inputs()[0].name
    31.  
    32.     for data in sess.get_outputs():
    33.  
    34.         print("outname",data.name)
    35.     label_names = [sess.get_outputs()[i].name for i in range(1)]
    36.  
    37.     print("input_name",input_name)
    38.  
    39.     pred_onnx = sess.run(label_names, {input_name:img_input})
    40.  
    41.     print("label_names", label_names,"pred size",pred_onnx[0].shape)
    42.     # output without nms
    43.     pred_onnx = dict(zip(label_names, pred_onnx))
    44.  
    45.     image = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
    46.     for box in pred_onnx['rpn_boxes'][0]:
    47.         box = box[1:]
    48.         cv2.rectangle(image, tuple(map(int,box[0:2])), tuple(map(int,box[2:4])), (0,255,0))
    49.     cv2.imshow("img", img)
    50.     cv2.imshow("win", image)
    51.     cv2.waitKey()
    52.     cv2.destroyWindow("win")
    53.  
    54.

    第5步做了修改：

    改完支持多batch_size，但是多张图片预测，结果都是第一张图片的。

    1. import torch
    2. import os
    3. import sys
    4. sys.path.insert(0, os.path.abspath('..'))
    5. from model import fasterrpn_resnet50_fpn, fasterroiheader_resnet50_fpn
    6. import math
    7. import glob
    8. from torchvision import transforms
    9. import cv2
    10. import os
    11.  
    12.  
    13. if __name__ == '__main__':
    14.     model = fasterrpn_resnet50_fpn(pretrained=True)
    15.     model_header = fasterroiheader_resnet50_fpn(pretrained=True, transform=model.transform, box_score_thresh=0.5,box_nms_thresh=0.3)
    16.     model.eval()
    17.     model_header.eval()
    18.     img_tensor_list = list()
    19.  
    20.     transform_func = transforms.Compose([
    21.         transforms.ToTensor(),
    22.         # transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    23.     ])
    24.     input_height, input_width = (600 + 31) // 32 * 32, (800 + 31) // 32 * 32
    25.     image_list = list()
    26.     for item in glob.glob("./*.jpg"):
    27.         image_list.append(cv2.resize(cv2.imread(item), dsize=(input_width, input_height)))
    28.         img_tensor_list.append(
    29.             transform_func(cv2.cvtColor(cv2.resize(cv2.imread(item), dsize=(input_width, input_height)), cv2.COLOR_BGR2RGB)))
    30.  
    31.  
    32.     with torch.no_grad():
    33.         proposals, features, images, original_image_sizes = model(img_tensor_list)
    34.         if not os.path.exists('buffle.pkl'):
    35.             with open('buffle.pkl', 'wb') as f:
    36.                 torch.save({
    37.                     'proposals':proposals,
    38.                     'features':features,
    39.                     'image_sizes':images.image_sizes
    40.                 }, f)
    41.         for feature in features.values():
    42.             print('feature.shape',feature.shape)
    43.         proposals = [item[:,1:] for item in proposals]
    44.         if not os.path.exists('roi_result.pkl'):
    45.             roi_result = model_header.roi_heads.box_roi_pool(features, proposals, images.image_sizes)
    46.  
    47.             with open('roi_result.pkl', 'wb') as f:
    48.                 torch.save({
    49.                     'roi_result':roi_result
    50.                 }, f)
    51.  
    52.             dummy_image = torch.rand(1, 3, input_height, input_width)
    53.             batch_size = int(dummy_image.size(0))
    54.             dummy_proposals = [torch.rand((model.rpn.post_nms_top_n(), 4)) for _ in range(batch_size)]
    55.             height, width = int(dummy_image.size(2)), int(dummy_image.size(3))
    56.             dummy_features = {
    57.                 key: torch.rand(batch_size, model.backbone.out_channels, math.ceil(height / (2 ** (i + 2))),
    58.                                 math.ceil(width / (2 ** (i + 2)))) for i, key in enumerate(features.keys())}
    59.  
    60.             input_names = [*tuple(['feature_' + key for key in dummy_features.keys()]), 'proposals']
    61.  
    62.             dynamic_axes = {'proposals': {0: "N"}}
    63.             dynamic_axes.update({'feature_'+key: {0: "B"} for key in dummy_features.keys()})
    64.             dynamic_axes.update({name: {0: "N"} for name in ['outputs']})
    65.  
    66.             class Wrapper(torch.nn.Module):
    67.  
    68.                 def __init__(self, image_sizes, model):
    69.                     super(Wrapper, self).__init__()
    70.                     self.image_sizes = image_sizes
    71.                     self.model = model
    72.                 
    73.                 def forward(self, x, boxes):
    74.                     return self.model(x, boxes, self.image_sizes)
    75.             """
    76.             torch.onnx.export(Wrapper(images.image_sizes, model_header.roi_heads.box_roi_pool), (features, dummy_proposals),
    77.                               "roialign.onnx", verbose=True,
    78.                               do_constant_folding=True,
    79.                               input_names=input_names, output_names=["outputs"],
    80.                               dynamic_axes=dynamic_axes,
    81.                               opset_version=11)
    83.             print(roi_result.shape)
    84.             """
    85.  
    86.         result = model_header(features, proposals, images, original_image_sizes)
    87.  
    88.     for i, item in enumerate(result):
    89.         image = image_list[i].copy()
    90.         for score_box in item['boxes']:
    91.             box = score_box
    92.             box = box.numpy()
    93.             cv2.rectangle(image, tuple(map(int, box[0:2])), tuple(map(int, box[2:4])), (0, 255, 0))
    94.         cv2.imshow("win", image)
    95.         cv2.waitKey()
    96.         cv2.destroyWindow("win")
    97.     output_names = ["boxes", "labels", "scores"]
    98.  
    99.     dummy_image = torch.rand(1, 3, input_height, input_width)
    100.     batch_size = int(dummy_image.size(0))
    101.     dummy_proposals = [torch.rand((model.rpn.post_nms_top_n(), 4)) for _ in range(batch_size)]
    102.     height,width = int(dummy_image.size(2)),int(dummy_image.size(3))
    103.     dummy_features = {key:torch.rand(batch_size, model.backbone.out_channels, math.ceil(height / (2 ** (i + 2))), math.ceil(width / (2 ** (i + 2)))) for i,key in enumerate(features.keys())}
    104.     print(dummy_features.keys())
    105.     input_names = [*tuple(['feature_'+key for key in dummy_features.keys()]), 'proposals']
    106.     dynamic_axes = {'proposals': {0: "N"}}
    107.     dynamic_axes.update({name: {0: "N"} for name in output_names})
    108.     onnx_save_path = "header.onnx"
    109.     torch.onnx.export(model_header, (dummy_features, dummy_proposals, dummy_image), onnx_save_path, verbose=True,
    110.                       do_constant_folding=True,
    111.                       input_names=input_names, output_names=output_names,
    112.                       dynamic_axes=dynamic_axes,
    113.                       opset_version=11)
    114.

    第6步测试backbone输出维度：

    1.  
    2. import onnx_graphsurgeon as gs
    3. import onnx
    4.  
    5. def cutOnnx():
    6.     onnx_save_path = "rpn_backbone_resnet50.onnx"
    7.     graph = gs.import_onnx(onnx.load(onnx_save_path))
    8.  
    9.     for output in graph.outputs:
    10.  
    11.         print(0,output)
    12.     graph.outputs = graph.outputs[0:-1]
    13.     for output in graph.outputs:
    14.         print(1,output)
    15.     graph.cleanup()
    16.     # remove feature pool
    17.     onnx.save(gs.export_onnx(graph), onnx_save_path)
    18.  
    19. if __name__ == '__main__':
    20.     cutOnnx()

    结果：

    #0 Variable (rpn_boxes): (shape=[1, 4390, 6], dtype=float32)
# 0 Variable (feature_0): (shape=[1, 256, 152, 200], dtype=float32)
# 0 Variable (feature_1): (shape=[1, 256, 76, 100], dtype=float32)
# 0 Variable (feature_2): (shape=[1, 256, 38, 50], dtype=float32)
# 0 Variable (feature_3): (shape=[1, 256, 19, 25], dtype=float32)
# 1 Variable (rpn_boxes): (shape=[1, 4390, 6], dtype=float32)
# 1 Variable (feature_0): (shape=[1, 256, 152, 200], dtype=float32)
# 1 Variable (feature_1): (shape=[1, 256, 76, 100], dtype=float32)
# 1 Variable (feature_2): (shape=[1, 256, 38, 50], dtype=float32)

    第7步测试：

    x07reduce_header_onnx.py：精简header onnx，仅保留全连接层部分。具体细节：

    1.将网络的输入更改为roialigned_feature和proposals，去掉roi align和fpn_level的计算部分。

    tensor = tensors["218"] Line11 这段代码即对该reshape操作输入的替换。

    218哪里来的？

    netron软件查看的，右下角INPUTS的 data name218

    这个层在整个结构的大概位置：

    这里可以根据自己的生成的onnx改变名字，下面是局部放大点的图：

    1. 对输出的box和score节点的前面的reshape操作进行处理

    3. 对应一下几行代码

      这里需要根据自己的生成的onnx修改，切记

      shape_score = gs.Constant(name="shape_score", values=np.array((-1, 90), dtype=np.int64))
    shape_boxes = gs.Constant(name="shape_boxes", values=np.array((-1, 90, 4), dtype=np.int64))
    shape_boxes_last_node = gs.Constant(name="shape_boxes_last_node", values=np.array((-1, 91, 4), dtype=np.int64))

# 这里的Reshape_320和Reshape_322是box和score的上一个reshape节点
# 这里填写上面的框选的部分分
for node in graph.nodes:
    if node.name == "Reshape_320":
        node.inputs[-1] = shape_boxes
    elif node.name == "Reshape_322":
        node.inputs[-1] = shape_score
    # the last second reshape node relative to box output
    elif node.name == "Reshape_308":
        node.inputs[-1] = shape_boxes_last_node

    好像报错了：

    [W] colored module is not installed, will not use colors when logging. To enable colors, please install the colored module: python3 -m pip install colored
    [E] No function: shape registered for opset: 11
    [W] colored module is not installed, will not use colors when logging. To enable colors, please install the colored module: python3 -m pip install colored
    [E] No function: __len__ registered for opset: 11

    但是没有红色显示

    清华园好像不能安装了，用豆瓣的源可以安装：

    pip install colored -i https://pypi.doubanio.com/simple

    第7步按照作者的转换脚本，开始的时候报错，代码：

    1. import onnx_graphsurgeon as gs
    2. import onnx
    3. import numpy as np
    4.  
    5.  
    6. def cutOnnx():
    7.     onnx_save_path = "header.onnx"
    8.     graph = gs.import_onnx(onnx.load(onnx_save_path))
    9.  
    10.     tensors = graph.tensors()
    11.     tensor = tensors["218"]
    12.     graph.inputs = [graph.inputs[-1], tensor.to_variable(dtype=np.float32, shape=('N', 256, 7, 7))]
    13.     graph.inputs[-1].name = "roialigned_feature"
    14.  
    15.     graph.outputs = [graph.outputs[0], graph.outputs[-1]]
    16.  
    17.     shape_score = gs.Constant(name="shape_score", values=np.array((-1, 90), dtype=np.int64))
    18.     shape_boxes = gs.Constant(name="shape_boxes", values=np.array((-1, 90, 4), dtype=np.int64))
    19.     shape_boxes_last_node = gs.Constant(name="shape_boxes_last_node", values=np.array((-1, 91, 4), dtype=np.int64))
    20.  
    21.     # 这里的Reshape_320和Reshape_322是box和score的上一个reshape节点
    22.     for node in graph.nodes:
    23.         if node.name == "Reshape_320":
    24.             node.inputs[-1] = shape_boxes
    25.         elif node.name == "Reshape_322":
    26.             node.inputs[-1] = shape_score
    27.         # the last second reshape node relative to box output
    28.         elif node.name == "Reshape_308":
    29.             node.inputs[-1] = shape_boxes_last_node
    30.     
    31.     # 添加N,90,4 和 N,90,1的结点
    32.     for item in graph.outputs:
    33.         item.shape.insert(1, 90)
    34.         # print(item.shape)
    35.     for graph_output in graph.outputs:
    36.         graph_output.shape[0] = 'N'
    37.     graph.cleanup()
    38.     new_onnx_filepath = 'new_'+onnx_save_path
    39.     onnx.save(gs.export_onnx(graph), new_onnx_filepath)
    40.  
    41.     import onnxsim
    42.     model = onnx.load(new_onnx_filepath)
    43.     # convert model
    44.     model_simp, check = onnxsim.simplify(model, check_n=0,input_shapes={'roialigned_feature':[1,256, 7, 7],'proposals':[1,4]}, 
    45.                                 dynamic_input_shape=True)
    46.  
    47.     onnx.save(model_simp, new_onnx_filepath)
    48.  
    49.  
    50.  
    51. if __name__ == '__main__':
    52.     cutOnnx()

    报错： 

      tensor = tensors["218"]
    KeyError: '218'

    torch换到作者的版本1.10，这个报错没有了。

    修改了一下，支持批量预测：

    1. import onnx_graphsurgeon as gs
    2. import onnx
    3. import numpy as np
    4.  
    5.  
    6. def cutOnnx():
    7.     onnx_save_path = "header.onnx"
    8.     graph = gs.import_onnx(onnx.load(onnx_save_path))
    9.  
    10.     tensors = graph.tensors()
    11.  
    12.     for key, value in tensors.items():
    13.         print(key , value)
    14.  
    15.     # tensor = tensors["onnx::Reshape_325"]
    16.     tensor = tensors["218"]
    17.     graph.inputs = [graph.inputs[-1], tensor.to_variable(dtype=np.float32, shape=('batch_size', 256, 7, 7))]
    18.     graph.inputs[-1].name = "roialigned_feature"
    19.  
    20.     graph.outputs = [graph.outputs[0], graph.outputs[-1]]
    21.  
    22.     shape_score = gs.Constant(name="shape_score", values=np.array((-1, 90), dtype=np.int64))
    23.     shape_boxes = gs.Constant(name="shape_boxes", values=np.array((-1, 90, 4), dtype=np.int64))
    24.     shape_boxes_last_node = gs.Constant(name="shape_boxes_last_node", values=np.array((-1, 91, 4), dtype=np.int64))
    25.  
    26.     # 这里的Reshape_320和Reshape_322是box和score的上一个reshape节点
    27.     for node in graph.nodes:
    28.         if node.name == "Reshape_320":
    29.             node.inputs[-1] = shape_boxes
    30.         elif node.name == "Reshape_322":
    31.             node.inputs[-1] = shape_score
    32.         # the last second reshape node relative to box output
    33.         elif node.name == "Reshape_308":
    34.             node.inputs[-1] = shape_boxes_last_node
    35.     
    36.     # 添加N,90,4 和 N,90,1的结点
    37.     for item in graph.outputs:
    38.         item.shape.insert(1, 90)
    39.         # print(item.shape)
    40.     for graph_output in graph.outputs:
    41.         graph_output.shape[0] = 'N'
    42.     graph.cleanup()
    43.     new_onnx_filepath = 'new_'+onnx_save_path
    44.     onnx.save(gs.export_onnx(graph), new_onnx_filepath)
    45.  
    46.     import onnxsim
    47.     model = onnx.load(new_onnx_filepath)
    48.     # convert model
    49.     model_simp, check = onnxsim.simplify(model, check_n=0,input_shapes={'roialigned_feature':[-1,256, 7, 7],'proposals':[-1,4]},
    50.                                 dynamic_input_shape=True)
    51.  
    52.     onnx.save(model_simp, new_onnx_filepath)
    53.  
    54.  
    55.  
    56. if __name__ == '__main__':
    57.     cutOnnx()

    第8步测试

    1. import onnxruntime as rt
    2. import numpy as  np
    3.  
    4.  
    5. if __name__ == '__main__':
    6.  
    7.     sess = rt.InferenceSession('new_header.onnx')
    8.     input_names = [item.name for item in sess.get_inputs()]
    9.     output_names = [item.name for item in sess.get_outputs()]
    10.  
    11.     # proposal = np.array([1,1,10,10], dtype=np.float32).reshape(-1, 4)
    12.     batch_size = 1
    13.     input_dict = dict(
    14.         proposals = np.random.randn(batch_size, 4).astype(dtype=np.float32),
    15.         roialigned_feature = np.random.randn(batch_size, 256, 7, 7).astype(dtype=np.float32)
    16.     )
    17.     pred_onnx = sess.run(output_names, input_dict)
    18.     pred_onnx = dict(zip(output_names, pred_onnx))
    19.     print(pred_onnx['boxes'].shape)
    20.     # print(pred_onnx['boxes'])
    21.     print(pred_onnx['scores'].shape)
    22.     # print(pred_onnx['scores'])

    结果bath_size为1时正确，大于1就报错。

    输入两个参数：proposals 和roialigned_feature

    维度为 batch_size,4

    和batch_size,256, 7, 7

    输出：

    (1, 90, 4)
    (1, 90)

    onnx转trt操作

    c++ 转换trt：

    builder\trt_builder.cpp

    报错代码：

        void set_layer_hook_reshape(const LayerHookFuncReshape& func){
            //register_layerhook_reshape(func);
        }

    register_layerhook_reshape函数是在NvOnnxParser.cpp中，

    OnnxParser代码是在生成的时候用，需要protobuf，版本未知，

    protobuf-cpp-3.11.4

    pytorch转onnx

    开源项目给了转换代码：

    test/x01export_FasterRCNN_onnx.py

    onnx测试代码：

    test/x02test_FasterRCNN_onnx.py

    Onnx转trt代码及操作：

    onnx转tensorrt 实战干货总结_AI视觉网奇的博客-CSDN博客_onnx转tensorrt

    博客的目录：onnx转tensorrt 分类成功

    转trt报错了

    1. [08/06/2022-11:35:29] [TRT] [E] [graphShapeAnalyzer.cpp::nvinfer1::builder::`anonymous-namespace'::ShapeNodeRemover::analyzeShapes::1285] Error Code 4: Miscellaneous (IShuffleLayer Reshape_1226: reshape changes volume. Reshaping [720588174] to [1,4507].)
    2. Completed parsing of ONNX file
    3. Building an engine from file F:\project\jushi\tensorrt-onnx-fasterrcnn-fpn-roialign-master\test\fasterrcnn_backbone_resnet50_fpn_roialign.onnx; this may take a while...
    4. [08/06/2022-11:35:29] [TRT] [E] 4: [network.cpp::nvinfer1::Network::validate::2633] Error Code 4: Internal Error (Network must have at least one output)

    解决方法，手动设置最后一层：

            last_layer = network.get_layer(network.num_layers - 1)
            network.mark_output(last_layer.get_output(0))

    代码：

    1. def ONNX_build_engine(onnx_file_path, write_engine=True):
    2.     # 通过加载onnx文件，构建engine
    3.     # :param onnx_file_path: onnx文件路径
    4.     # :return: engine
    5.  
    6.     G_LOGGER = trt.Logger(trt.Logger.WARNING)
    7.     # 1、动态输入第一点必须要写的
    8.     explicit_batch = 1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
    9.     batch_size = 8  # trt推理时最大支持的batchsize
    10.     with trt.Builder(G_LOGGER) as builder, builder.create_network(explicit_batch) as network, trt.OnnxParser(network, G_LOGGER) as parser:
    11.         builder.max_batch_size = batch_size
    12.         config = builder.create_builder_config()
    13.         config.max_workspace_size = GiB(2)
    14.         config.set_flag(trt.BuilderFlag.FP16)
    15.         print('Loading ONNX file from path {}...'.format(onnx_file_path))
    16.         with open(onnx_file_path, 'rb') as model:
    17.             print('Beginning ONNX file parsing')
    18.             parser.parse(model.read())
    19.         print('Completed parsing of ONNX file')
    20.         print('Building an engine from file {}; this may take a while...'.format(onnx_file_path))
    21.         # 重点
    22.         profile = builder.create_optimization_profile()  # 动态输入时候需要 分别为最小输入、常规输入、最大输入
    23.         # 有几个输入就要写几个profile.set_shape 名字和转onnx的时候要对应
    24.         # tensorrt6以后的版本是支持动态输入的，需要给每个动态输入绑定一个profile，用于指定最小值，常规值和最大值，如果超出这个范围会报异常。
    25.         profile.set_shape("inputs", (1, 3, 600, 600), (8, 3, 600, 600), (16, 3, 600, 600))
    26.         config.add_optimization_profile(profile)
    27.         last_layer = network.get_layer(network.num_layers - 1)
    28.         network.mark_output(last_layer.get_output(0))
    29.         engine = builder.build_engine(network, config)
    30.         print("Completed creating Engine")
    31.         # 保存engine文件
    32.         if write_engine:
    33.             engine_file_path = 'efficientnet_b1.trt'
    34.             with open(engine_file_path, "wb") as f:
    35.                 f.write(engine.serialize())
    36.         return engine

    但是维度不能对齐：

    ShapeNodeRemover::analyzeShapes::1285] Error Code 4: Miscellaneous (IShuffleLayer Reshape_1226: reshape changes volume. Reshaping [720588174] to [1,4507].)

    torch升级版本后，转trt报错变成了：

     Error Code 9: Internal Error (Floor_45: IUnaryLayer cannot be used to compute a shape tensor)

    backbone部分可以转trt：
rpn_backbone_resnet50.onnx

    c++ tensorrt推理部分

    c++onnx转tensorrt

    也可以python转，

    onnx生成tensorrt的时候，用的自带的代码，自带的，需要protobuf，

    “google/protobuf/port_def.inc”:

    官方的faster rcnn导出trt报错：

    reshape changes volume. Reshaping [720588174] to [1,4507].)

    fasterrcnn.cpp：

    缩放尺寸，归一化:

    virtual bool preprocess(Job& job, const Mat& image) override{

    打印模型输入输出参数：

    void InferImpl::print(){

    xxxxx

    }

    宽800，高608，宽高比和car图片相反。

    推理报错：

    sub_model推理报错：

    [][error][trt_builder.cpp:30]:NVInfer: 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::setBindingDimensions::944] Error Code 3: API Usage Error (Parameter check failed at: executionContext.cpp::nvinfer1::rt::ExecutionContext::setBindingDimensions::944, condition: profileMaxDims.d[i] >= dimensions.d[i]. Supplied binding dimension [787,256,7,7] for bindings[1] exceed min ~ max range at index 0, maximum dimension in profile is 1, minimum dimension in profile is 1, but supplied dimension is 787.
    )

    787是anchors的数量：new_header需要的为1。

    int number_anchors = roi_align_inputs_index / 6; 

    感觉就是backbone和new_header的维度没对上。

    new_header.onnx

    tensorrt推理：

    app_fasterrcnn\fasterrcnn.cpp

    1. //forward
    2. engine->forward(false);
    3. engine->synchronize();

    infer\trt_infer.cpp

    bool execute_result = context->context_->enqueueV2

    N和bach_size是一样的。

  • 相关阅读:
    qt在不同的线程中传递自定义结构体参数
    基于UE4 的AirSim虚拟仿真
    Linux面试常考命令
    计算机视觉与深度学习 | 视觉惯性SLAM的基础理论
    [附源码]java毕业设计校园博客系统
    [附源码]计算机毕业设计JAVA基于ssm的电子网上商城
    Android 面经总结分享（相当走心）
    若依前端vue设置子路径
    深度剖析Java的volatile实现原理，再也不怕面试官问了
    MongoDB数据库入门到精通看这一篇就够了
  • 原文地址：https://blog.csdn.net/jacke121/article/details/126191184