• 发票关键信息抽取SER

    SER（Semantic Entity Recognition）：语义实体识别。语义实体识别指的是给定一段文本行，确定其类别（如姓名、住址等类别）。本文采用基于VI-LayoutXLM的多模态语义实体识别方法。

    1、增值税发票数据集 

        https://download.csdn.net/download/ronshi/88467149

    2、训练模型

    python tools/train.py -c my/ser_vi_layoutxlm_xfund_zh_udml.yml 

    配置文件 ser_vi_layoutxlm_xfund_zh_udml.yml

    1. Global:
    2.   use_gpu: true
    3.   epoch_num: &epoch_num 200
    4.   log_smooth_window: 10
    5.   print_batch_step: 10
    6.   save_model_dir: ./output/ser_vi_layoutxlm_xfund_zh_udml
    7.   save_epoch_step: 2000
    8.   # evaluation is run every 10 iterations after the 0th iteration
    9.   eval_batch_step: [ 0, 19 ]
    10.   cal_metric_during_train: False
    11.   save_inference_dir:
    12.   use_visualdl: False
    13.   seed: 2022
    14.   infer_img: my/b201.jpg
    15.   save_res_path: ./output/ser_layoutxlm_xfund_zh/res
    16.  
    17.  
    18. Architecture:
    19.   model_type: &model_type "kie"
    20.   name: DistillationModel
    21.   algorithm: Distillation
    22.   Models:
    23.     Teacher:
    24.       pretrained:
    25.       freeze_params: false
    26.       return_all_feats: true
    27.       model_type: *model_type
    28.       algorithm: &algorithm "LayoutXLM"
    29.       Transform:
    30.       Backbone:
    31.         name: LayoutXLMForSer
    32.         pretrained: True
    33.         # one of base or vi
    34.         mode: vi
    35.         checkpoints:
    36.         num_classes: &num_classes 5
    37.     Student:
    38.       pretrained:
    39.       freeze_params: false
    40.       return_all_feats: true
    41.       model_type: *model_type
    42.       algorithm: *algorithm
    43.       Transform:
    44.       Backbone:
    45.         name: LayoutXLMForSer
    46.         pretrained: True
    47.         # one of base or vi
    48.         mode: vi
    49.         checkpoints:
    50.         num_classes: *num_classes
    51.  
    52.  
    53. Loss:
    54.   name: CombinedLoss
    55.   loss_config_list:
    56.   - DistillationVQASerTokenLayoutLMLoss:
    57.       weight: 1.0
    58.       model_name_list: ["Student", "Teacher"]
    59.       key: backbone_out
    60.       num_classes: *num_classes
    61.   - DistillationSERDMLLoss:
    62.       weight: 1.0
    63.       act: "softmax"
    64.       use_log: true
    65.       model_name_pairs:
    66.       - ["Student", "Teacher"]
    67.       key: backbone_out
    68.   - DistillationVQADistanceLoss:
    69.       weight: 0.5
    70.       mode: "l2"
    71.       model_name_pairs:
    72.         - ["Student", "Teacher"]
    73.       key: hidden_states_5
    74.       name: "loss_5"
    75.   - DistillationVQADistanceLoss:
    76.       weight: 0.5
    77.       mode: "l2"
    78.       model_name_pairs:
    79.         - ["Student", "Teacher"]
    80.       key: hidden_states_8
    81.       name: "loss_8"
    82.   
    83.   
    84.  
    85. Optimizer:
    86.   name: AdamW
    87.   beta1: 0.9
    88.   beta2: 0.999
    89.   lr:
    90.     name: Linear
    91.     learning_rate: 0.00005
    92.     epochs: *epoch_num
    93.     warmup_epoch: 10
    94.   regularizer:
    95.     name: L2
    96.     factor: 0.00000
    97.     
    98. PostProcess:
    99.   name: DistillationSerPostProcess
    100.   model_name: ["Student", "Teacher"]
    101.   key: backbone_out
    102.   class_path: &class_path my/zzsfp/class_list.txt
    103.  
    104. Metric:
    105.   name: DistillationMetric
    106.   base_metric_name: VQASerTokenMetric
    107.   main_indicator: hmean
    108.   key: "Student"
    109.  
    110. Train:
    111.   dataset:
    112.     name: SimpleDataSet
    113.     data_dir: my/zzsfp/imgs
    114.     label_file_list: 
    115.       - my/zzsfp/train.json
    116.     ratio_list: [ 1.0 ]
    117.     transforms:
    118.       - DecodeImage: # load image
    119.           img_mode: RGB
    120.           channel_first: False
    121.       - VQATokenLabelEncode: # Class handling label
    122.           contains_re: False
    123.           algorithm: *algorithm
    124.           class_path: *class_path
    125.           # one of [None, "tb-yx"]
    126.           order_method: &order_method "tb-yx"
    127.       - VQATokenPad:
    128.           max_seq_len: &max_seq_len 512
    129.           return_attention_mask: True
    130.       - VQASerTokenChunk:
    131.           max_seq_len: *max_seq_len
    132.       - Resize:
    133.           size: [224,224]
    134.       - NormalizeImage:
    135.           scale: 1
    136.           mean: [ 123.675, 116.28, 103.53 ]
    137.           std: [ 58.395, 57.12, 57.375 ]
    138.           order: 'hwc'
    139.       - ToCHWImage:
    140.       - KeepKeys:
    141.           keep_keys: [ 'input_ids', 'bbox', 'attention_mask', 'token_type_ids', 'image', 'labels'] # dataloader will return list in this order
    142.   loader:
    143.     shuffle: True
    144.     drop_last: False
    145.     batch_size_per_card: 4
    146.     num_workers: 4
    147.  
    148. Eval:
    149.   dataset:
    150.     name: SimpleDataSet
    151.     data_dir: my/zzsfp/imgs
    152.     label_file_list:
    153.       - my/zzsfp/val.json
    154.     transforms:
    155.       - DecodeImage: # load image
    156.           img_mode: RGB
    157.           channel_first: False
    158.       - VQATokenLabelEncode: # Class handling label
    159.           contains_re: False
    160.           algorithm: *algorithm
    161.           class_path: *class_path
    162.           order_method: *order_method
    163.       - VQATokenPad:
    164.           max_seq_len: *max_seq_len
    165.           return_attention_mask: True
    166.       - VQASerTokenChunk:
    167.           max_seq_len: *max_seq_len
    168.       - Resize:
    169.           size: [224,224]
    170.       - NormalizeImage:
    171.           scale: 1
    172.           mean: [ 123.675, 116.28, 103.53 ]
    173.           std: [ 58.395, 57.12, 57.375 ]
    174.           order: 'hwc'
    175.       - ToCHWImage:
    176.       - KeepKeys:
    177.           keep_keys: [ 'input_ids', 'bbox', 'attention_mask', 'token_type_ids', 'image', 'labels'] # dataloader will return list in this order
    178.   loader:
    179.     shuffle: False
    180.     drop_last: False
    181.     batch_size_per_card: 4
    182.     num_workers: 4

    3、模型评估

    python tools/eval.py -c my/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh_udml/best_accuracy

     配置文件 ser_vi_layoutxlm_xfund_zh.yml

    1. Global:
    2.   use_gpu: True
    3.   epoch_num: &epoch_num 200
    4.   log_smooth_window: 10
    5.   print_batch_step: 10
    6.   save_model_dir: ./output/ser_vi_layoutxlm_xfund_zh
    7.   save_epoch_step: 2000
    8.   # evaluation is run every 10 iterations after the 0th iteration
    9.   eval_batch_step: [ 0, 19 ]
    10.   cal_metric_during_train: False
    11.   save_inference_dir:
    12.   use_visualdl: False
    13.   seed: 2022
    14.   infer_img: my/b201.jpg
    15.   d2s_train_image_shape: [3, 224, 224]
    16.   # if you want to predict using the groundtruth ocr info,
    17.   # you can use the following config
    18.   # infer_img: train_data/XFUND/zh_val/val.json
    19.   # infer_mode: False
    20.  
    21.   save_res_path: ./output/ser/xfund_zh/res
    22.   kie_rec_model_dir: 
    23.   kie_det_model_dir:
    24.   amp_custom_white_list: ['scale', 'concat', 'elementwise_add']
    25.  
    26. Architecture:
    27.   model_type: kie
    28.   algorithm: &algorithm "LayoutXLM"
    29.   Transform:
    30.   Backbone:
    31.     name: LayoutXLMForSer
    32.     pretrained: True
    33.     checkpoints:
    34.     # one of base or vi
    35.     mode: vi
    36.     num_classes: &num_classes 5
    37.  
    38. Loss:
    39.   name: VQASerTokenLayoutLMLoss
    40.   num_classes: *num_classes
    41.   key: "backbone_out"
    42.  
    43. Optimizer:
    44.   name: AdamW
    45.   beta1: 0.9
    46.   beta2: 0.999
    47.   lr:
    48.     name: Linear
    49.     learning_rate: 0.00005
    50.     epochs: *epoch_num
    51.     warmup_epoch: 2
    52.   regularizer:
    53.     name: L2
    54.     factor: 0.00000
    55.     
    56. PostProcess:
    57.   name: VQASerTokenLayoutLMPostProcess
    58.   class_path: &class_path my/zzsfp/class_list.txt
    59.  
    60. Metric:
    61.   name: VQASerTokenMetric
    62.   main_indicator: hmean
    63.  
    64. Train:
    65.   dataset:
    66.     name: SimpleDataSet
    67.     data_dir: my/zzsfp/imgs
    68.     label_file_list: 
    69.       - my/zzsfp/train.json
    70.     ratio_list: [ 1.0 ]
    71.     transforms:
    72.       - DecodeImage: # load image
    73.           img_mode: RGB
    74.           channel_first: False
    75.       - VQATokenLabelEncode: # Class handling label
    76.           contains_re: False
    77.           algorithm: *algorithm
    78.           class_path: *class_path
    79.           use_textline_bbox_info: &use_textline_bbox_info True
    80.           # one of [None, "tb-yx"]
    81.           order_method: &order_method "tb-yx"
    82.       - VQATokenPad:
    83.           max_seq_len: &max_seq_len 512
    84.           return_attention_mask: True
    85.       - VQASerTokenChunk:
    86.           max_seq_len: *max_seq_len
    87.       - Resize:
    88.           size: [224,224]
    89.       - NormalizeImage:
    90.           scale: 1
    91.           mean: [ 123.675, 116.28, 103.53 ]
    92.           std: [ 58.395, 57.12, 57.375 ]
    93.           order: 'hwc'
    94.       - ToCHWImage:
    95.       - KeepKeys:
    96.           keep_keys: [ 'input_ids', 'bbox', 'attention_mask', 'token_type_ids', 'image', 'labels'] # dataloader will return list in this order
    97.   loader:
    98.     shuffle: True
    99.     drop_last: False
    100.     batch_size_per_card: 1
    101.     num_workers: 1
    102.  
    103. Eval:
    104.   dataset:
    105.     name: SimpleDataSet
    106.     data_dir: my/zzsfp/imgs
    107.     label_file_list:
    108.       - my/zzsfp/val.json
    109.     transforms:
    110.       - DecodeImage: # load image
    111.           img_mode: RGB
    112.           channel_first: False
    113.       - VQATokenLabelEncode: # Class handling label
    114.           contains_re: False
    115.           algorithm: *algorithm
    116.           class_path: *class_path
    117.           use_textline_bbox_info: *use_textline_bbox_info
    118.           order_method: *order_method
    119.       - VQATokenPad:
    120.           max_seq_len: *max_seq_len
    121.           return_attention_mask: True
    122.       - VQASerTokenChunk:
    123.           max_seq_len: *max_seq_len
    124.       - Resize:
    125.           size: [224,224]
    126.       - NormalizeImage:
    127.           scale: 1
    128.           mean: [ 123.675, 116.28, 103.53 ]
    129.           std: [ 58.395, 57.12, 57.375 ]
    130.           order: 'hwc'
    131.       - ToCHWImage:
    132.       - KeepKeys:
    133.           keep_keys: [ 'input_ids', 'bbox', 'attention_mask', 'token_type_ids', 'image', 'labels'] # dataloader will return list in this order
    134.   loader:
    135.     shuffle: False
    136.     drop_last: False
    137.     batch_size_per_card: 1
    138.     num_workers: 1

    4、模型预测 

    python tools/infer_kie_token_ser.py -c my/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh_udml/best_accuracy Global.infer_img=./my/zzsfp/val.json Global.infer_mode=False

    python tools/infer_kie_token_ser.py -c my/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh_udml/best_accuracy Global.infer_img=./my/b201.jpg Global.infer_mode=True 

    5、模型导出 

    python tools/export_model.py -c my/ser_vi_layoutxlm_xfund_zh.yml -o Global.pretrained_model=./output/ser_vi_layoutxlm_xfund_zh_udml/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm

     6、模型推理

    python ppstructure/kie/predict_kie_token_ser.py \
      --kie_algorithm=LayoutXLM \
      --ser_model_dir=./inference/ser_vi_layoutxlm \
      --image_dir=./my/b201.jpg \
      --ser_dict_path=./my/zzsfp/class_list.txt \
      --vis_font_path=./doc/fonts/simfang.ttf \
      --ocr_order_method="tb-yx"

  • 相关阅读:
    python与pycharm配置http服务
    线性同余方程（ 数学知识 + 同余 + 扩展欧几里得算法 ）
    计算机毕业设计Java自行车在线租赁管理系统2021（源码+系统+mysql数据库+Lw文档）
    AAC算法
    tcpdump工具使用
    Java自定义注解解析
    python版超市信息管理系统源代码，基于tkinter带界面
    关于#搜索引擎#的问题：虽然静下心学英语，更适合中国宝宝认知的学习英语，应试英语的方法，针对于听力、阅读、单词量（最头疼）、作文，有什么方向或经验能推荐、分享一下吗
    Java版分布式微服务云开发架构 Spring Cloud+Spring Boot+Mybatis 电子招标采购系统功能清单
    JuiceFS 在多云存储架构中的应用 | 深势科技分享
  • 原文地址：https://blog.csdn.net/ronshi/article/details/134017751