Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PaddleDetection/configs /rtdetrv2/][部署章节 -> 导出模型]python3 tools/export_model.py失败 #9281

Open
3 tasks done
zjykzj opened this issue Jan 14, 2025 · 0 comments
Open
3 tasks done

[PaddleDetection/configs /rtdetrv2/][部署章节 -> 导出模型]python3 tools/export_model.py失败 #9281

zjykzj opened this issue Jan 14, 2025 · 0 comments
Assignees
@LokeZhou @zhangyubo0722

Comments

@zjykzj
Copy link

zjykzj commented Jan 14, 2025

问题确认 Search before asking

  • 我已经查询历史issue，没有发现相似的bug。I have searched the issues and found no similar bug report.

Bug组件 Bug Component

Export

Bug描述 Describe the Bug

在issues遇到两个类似的问题，看了里面的内容都没有解决，跟我的处理流程略有差别，所以单开一个issue，

  1. 将rtdetrv2动态图模型转化为静态图模型失败 #9140
  2. rtdetrv2导出模型失败 #9211

开发环境：

  1. NVIDIA信息：device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.3, Runtime API Version: 11.2
  2. Docker容器：paddlepaddle/paddle:2.6.1-gpu-cuda11.2-cudnn8.2-trt8.0
  3. Paddle版本：2.6.1

切换PaddleDetection到v2.8.0，这个是目前最新的tag。参考RTDETR-v2的README文件：https://github.com/PaddlePaddle/PaddleDetection/tree/v2.8.0/configs/rtdetrv2

直接使用<部署>章节的导出模型命令：

cd PaddleDetection
python tools/export_model.py -c configs/rtdetrv2/rtdetrv2_r50vd_6x_coco.yml \
              -o weights=https://bj.bcebos.com/v1/paddledet/models/rtdetrv2_r50vd_6x_coco.pdparams trt=True \
              --output_dir=output_inference

失败了，显示如下信息：

λ 9686674b44f1 /data/zj/paddle/PaddleDetection python tools/export_model.py -c configs/rtdetrv2/rtdetrv2_r50vd_6x_coco.yml -o weights=https://bj.bcebos.com/v1/paddledet/models/rtdetrv2_r50vd_6x_coco.pdparams trt=True --output_dir=output_inference
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly
[01/14 06:55:07] ppdet.utils.checkpoint INFO: Finish loading model weights: /root/.cache/paddle/weights/rtdetrv2_r50vd_6x_coco.pdparams
[01/14 06:55:08] ppdet.data.source.category WARNING: anno_file 'dataset/coco/annotations/instances_val2017.json' is None or not set or not exist, please recheck TrainDataset/EvalDataset/TestDataset.anno_path, otherwise the default categories will be used by metric_type.
[01/14 06:55:08] ppdet.data.source.category WARNING: metric_type: COCO, load default categories of COCO.
[01/14 06:55:08] ppdet.engine INFO: Export inference config file to output_inference/rtdetrv2_r50vd_6x_coco/infer_cfg.yml
Traceback (most recent call last):
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 118, in <module>
    main()
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 114, in main
    run(FLAGS, cfg)
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 80, in run
    trainer.export(FLAGS.output_dir, for_fd=FLAGS.for_fd)
  File "/data/zj/paddle/PaddleDetection/ppdet/engine/trainer.py", line 1282, in export
    static_model, pruned_input_spec = self._get_infer_cfg_and_input_spec(
  File "/data/zj/paddle/PaddleDetection/ppdet/engine/trainer.py", line 1233, in _get_infer_cfg_and_input_spec
    input_spec, static_model.forward.main_program,
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 1062, in main_program
    concrete_program = self.concrete_program
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 941, in concrete_program
    return self.concrete_program_specify_input_spec(input_spec=None)
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 986, in concrete_program_specify_input_spec
    concrete_program, _ = self.get_concrete_program(
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 875, in get_concrete_program
    concrete_program, partial_program_layer = self._program_cache[
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 1648, in __getitem__
    self._caches[item_id] = self._build_once(item)
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 1575, in _build_once
    concrete_program = ConcreteProgram.from_func_spec(
  File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/usr/local/lib/python3.10/dist-packages/paddle/base/wrapped_decorator.py", line 26, in __impl__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/paddle/base/dygraph/base.py", line 68, in __impl__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 1346, in from_func_spec
    error_data.raise_new_exception()
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/error.py", line 452, in raise_new_exception
    raise new_exception from None
AssertionError: In transformed code:

    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 59, in forward
        if self.training:
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 69, in forward
        for inp in inputs_list:
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 76, in forward
        outs.append(self.get_pred())
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/detr.py", line 118, in get_pred
        return self._forward()
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/detr.py", line 89, in _forward
        out_transformer = self.transformer(body_feats, pad_mask, self.inputs)
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/rtdetr_transformerv2.py", line 480, in forward
        out_bboxes, out_logits = self.decoder(
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/rtdetr_transformer.py", line 235, in forward
        output = layer(output, ref_points_input, memory,
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/rtdetr_transformerv2.py", line 259, in forward
        tgt2 = self.cross_attn(
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/rtdetr_transformerv2.py", line 173, in forward
                format(reference_points.shape[-1]))

        output = self.ms_deformable_attn_core(value, value_spatial_shapes,
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                                              sampling_locations, attention_weights)
        output = self.output_proj(output)

    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/utils.py", line 162, in deformable_attention_core_func_v2
        value_list = [
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/utils.py", line 163, in <listcomp>
        value.reshape([batch_num * num_heads, head_dim, h, w])
    File "/usr/local/lib/python3.10/dist-packages/paddle/tensor/manipulation.py", line 4017, in reshape
        attrs["shape"] = get_attr_shape(shape)
    File "/usr/local/lib/python3.10/dist-packages/paddle/tensor/manipulation.py", line 3921, in get_attr_shape
        assert dim_size > 0, (

    AssertionError: Each dimension value of 'shape' in reshape must not be negative except one unknown dimension. But received shape[0] = -8.

参考前面的回答，尝试显式指定输入shape大小，重新执行后仍旧报错

λ 9686674b44f1 /data/zj/paddle/PaddleDetection python tools/export_model.py -c configs/rtdetrv2/rtdetrv2_r50vd_6x_coco.yml -o weights=https://bj.bcebos.com/v1/paddledet/models/rtdetrv2_r50vd_6x_coco.pdparams trt=True --output_dir=output_inference -o TestReader.inputs_def.image_shape=[1,3,640,640]
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly
Traceback (most recent call last):
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 118, in <module>
    main()
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 114, in main
    run(FLAGS, cfg)
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 77, in run
    trainer.load_weights(cfg.weights)
  File "/data/zj/paddle/PaddleDetection/ppdet/engine/trainer.py", line 472, in load_weights
    load_pretrain_weight(self.model, weights, ARSL_eval)
  File "/data/zj/paddle/PaddleDetection/ppdet/utils/checkpoint.py", line 243, in load_pretrain_weight
    raise ValueError("Model pretrain path `{}` does not exists. "
ValueError: Model pretrain path `output/rtdetrv2_r50vd_6x_coco/model_final` does not exists. If you don't want to load pretrain model, please delete `pretrain_weights` field in config file.

把文件https://bj.bcebos.com/v1/paddledet/models/rtdetrv2_r50vd_6x_coco.pdparams下载到output/rtdetrv2_r50vd_6x_coco/model_final/，然后修改配置文件rtdetrv2_r50vd_6x_coco.yml的weights选项：

#weights: output/rtdetrv2_r50vd_6x_coco/model_final
weights: output/rtdetrv2_r50vd_6x_coco/model_final/rtdetrv2_r50vd_6x_coco.pdparams

重新执行命令，看起来又回到了原先的错误？？？

λ 9686674b44f1 /data/zj/paddle/PaddleDetection python tools/export_model.py -c configs/rtdetrv2/rtdetrv2_r50vd_6x_coco.yml -o weights=https://bj.bcebos.com/v1/paddledet/models/rtdetrv2_r50vd_6x_coco.pdparams trt=True --output_dir=output_inference -o TestReader.inputs_def.image_shape=[1,3,640,640]
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly
[01/14 07:09:56] ppdet.utils.checkpoint INFO: Finish loading model weights: output/rtdetrv2_r50vd_6x_coco/model_final/rtdetrv2_r50vd_6x_coco.pdparams
[01/14 07:09:57] ppdet.data.source.category WARNING: anno_file 'dataset/coco/annotations/instances_val2017.json' is None or not set or not exist, please recheck TrainDataset/EvalDataset/TestDataset.anno_path, otherwise the default categories will be used by metric_type.
[01/14 07:09:57] ppdet.data.source.category WARNING: metric_type: COCO, load default categories of COCO.
[01/14 07:09:57] ppdet.engine INFO: Export inference config file to output_inference/rtdetrv2_r50vd_6x_coco/infer_cfg.yml
Traceback (most recent call last):
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 118, in <module>
    main()
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 114, in main
    run(FLAGS, cfg)
  File "/data/zj/paddle/PaddleDetection/tools/export_model.py", line 80, in run
    trainer.export(FLAGS.output_dir, for_fd=FLAGS.for_fd)
  File "/data/zj/paddle/PaddleDetection/ppdet/engine/trainer.py", line 1282, in export
    static_model, pruned_input_spec = self._get_infer_cfg_and_input_spec(
  File "/data/zj/paddle/PaddleDetection/ppdet/engine/trainer.py", line 1233, in _get_infer_cfg_and_input_spec
    input_spec, static_model.forward.main_program,
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 1062, in main_program
    concrete_program = self.concrete_program
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 941, in concrete_program
    return self.concrete_program_specify_input_spec(input_spec=None)
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 986, in concrete_program_specify_input_spec
    concrete_program, _ = self.get_concrete_program(
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 875, in get_concrete_program
    concrete_program, partial_program_layer = self._program_cache[
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 1648, in __getitem__
    self._caches[item_id] = self._build_once(item)
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 1575, in _build_once
    concrete_program = ConcreteProgram.from_func_spec(
  File "/usr/local/lib/python3.10/dist-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/usr/local/lib/python3.10/dist-packages/paddle/base/wrapped_decorator.py", line 26, in __impl__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/paddle/base/dygraph/base.py", line 68, in __impl__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/program_translator.py", line 1346, in from_func_spec
    error_data.raise_new_exception()
  File "/usr/local/lib/python3.10/dist-packages/paddle/jit/dy2static/error.py", line 452, in raise_new_exception
    raise new_exception from None
AssertionError: In transformed code:

    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 59, in forward
        if self.training:
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 69, in forward
        for inp in inputs_list:
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 76, in forward
        outs.append(self.get_pred())
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/detr.py", line 118, in get_pred
        return self._forward()
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/architectures/detr.py", line 89, in _forward
        out_transformer = self.transformer(body_feats, pad_mask, self.inputs)
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/rtdetr_transformerv2.py", line 480, in forward
        out_bboxes, out_logits = self.decoder(
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/rtdetr_transformer.py", line 235, in forward
        output = layer(output, ref_points_input, memory,
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/rtdetr_transformerv2.py", line 259, in forward
        tgt2 = self.cross_attn(
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/rtdetr_transformerv2.py", line 173, in forward
                format(reference_points.shape[-1]))

        output = self.ms_deformable_attn_core(value, value_spatial_shapes,
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                                              sampling_locations, attention_weights)
        output = self.output_proj(output)

    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/utils.py", line 162, in deformable_attention_core_func_v2
        value_list = [
    File "/data/zj/paddle/PaddleDetection/ppdet/modeling/transformers/utils.py", line 163, in <listcomp>
        value.reshape([batch_num * num_heads, head_dim, h, w])
    File "/usr/local/lib/python3.10/dist-packages/paddle/tensor/manipulation.py", line 4017, in reshape
        attrs["shape"] = get_attr_shape(shape)
    File "/usr/local/lib/python3.10/dist-packages/paddle/tensor/manipulation.py", line 3921, in get_attr_shape
        assert dim_size > 0, (

    AssertionError: Each dimension value of 'shape' in reshape must not be negative except one unknown dimension. But received shape[0] = -8.

复现环境 Environment

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

Bug描述确认 Bug description confirmation

  • 我确认已经提供了Bug复现步骤、代码改动说明、以及环境信息，确认问题是可以复现的。I confirm that the bug replication steps, code change instructions, and environment information have been provided, and the problem can be reproduced.

是否愿意提交PR？ Are you willing to submit a PR?

  • 我愿意提交PR！I'd like to help by submitting a PR!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LokeZhou LokeZhou

@zhangyubo0722 zhangyubo0722

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@LokeZhou @zjykzj @zhangyubo0722