Training issue

[suyu-star]

Hi,
Thank you for your great work!

I perform training and validation simultaneously, an error arises in GeometryHistoryFusion.

https://github.com/cdb342/GDFusion/blob/main/mmdet3d/models/alocc/modules/temporal_fusion.py

        if sum(start_of_sequence)>0:
            self.iter[start_of_sequence]=0

        assert (self.history_seq_ids != seq_ids)[~start_of_sequence].sum() == 0, \
            "{}, {}, {}".format(self.history_seq_ids, seq_ids, start_of_sequence)

It is possible that during validation, the batch size (bs) of seq_ids and start_of_sequence is 1, while the leftover historical information has a batch size equal to our configured bs. When these batch sizes do not match, an error occurs. How should this be handled?

[cdb342]

Hi, Thank you for your great work!

I perform training and validation simultaneously, an error arises in GeometryHistoryFusion.

https://github.com/cdb342/GDFusion/blob/main/mmdet3d/models/alocc/modules/temporal_fusion.py

        if sum(start_of_sequence)>0:
            self.iter[start_of_sequence]=0

        assert (self.history_seq_ids != seq_ids)[~start_of_sequence].sum() == 0, \
            "{}, {}, {}".format(self.history_seq_ids, seq_ids, start_of_sequence)

It is possible that during validation, the batch size (bs) of seq_ids and start_of_sequence is 1, while the leftover historical information has a batch size equal to our configured bs. When these batch sizes do not match, an error occurs. How should this be handled?

Hi, I have re-tested the code but could not reproduce this issue. The training bs should not affect the validation phase, as the logic for simultaneous training/validation is consistent with standalone testing. Could you please double-check your configuration or provide more details?

[suyu-star]

Hi, Thank you for your great work!
I perform training and validation simultaneously, an error arises in GeometryHistoryFusion.

https://github.com/cdb342/GDFusion/blob/main/mmdet3d/models/alocc/modules/temporal_fusion.py

        if sum(start_of_sequence)>0:
            self.iter[start_of_sequence]=0

        assert (self.history_seq_ids != seq_ids)[~start_of_sequence].sum() == 0, \
            "{}, {}, {}".format(self.history_seq_ids, seq_ids, start_of_sequence)

It is possible that during validation, the batch size (bs) of seq_ids and start_of_sequence is 1, while the leftover historical information has a batch size equal to our configured bs. When these batch sizes do not match, an error occurs. How should this be handled?

Hi, I have re-tested the code but could not reproduce this issue. The training bs should not affect the validation phase, as the logic for simultaneous training/validation is consistent with standalone testing. Could you please double-check your configuration or provide more details?

Hello, I have reproduced this issue. I found that the code works perfectly when using dist_train.sh, but the error occurs when I run it on a single GPU without distributed training (e.g., using python train.py).

[cdb342]

Hi, Thank you for your great work!
I perform training and validation simultaneously, an error arises in GeometryHistoryFusion.

https://github.com/cdb342/GDFusion/blob/main/mmdet3d/models/alocc/modules/temporal_fusion.py

        if sum(start_of_sequence)>0:
            self.iter[start_of_sequence]=0

        assert (self.history_seq_ids != seq_ids)[~start_of_sequence].sum() == 0, \
            "{}, {}, {}".format(self.history_seq_ids, seq_ids, start_of_sequence)

It is possible that during validation, the batch size (bs) of seq_ids and start_of_sequence is 1, while the leftover historical information has a batch size equal to our configured bs. When these batch sizes do not match, an error occurs. How should this be handled?

Hi, I have re-tested the code but could not reproduce this issue. The training bs should not affect the validation phase, as the logic for simultaneous training/validation is consistent with standalone testing. Could you please double-check your configuration or provide more details?

Hello, I have reproduced this issue. I found that the code works perfectly when using dist_train.sh, but the error occurs when I run it on a single GPU without distributed training (e.g., using python train.py).

The temporal module currently relies on the distributed sampler logic. Please use dist_train.sh as the standard entry point (e.g., set GPUS=1 for single-GPU training).

[suyu-star]

Hi, Thank you for your great work!
I perform training and validation simultaneously, an error arises in GeometryHistoryFusion.

https://github.com/cdb342/GDFusion/blob/main/mmdet3d/models/alocc/modules/temporal_fusion.py

        if sum(start_of_sequence)>0:
            self.iter[start_of_sequence]=0

        assert (self.history_seq_ids != seq_ids)[~start_of_sequence].sum() == 0, \
            "{}, {}, {}".format(self.history_seq_ids, seq_ids, start_of_sequence)

It is possible that during validation, the batch size (bs) of seq_ids and start_of_sequence is 1, while the leftover historical information has a batch size equal to our configured bs. When these batch sizes do not match, an error occurs. How should this be handled?

Hi, I have re-tested the code but could not reproduce this issue. The training bs should not affect the validation phase, as the logic for simultaneous training/validation is consistent with standalone testing. Could you please double-check your configuration or provide more details?

Hello, I have reproduced this issue. I found that the code works perfectly when using dist_train.sh, but the error occurs when I run it on a single GPU without distributed training (e.g., using python train.py).

The temporal module currently relies on the distributed sampler logic. Please use dist_train.sh as the standard entry point (e.g., set GPUS=1 for single-GPU training).

好的，谢谢您