details of n_gpu and batch size

[Leonmel]

Hi @fcakyon, thanks for your sharing this impressive project!

I would like to know the specific settings of BATCH_MULTIPLIER and n_gpus and lr in your configs, and I am confused by the current comments:

BATCH_MULTIPLIER = 16
...
samples_per_gpu=2 * BATCH_MULTIPLIER,
...
# optimizer
# default 8 gpu
# /8 for 1 gpu
optimizer = dict(
    lr=0.01 / 8 * BATCH_MULTIPLIER * LR_MULTIPLIER, paramwise_cfg=dict(bias_lr_mult=2.0, bias_decay_mult=0.0)
)

Is this configuration for single gpu? Do I need to modify BATCH_MULTIPLIER or anything else if I change it to 8 gpus.

It seems like I cannot train with such a large batch size even with 32GB GPU memory.

Hope for your reply.

[fcakyon]

This calculation is for a single GPU. If you don't have enough vram then you should lower the BATCH_MULTIPLIER value and lr will be scaled automatically.

If you will train with more than 1 GPU then multiply with number of GPUs as lr=0.01 / 8 * BATCH_MULTIPLIER * LR_MULTIPLIER * NUM_GPUS

[nalemadi]

Hi @fcakyon, Thanks for sharing the great job.

I'm trying to replicate the training task with xView dataset, but I got a different result. Could you please share the resources' information used in the experiment so that I can figure out what went wrong with my experiment. And, does the difference in resources setup affect the result?

Thanks in advance

[fcakyon]

Hello @nalemadi. Im glad you liked the project. What is your GPU? You can direcly use the pretrained weights published in reqdme if you dont have a GPU with 80gb ram.

[nalemadi]

I'm now using 2 GPUs, each with 32GB.
Yeah, thanks; I used the pretrained weights. But also, I'm trying to replicate the training experiments with xView to achieve your announced result so I can train a subset of xView and another dataset.

[fcakyon]

If your GPU setup is different, results might change (you might have to tune the learning rate and schedule).

[Leonmel]

Thank you for your prompt reply!
btw, could you tell me the meaning of num_cls=60 in TAGS

[fcakyon]

TAGS are optional and can be used while logging to experiment trackers (wandb/neptune/comet).

num_cls=60 means there are 60 classes in the dataset.

[Leonmel]

Thanks a lot

[fcakyon]

@Leonmel thanks for your interest and insightful discussion! Have a great day 💯