ERROR : OUT OF MEMORY

[Liuyan9627]

Thanks for your awesome work and code!
I tried to run the code you released on the mip-360 dataset:
ns-train block-merf --data /path/to/dataset
while an error happens about out of memory at line 77 of the merf/engine/merf_trainer.py
_, loss_dict, metrics_dict = self.pipeline.get_train_loss_dict(step=step)

I know the scene trained on an A100 GPU, which is capable of training a large scene.
While my graphic is only 24GB, could you please give me some advice to run your code?
Thanks!

[kevin2000A]

You can change the number of blocks in

MERFStudio/blockmerf/block_merf/block_model.py

Line 287 in 9c1446f

num_xy: List[List[int]] = field(default_factory=lambda:[[1,1]])

Or decrease the number of samples per ray in

MERFStudio/blockmerf/block_merf/block_model.py

Line 276 in 9c1446f

num_nerf_samples_per_ray: int = 32

[Liuyan9627]

You can change the number of blocks in

MERFStudio/blockmerf/block_merf/block_model.py

Line 287 in 9c1446f

num_xy: List[List[int]] = field(default_factory=lambda:[[1,1]])

Or decrease the number of samples per ray in

MERFStudio/blockmerf/block_merf/block_model.py

Line 276 in 9c1446f

num_nerf_samples_per_ray: int = 32

I reduced the number of samples per ray to 16， which worked well!

Another question about multiple-GPU training:

I use the customer data, and try the command to train the nerf model via multiple GPU:

ns-train block-merf --data /path/to/data --machine.num-devices 3

I found that each GPU uses the same memory as using a single GPU, and using multiple GPUs (10h) costs more time than using a single GPU (6h).