Skip to content

Train Log file is missing information show at the console  #658

Closed
Project-MONAI/MONAI
#8142
@AHarouni

Description

@AHarouni

Describe the bug
Log files generated doesn't include all lines shown on the console

To Reproduce
I testing this on vista3d and whole body bundle. Run a training job see output on the console showing lines as

2024-09-14 03:19:12,752 - INFO - Epoch: 881/900, Iter: 24/25 -- train_loss: 0.4078 
2024-09-14 03:19:13,500 - INFO - Epoch: 881/900, Iter: 25/25 -- train_loss: 0.4699 
2024-09-14 03:19:13,500 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[881] Complete. Time taken: 00:00:19.345
2024-09-14 03:19:14,636 - INFO - Epoch: 882/900, Iter: 1/25 -- train_loss: 0.3655 
2024-09-14 03:19:15,386 - INFO - Epoch: 882/900, Iter: 2/25 -- train_loss: 0.3317 
2024-09-14 03:19:16,135 - INFO - Epoch: 882/900, Iter: 3/25 -- train_loss: 0.3545 
2024-09-14 03:19:16,884 - INFO - Epoch: 882/900, Iter: 4/25 -- train_loss: 0.2083 
2024-09-14 03:19:17,633 - INFO - Epoch: 882/900, Iter: 5/25 -- train_loss: 0.2773 
....
2024-09-14 03:19:31,124 - INFO - Epoch: 882/900, Iter: 23/25 -- train_loss: 0.2539 
2024-09-14 03:19:31,874 - INFO - Epoch: 882/900, Iter: 24/25 -- train_loss: 0.1814 
2024-09-14 03:19:32,622 - INFO - Epoch: 882/900, Iter: 25/25 -- train_loss: 0.2402 
2024-09-14 03:19:32,622 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[882] Complete. Time taken: 00:00:19.122
2024-09-14 03:19:33,923 - INFO - Epoch: 883/900, Iter: 1/25 -- train_loss: 0.2866 

however the log file will show only


2024-09-14 03:19:13,500 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[881] Complete. Time taken: 00:00:19.345
2024-09-14 03:19:32,622 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[882] Complete. Time taken: 00:00:19.122

Expected behavior

The file log should show exactly the same as what is on the console.

Additional context

I have tried changing the loggin.config file but it seems the monai bundle is loading logging config from else where.

Activity

Nic-Ma

Nic-Ma commented on Oct 11, 2024

@Nic-Ma
Collaborator

Hi @AHarouni ,

May I know how do you collect the log in file? Do you have a logging config, maybe the logging level is not correct?
Have you seen this log message in your side:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/bundle/workflows.py#L319

Thanks.

KumoLiu

KumoLiu commented on Oct 11, 2024

@KumoLiu
Collaborator

Hi @AHarouni, @Nic-Ma, try to fix the issue in Project-MONAI/MONAI#8142
After the pr merge, users can simply add a FileHandler in the logging.conf to log all the things in the console.

...
[handlers]
keys=consoleHandler, fileHandler

[logger_root]
level=INFO
handlers=consoleHandler, fileHandler

[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=fullFormatter
args=('./train.log', 'a')
KumoLiu

KumoLiu commented on Oct 29, 2024

@KumoLiu
Collaborator

To enable logging output from MONAI, users can modify the logging.conf file as follows:
Add a new logger named monai and attach the filehandler to it, set propagate=0 to prevent duplicate logging..

[loggers]
keys=root, monai

[logger_monai]
level=INFO
handlers=consoleHandler, fileHandler
qualname=monai
propagate=0

cc @AHarouni

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Participants

    @Nic-Ma@KumoLiu@AHarouni

    Issue actions

      Train Log file is missing information show at the console · Issue #658 · Project-MONAI/model-zoo