Problem with tio.Mask()

[gkrisp98]

Hi, I tried to use the tio.Mask() function but there is a problem as it appears in the picture below with the sagittal and the axial plane. It seems that they are flipped somehow. The coronal plane is ok. Does anybody know why this is happening and how I can fix that

?

[fepegar]

Can you please share code and data to reproduce this?

[gkrisp98]

So this mask is not the mask of this specific MRI and I can get this correct by applying tio.Resample() to the original MRI ? I am a bit confused :/

[fepegar]

Let's say this is MRI A. It looks like someone annotated a different MRI, let's call it B, to obtain segmentation S_B. Therefore, your segmentation is in the space of B. You need to register B to A to obtain the transform T_B→A and apply that transform to S_B to obtain a segmentation that is aligned with A, which you'd need for training.

[gkrisp98]

And how do I register B to A ? Just apply tio.Resample() to MRI A ?
PS: Thanks for your time!

[fepegar]

No worries. It sounds like you need to learn about medical image registration. Here are some resources:

• Image registration - Wikipedia
• Medical image registration: a review - PubMed
• Registration Overview — SimpleITK 2.0rc2 documentation
• Basic Registration - SimpleITK

I typically use NiftyReg for registration, but there are many tools around. For example, FSL's FLIRT.

[gkrisp98]

Thanks a lot.

[gkrisp98]

Hi again, I used Resample() and ToCanonical to set the MRI and the mask to the same spacing and orientation but when I try to use CropOrPad() I get the following error :

More than one value for "direction" found in subject images:
{'seg': (1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0),
 't1': (0.9902680750462728,
        -0.13917306439563465,
        0.0,
        0.13917305609976896,
        0.9902680738803652,
        0.0,
        0.0,
        0.0,
        1.0)}

Do you know how I can fix this ?
The transforms I used were those:

transform = tio.Compose([tio.ToCanonical(),
                         tio.Mask(masking_method = 'seg'),
                         tio.Resample((0.70,0.36,0.36)),
                         tio.RemapLabels({0:0, 1:1, 2:2, 3:1, 4:2}),
                         tio.CropOrPad((128,128,128))
                         ])

[romainVala]

is it on the same data ?
please be more precise on the code example so that we can easily reproduce

[gkrisp98]

So, I declare a subject:

import torchio as tio
a = tio.Subject(
    t1 = tio.ScalarImage('/content/drive/MyDrive/images_nifti_old/patient001.nii.gz'),
    seg = tio.LabelMap('/content/drive/MyDrive/segmentation_masks/patient001.mhd')
)

Then I apply to the subject the transform from the previous answer and I get the error I posted

[romainVala]

Hi

As pointed by fernando, you input image do not match. Although the segmentation may com (I guess) from the mri volume both affine differ (avec the axis order is different too)
A solution that seems to work is :

import torchio as tio

suj = tio.Subject(t1=tio.ScalarImage('patient1.nii.gz'), seg = tio.LabelMap('9001104.segmentation_masks.nii.gz'))

transform_repair_affine =  tio.Compose([tio.ToCanonical(), 
                                         tio.CopyAffine(target='t1')])
suj.t1.save('patient_ok.nii')
suj.seg.save('mask_ok.nii')

note that because the axis ordering is different, it is important to first go to Canonical (same axis order) and then copy the affine
saving the new patient volume and mask, they now do superpose properly (I test with my favorite viewer, mrview (from mrtrix)

I then try this full transform

transform = tio.Compose([tio.ToCanonical(), 
                                         tio.CopyAffine(target='t1'),
                                         tio.Resample((0.70,0.36,0.36)),
                                         tio.RemapLabels({0:0, 1:1, 2:2, 3:1, 4:2}),
                                         tio.CropOrPad((128,128,128))  ])

and did not get any error

[gkrisp98]

Yeap that seems to work! Is it necessary to first save the new patient and volume mask ? Because I only applied the transform that you proposed and it worked (note: I want to create a subjects dataset and classify the masked mris)

Thanks a lot for sharing your knowledge!!

[romainVala]

no it is not necessary, the save was just for checking in a external viewer how was the overlay

[gkrisp98]

Thanks a lot again!!

[gkrisp98]

I am currently facing another problem and I was wondering if you could help me again
I create a SubjectsDataset from the subjects on which I applied the transforms above and then I feed them to a model
Some batches pass normally but at others it shows this error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-56-2a7d5ddf84fc> in <module>()
     26 for epoch in range(1, num_epochs+1):
     27 
---> 28     train_loss, train_accuracy = training(epoch,model,train_loader,optimizer,criterion)
     29     valid_loss, valid_accuracy = validation(epoch,model,valid_loader,criterion)
     30 

6 frames
<ipython-input-55-59a55a26f2b7> in training(epoch, model, train_loader, optimizer, criterion)
      8   model.train()
      9 
---> 10   for i, batch in tqdm(enumerate(train_loader,0)):
     11     images = batch['t1'][tio.DATA].cuda()
     12     labels = batch['label']

/usr/local/lib/python3.7/dist-packages/tqdm/notebook.py in __iter__(self)
    255     def __iter__(self):
    256         try:
--> 257             for obj in super(tqdm_notebook, self).__iter__():
    258                 # return super(tqdm...) will not catch exception
    259                 yield obj

/usr/local/lib/python3.7/dist-packages/tqdm/std.py in __iter__(self)
   1193 
   1194         try:
-> 1195             for obj in iterable:
   1196                 yield obj
   1197                 # Update and possibly print the progressbar.

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
   1201             else:
   1202                 del self._task_info[idx]
-> 1203                 return self._process_data(data)
   1204 
   1205     def _try_put_index(self):

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _process_data(self, data)
   1227         self._try_put_index()
   1228         if isinstance(data, ExceptionWrapper):
-> 1229             data.reraise()
   1230         return data
   1231 

/usr/local/lib/python3.7/dist-packages/torch/_utils.py in reraise(self)
    432             # instantiate since we don't know how to
    433             raise RuntimeError(msg) from None
--> 434         raise exception
    435 
    436 

RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/collate.py", line 74, in <dictcomp>
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/collate.py", line 74, in <dictcomp>
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: result type Float can't be cast to the desired output type Byte

The most confusing part is that the error dissapears when I set the batch_size = 1 and then all mris pass through the network normally but it seems the model does not fit to the training data.
This is the training function I used:

def training(epoch, model, train_loader, optimizer, criterion):
  "Training over an epoch"
  metric_monitor = MetricMonitor()
  model.train()

  for i, batch in tqdm(enumerate(train_loader,0)):
    images = batch['t1'][tio.DATA].cuda()
    labels = batch['label'].cuda()
    if images.sum() != 0:
      output = F.softmax(model(images), dim =1)
  
      loss = criterion(output, labels)

      output = output.data.max(dim=1,keepdim=True)[1]
      output = output.view(-1)
      
      acc = calculate_acc(output, labels)
      
      metric_monitor.update("Loss", loss.item())
      metric_monitor.update("Accuracy", acc)

      optimizer.zero_grad()

      loss.backward()
      optimizer.step()
  print("[Epoch: {epoch:03d}] Train      | {metric_monitor}".format(epoch=epoch, metric_monitor=metric_monitor))
  return metric_monitor.metrics['Loss']['avg'], metric_monitor.metrics['Accuracy']['avg']

[romainVala]

Hi
sorry, but it is difficult to guess what is going on. If you can come on with a reproducible working example ...
it seems you have an issue when concatening different subject into a batch (collate function), so it all depends what data you try to concatenate ...

[fepegar]

Please open a new issue and follow the instructions in the template. This discussion is about a problem with Mask.