Possible to train the backbone?

[arjun-kg]

I was just wondering if it's possible to train the backbone as well. In my understanding, when we set pretrained: true, pretrained imagenet weights are loaded in for eg. Resnet. And even if we set pretrained: false, the weights seem to be randomly initialized and frozen, and we get a feature extractor that is not useful unless we separately load other weights offline. Is this accurate?

Is it possible to train the backbone as well while training the anomaly model, i.e., backpropagate all the way through the backbone network during training?

[samet-akcay]

It is possible, but depends on the design of the algorithm. pretrained flag only determines whether to use the pre-trained weights of the backbone.

Some algorithms freeze the backbone and use it as a feature extractor, while others continue to fine-tune the backbone. For example, memory-bank based approaches such as PADIM and Patchcore do not train the backbone.

anomalib/src/anomalib/models/patchcore/torch_model.py

Lines 65 to 66 in 161d66b

	with torch.no_grad():
	features = self.feature_extractor(input_tensor)

Some other models contiue to train the backbone. For example, knowledge distillation models such as student-teacher, train the backbone of the student model further, as shown below:

anomalib/src/anomalib/models/stfpm/torch_model.py

Lines 34 to 41 in 161d66b

	self.teacher_model = FeatureExtractor(backbone=self.backbone, pre_trained=True, layers=layers)
	self.student_model = FeatureExtractor(
	backbone=self.backbone, pre_trained=False, layers=layers, requires_grad=True
	)
	# teacher model is fixed
	for parameters in self.teacher_model.parameters():
	parameters.requires_grad = False

Overall, it is based on the design of the algorithm itself. Let us know if you have any other questions. Thanks!

[arjun-kg]

Thank you for your detailed reply! This means if I use PADIM or PatchCore with pretrained=False, that would just be using a fixed backbone with random weights (which basically creates unusable features)?

And just checking, if the below code means that weights are frozen for Fastflow too?

anomalib/src/anomalib/models/fastflow/torch_model.py

Lines 155 to 156 in 0c7444e

	for parameter in self.feature_extractor.parameters():
	parameter.requires_grad = False

Suppose the pretrained Imagenet features are not good for my specific dataset, what would you recommend? It seems it's not possible to train something like Fastflow end-to-end including the feature extractor.

[samet-akcay]

I'm moving this to the Q&A section in Discussions. We could continue our discussion there.