What is permuter and why we need permuter

[vivian556123]

Hi, thanks for your great work. I have a question about the permuter (SpecVQGAN/specvqgan/modules/transformer/permuter.py), why we need the ColumnMajor(AbstractPermuter), what is the advantage of applying this function? is that possible to not use it? Since I think the ColumnMajor(AbstractPermuter) can only process the fixed-length mel spectrogram. If there is a longer mel spectrogram of a shorter mel spectrogram, it will raise an error (asset failure for the first branch L>Lidx or the index is out of bound for the else branch)

[v-iashin]

Hi, thanks, I am glad you like it.

The purpose of the Permuter is to ensure that the transformer code prediction order is done correctly on spectrograms (F, T). In particular, we want to generate codebook codes as follows: first, from higher to lower frequencies F, then, from earlier to later T -- aka. column-major.

why we need the ColumnMajor(AbstractPermuter)

It does a customized-order .view() operation. The .view() makes row-major ordering:

[1, 2, 3, 4, 5, 6].view(2, 3)
will give you a matrix
[1, 2, 3]
[4, 5, 6]

and .view(-1) on the 2D matrix will give you [1, 2, 3, ..., 6]

However, for spectrograms, we want to sample codes in column-major order:

permuter([1, 2, 3, 4, 5, 6], reverse=False)
will give you a matrix
[1, 3, 5]
[2, 4, 6]

and `reverse=True` 
will flatten the 2D matrix to [1, 2, 3, ..., 6] and not to [1, 3, 5, 2, 4, 6]

what is the advantage of applying this function
One could perhaps transpose the spectrogram and use .view(). However, having this abstraction allows the implementation of other types of code ordering as in 2012.09841, Figure 47, without changing the implementation of sampling, and only changing the permute module initialization in the configs. This code is inherited from the VQGAN.

is that possible to not use it?
To make sampling column-major instead of row-major, you need to either replace the function with a version you are comfortable with or use it as it is. If you will try to train a model in a row-major order, you will give it information "from the future" and train it to generate other frequencies given the higher ones. We don't want it. We want to train to generate temporally new content.

can only process the fixed-length mel spectrogram
It can process any length of the input unless it is not the multiple of the number of elements in a row. The code ensures that you sample full columns:

SpecVQGAN/specvqgan/modules/transformer/permuter.py

Lines 34 to 44 in 5bc54f3

	if L > L_idx:
	# an ugly patch for "infinite" sampling because self.*_shuffle_idx are shorter
	# otherwise even uglier patch in other places. 'if' is triggered only on sampling.
	assert L % L_idx == 0 and L / L_idx == int(L / L_idx), f'L: {L}, L_idx: {L_idx}'
	W_scale = L // L_idx
	print(f'Permuter is making a guess on the temp scale: {W_scale}. Ignore on "infinite" sampling')
	idx = self.make_idx(self.H, self.W * W_scale)
	if not reverse:
	return x[:, idx]
	else:
	return x[:, torch.argsort(idx)]

.
If it throws you an error, it means you give it a sequence which cannot be permuted into a 2D matrix. You may check your L and you will notice that it is not a multiple of the number of elements in a row.

[vivian556123]

Thanks for your reply. But I am a little bit confused. The assert L % L_idx == 0 and L / L_idx == int(L / L_idx), f'L: {L}, L_idx: {L_idx}' need the L(length of original mel 1d sequence, in my case is 245 or 270 or 255 etc) and L_idx(553=265), it raise error. However, 245=549, 270=554, 255=551, which are the multiple of 5(number of rows). Besides, the if L > L_idx: only works in the case of longer sequence? What about the shorter sequences? Thanks a lot!

[v-iashin]

Yes, you are right sorry (I wrote it a long time ago). I think this chunk ensures that if the W_scale is an integer, it will have enough indices to permute the incoming tensors (e.g. for the "longest drum demo"). Anyway, one could adapt the code to serve tensors of any length which is a multiple of 5 or, perhaps, just transpose the tensor and view it appropriately.