cross attention issue, topic_12.3. Attention and the Transformer

[omniaalwazzan]

Thanks a lot for this valuable repository

I have a question regarding cross attention presented in the topic 12.3. Attention and the Transformer.

Is this line working properly d_xq, d_xk, d_xv = d_input

In my case, it doesn't work.

Thanks,
Omnia

[Atcold]

What does it mean "it doesn't work"?
Is it issuing some error to you?

[omniaalwazzan]

Thanks a lot for replying @Atcold. I am sorry I was quite busy with some deadlines for my PhD.

Sorry for the confusion, you are right, I should have mentioned the error, so the error is in this line:
d_xq, d_xk, d_xv = d_input TypeError: cannot unpack non-iterable int object

When I pass d_input to use cross-attention.

As far as I understand when we want to initialize or assign a value/integer to multiple variables we need unpacked these values as the int object has no iter method, so maybe to fix that line causing the error, we do this instead:
d_xq, d_xk, d_xv = d_input, d_input, d_input

However, when I do the mentioned solution, I get the same result of self-attention, is this possible? I think the result should be slightly different according to the cross-attention definition. Please feel free to correct me at any point, I have far less experience than you :)

[Atcold]

d_input is a tuple of 3 elements specifying the dimension d for query, key, and values.
This functionality is not used in 15-transformer.ipynb, so I'm confused about where you're getting the error.
Could you explain to me what you're doing so I can help you?