ASomeoneWhoInterestedWithAI/LookThem_V8.5_ImageNet100

LookThem V8.5

I was courious, what if a token look at other tokens, but without QKV? Instead, it's like make transformation for two tokens (current token and another token), then divide them. Let's say current token is token A and another token is token B. It's divide like "transformA(A) / transformB(B)" which the transform is a linear NN. With tanh for normalizing (to make it don't explode). And, the reverse ("transformA(B) / transformB(A)"). Then, the result of "transformA(A) / transformB(B)" multiply with A, and the reverse multiply with B. Then add them, then divide by 2. That's the new number for that interaction. Add to temp variable. Loop again for another token interaction (but for the code it's vectorized). Then, that variable averaged. That's the new A.

Result:

• Training ~18 epochs, train accuracy (with dropout): ~63.29%
• Test accuracy: 53.48%
• The provided model in this repo is the final model before the classifier-only finetune
• Full experiment code provided in this code