Skip to content

pointlander/lemma

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
LICENSE
LICENSE
 
 
README.md
README.md
 
 
go.mod
go.mod
 
 
go.sum
go.sum
 
 
iris.zip
iris.zip
 
 
main.go
main.go
 
 

Repository files navigation

Self attention based computation of principal eigenvector under certain conditions

Abstract

This code generates the principal eigenvector using self attention under the conditions:

  • The input vectors are positive.
  • No input transforms are used.

The self attention based eigenvectors do not have the same magnitude as the principal eigenvector, only the same angle. The angle between the vectors is smaller when softmax isn't used. See code for details.

Algorithm

The iris dataset is used alongside randomly generated datasets. For each dataset the adjacency matrix is computed. The eigenvectors and eigenvalues of the adjacency matrix are computed. To compute self attention, the adjacency matrix and the softmax of the adjacency matrix are multiplied by the input dataset. The cosine similarity is then computed between the eigenvectors and the vectors computed with self attention.

Results

Summary

  • Principal eigenvector vs self attention 1st vector (works)
  • Principal eigenvector vs self attention 2nd vector (works)
  • Non-Principal eigenvector vs self attention 1st vector (doesn't work)

The first entry in each table is for the iris dataset. The rest of the entries are randomly generated datasets.

Principal eigenvector vs self attention 1st vector

eigenvalue mag eigenvector mag self attention cosine similarity (with softmax)
9206.530596 1.000000 95.267962 0.979676
153.035448 1.000000 6.677844 0.967490
146.985750 1.000000 6.469487 0.967032
160.615877 1.000000 6.880530 0.965821
145.946463 1.000000 6.241726 0.963127
165.390938 1.000000 6.646201 0.971719
168.939761 1.000000 6.637303 0.967610
173.455692 1.000000 7.324602 0.964891
164.364278 1.000000 7.093303 0.965082
170.809366 1.000000 6.815885 0.964670
168.725411 1.000000 6.963146 0.971783
162.228956 1.000000 6.471354 0.957326
152.398589 1.000000 6.420290 0.964964
152.788177 1.000000 6.011698 0.961383
159.292100 1.000000 6.641088 0.961978
164.603275 1.000000 6.781595 0.970627
165.112983 1.000000 7.025318 0.963097
165.635110 1.000000 6.455346 0.970197
157.807981 1.000000 6.499539 0.966751
177.225237 1.000000 6.971489 0.970760
158.409995 1.000000 6.420343 0.964178
160.217627 1.000000 6.233390 0.964693
170.408037 1.000000 7.133776 0.963515
168.460888 1.000000 6.594924 0.970950
153.034498 1.000000 6.524873 0.967838
169.224380 1.000000 6.500595 0.966955
174.409270 1.000000 7.013595 0.963185
153.403013 1.000000 6.277223 0.969020
157.568193 1.000000 6.493295 0.972137
155.412338 1.000000 6.163024 0.967136
162.250681 1.000000 6.594340 0.967418
162.858646 1.000000 6.511601 0.969240
148.066191 1.000000 6.250871 0.967192
161.412856 1.000000 6.308377 0.965175
154.030834 1.000000 6.039407 0.961977
174.718474 1.000000 6.969898 0.968564
165.243244 1.000000 6.815741 0.963750
163.703212 1.000000 6.490107 0.965378
168.063493 1.000000 6.475724 0.971855
167.906161 1.000000 6.497066 0.960129
167.534931 1.000000 6.784336 0.965059
160.578051 1.000000 6.815965 0.965940
164.235436 1.000000 6.536982 0.967962
155.008998 1.000000 6.548110 0.962167
168.404630 1.000000 6.042017 0.963351
158.640308 1.000000 6.138811 0.962273
169.846162 1.000000 7.062570 0.963550
166.141267 1.000000 6.706509 0.966519
154.979091 1.000000 6.249225 0.965967
160.257850 1.000000 6.502338 0.962483
161.575667 1.000000 6.861466 0.970643
163.650667 1.000000 7.000538 0.961950
163.903141 1.000000 6.550479 0.965754
163.390138 1.000000 7.016451 0.967132
153.517123 1.000000 6.939118 0.964153
163.341905 1.000000 6.181780 0.967710
163.202602 1.000000 6.498106 0.969162
157.279692 1.000000 6.649323 0.961826
157.167207 1.000000 6.459216 0.967796
164.519616 1.000000 6.337793 0.965765
172.830553 1.000000 6.946143 0.966555
157.045462 1.000000 6.116538 0.958633
158.939444 1.000000 6.169372 0.966601
166.080571 1.000000 6.873749 0.968836
155.330333 1.000000 6.276361 0.965957
157.490737 1.000000 6.500678 0.959955
169.363566 1.000000 6.415871 0.964399
162.087452 1.000000 6.448137 0.963823
166.146630 1.000000 6.409095 0.962624
152.824261 1.000000 6.379470 0.968502
146.829732 1.000000 6.456515 0.964139
168.877339 1.000000 6.621731 0.962982
168.070651 1.000000 6.735037 0.971279
160.946507 1.000000 6.945548 0.966535
167.129891 1.000000 6.698101 0.962313
169.302965 1.000000 6.386418 0.971382
162.711044 1.000000 6.820714 0.961286
171.184705 1.000000 6.479948 0.972740
163.758445 1.000000 6.786087 0.970029
157.228066 1.000000 6.897502 0.965146
168.130405 1.000000 7.153319 0.965005
166.130041 1.000000 6.869857 0.964518
175.154889 1.000000 7.173491 0.967136
160.073787 1.000000 6.545292 0.965389
152.161636 1.000000 6.392094 0.963388
164.625480 1.000000 6.573605 0.958175
159.881383 1.000000 6.868605 0.961161
156.692569 1.000000 6.437777 0.967282
158.534200 1.000000 6.013324 0.966350
179.911712 1.000000 7.174269 0.967939
158.838348 1.000000 5.587373 0.960218
162.954921 1.000000 7.203439 0.964688
161.096447 1.000000 6.860645 0.963565
156.183857 1.000000 6.461804 0.969288
169.187674 1.000000 6.765574 0.968920
151.841936 1.000000 5.958579 0.962246
163.061099 1.000000 6.349641 0.965186
150.880346 1.000000 6.106754 0.966595
161.731835 1.000000 6.297961 0.968711
174.998578 1.000000 6.804727 0.967600
168.526015 1.000000 6.916555 0.972225
162.954771 1.000000 6.502600 0.965421
169.277889 1.000000 6.708884 0.964870
168.228308 1.000000 6.492005 0.967914
159.667011 1.000000 7.356210 0.966465
150.853417 1.000000 6.376264 0.961893
156.269186 1.000000 6.799447 0.961121
173.503051 1.000000 6.649155 0.970533
168.339646 1.000000 6.439181 0.968147
155.678395 1.000000 6.491497 0.961954
159.265431 1.000000 6.530884 0.967143
168.691649 1.000000 6.532615 0.967979
161.585337 1.000000 6.650253 0.966058
165.321779 1.000000 6.535029 0.963632
155.662681 1.000000 6.734863 0.964913
176.757512 1.000000 6.260079 0.968214
177.075374 1.000000 6.768350 0.966107
150.342079 1.000000 5.996217 0.967129
167.786899 1.000000 6.814211 0.962453
158.571917 1.000000 6.793648 0.961036
172.169312 1.000000 7.102100 0.962579
155.700443 1.000000 6.887268 0.964365
155.115318 1.000000 6.317686 0.964678
159.181861 1.000000 6.494668 0.969242
160.698527 1.000000 6.389890 0.967635
165.051598 1.000000 6.528850 0.966169
160.228322 1.000000 6.611481 0.967602
141.442469 1.000000 6.018621 0.959190
161.683492 1.000000 6.363545 0.963881
eigenvalue mag eigenvector mag self attention cosine similarity (without softmax)
9206.530596 1.000000 663563.299286 0.999997
153.035448 1.000000 986.932529 0.999315
146.985750 1.000000 921.390981 0.999603
160.615877 1.000000 1062.004238 0.999347
145.946463 1.000000 881.471524 0.998908
165.390938 1.000000 1060.110611 0.999368
168.939761 1.000000 1076.486190 0.999434
173.455692 1.000000 1217.497038 0.999531
164.364278 1.000000 1122.958085 0.999475
170.809366 1.000000 1117.169061 0.999353
168.725411 1.000000 1130.548081 0.999492
162.228956 1.000000 1004.520971 0.998330
152.398589 1.000000 942.872802 0.999132
152.788177 1.000000 884.139813 0.999482
159.292100 1.000000 1015.351799 0.999181
164.603275 1.000000 1078.569778 0.999247
165.112983 1.000000 1107.108619 0.999607
165.635110 1.000000 1034.053413 0.999077
157.807981 1.000000 989.866195 0.999169
177.225237 1.000000 1191.653017 0.999147
158.409995 1.000000 981.265974 0.998862
160.217627 1.000000 963.461947 0.999021
170.408037 1.000000 1161.864926 0.999457
168.460888 1.000000 1068.981161 0.999337
153.034498 1.000000 961.107531 0.999387
169.224380 1.000000 1055.491698 0.999409
174.409270 1.000000 1169.397747 0.999559
153.403013 1.000000 932.864167 0.999047
157.568193 1.000000 989.713896 0.999436
155.412338 1.000000 923.322647 0.999214
162.250681 1.000000 1028.272934 0.998982
162.858646 1.000000 1026.139767 0.999016
148.066191 1.000000 895.530530 0.998744
161.412856 1.000000 977.204049 0.998893
154.030834 1.000000 897.585165 0.998908
174.718474 1.000000 1166.419430 0.999318
165.243244 1.000000 1079.371510 0.999283
163.703212 1.000000 1019.041048 0.999333
168.063493 1.000000 1054.097491 0.999116
167.906161 1.000000 1042.329824 0.999138
167.534931 1.000000 1088.410514 0.999461
160.578051 1.000000 1054.150942 0.999504
164.235436 1.000000 1032.632653 0.999083
155.008998 1.000000 973.848494 0.999164
168.404630 1.000000 969.162695 0.998880
158.640308 1.000000 932.434099 0.999005
169.846162 1.000000 1142.183379 0.999762
166.141267 1.000000 1068.792570 0.999324
154.979091 1.000000 930.404063 0.999272
160.257850 1.000000 993.183997 0.999284
161.575667 1.000000 1073.839519 0.998952
163.650667 1.000000 1098.034117 0.999714
163.903141 1.000000 1033.038479 0.998705
163.390138 1.000000 1104.663242 0.999437
153.517123 1.000000 1029.581357 0.999417
163.341905 1.000000 972.653563 0.998229
163.202602 1.000000 1023.257487 0.999532
157.279692 1.000000 999.641199 0.999220
157.167207 1.000000 980.416235 0.999180
164.519616 1.000000 1000.514799 0.999327
172.830553 1.000000 1151.895329 0.999575
157.045462 1.000000 915.874324 0.998832
158.939444 1.000000 944.702202 0.998937
166.080571 1.000000 1096.793760 0.999513
155.330333 1.000000 940.219388 0.999416
157.490737 1.000000 979.557709 0.999411
169.363566 1.000000 1040.782782 0.999386
162.087452 1.000000 1001.808777 0.998964
166.146630 1.000000 1021.310957 0.999389
152.824261 1.000000 936.832629 0.998848
146.829732 1.000000 911.813623 0.998692
168.877339 1.000000 1067.384958 0.999622
168.070651 1.000000 1091.169738 0.999083
160.946507 1.000000 1076.632457 0.999656
167.129891 1.000000 1068.878228 0.999293
169.302965 1.000000 1043.349807 0.999226
162.711044 1.000000 1064.869687 0.999423
171.184705 1.000000 1070.166797 0.999216
163.758445 1.000000 1072.393726 0.999408
157.228066 1.000000 1046.396507 0.999302
168.130405 1.000000 1147.423876 0.999691
166.130041 1.000000 1098.639801 0.999291
175.154889 1.000000 1198.024532 0.999477
160.073787 1.000000 1002.511192 0.999387
152.161636 1.000000 940.430782 0.999528
164.625480 1.000000 1032.317496 0.999344
159.881383 1.000000 1054.191665 0.998984
156.692569 1.000000 975.302779 0.999110
158.534200 1.000000 915.931027 0.998167
179.911712 1.000000 1232.841465 0.999751
158.838348 1.000000 848.501567 0.998799
162.954921 1.000000 1126.246139 0.999698
161.096447 1.000000 1062.143340 0.999456
156.183857 1.000000 974.546894 0.999037
169.187674 1.000000 1100.147076 0.999688
151.841936 1.000000 869.090949 0.998766
163.061099 1.000000 995.329331 0.999024
150.880346 1.000000 888.211331 0.999080
161.731835 1.000000 984.603202 0.999145
174.998578 1.000000 1135.187239 0.999652
168.526015 1.000000 1126.843098 0.999610
162.954771 1.000000 1017.808123 0.999334
169.277889 1.000000 1086.866835 0.999141
168.228308 1.000000 1049.004683 0.999157
159.667011 1.000000 1132.786460 0.999613
150.853417 1.000000 924.619091 0.999382
156.269186 1.000000 1018.898690 0.998538
173.503051 1.000000 1105.225452 0.999182
168.339646 1.000000 1042.579505 0.999230
155.678395 1.000000 969.937637 0.999090
159.265431 1.000000 999.033539 0.999304
168.691649 1.000000 1056.922474 0.998963
161.585337 1.000000 1032.916585 0.999591
165.321779 1.000000 1030.084105 0.999210
155.662681 1.000000 1006.992628 0.999525
176.757512 1.000000 1060.100038 0.999358
177.075374 1.000000 1140.121328 0.999639
150.342079 1.000000 870.263814 0.998531
167.786899 1.000000 1089.167103 0.999548
158.571917 1.000000 1027.084912 0.999306
172.169312 1.000000 1164.404668 0.999624
155.700443 1.000000 1032.449923 0.999476
155.115318 1.000000 946.423578 0.999128
159.181861 1.000000 996.326855 0.999385
160.698527 1.000000 987.887217 0.999378
165.051598 1.000000 1034.203451 0.999606
160.228322 1.000000 1025.394188 0.999493
141.442469 1.000000 823.310567 0.999125
161.683492 1.000000 992.011656 0.999337 
0/129 outside of cosine similarity of .95 (with softmax)
0/129 outside of cosine similarity of .99 (without softmax)

Principal eigenvector vs self attention 2nd vector

eigenvalue mag eigenvector mag self attention cosine similarity (with softmax)
9206.530596 1.000000 45.431212 0.979031
153.035448 1.000000 6.643875 0.963647
146.985750 1.000000 6.292347 0.964310
160.615877 1.000000 6.846133 0.968209
145.946463 1.000000 5.898011 0.966736
165.390938 1.000000 6.625122 0.969893
168.939761 1.000000 7.179233 0.967478
173.455692 1.000000 6.605861 0.964463
164.364278 1.000000 6.458686 0.961065
170.809366 1.000000 7.232203 0.968815
168.725411 1.000000 6.270382 0.971171
162.228956 1.000000 6.937579 0.963713
152.398589 1.000000 6.186679 0.963130
152.788177 1.000000 6.830993 0.959128
159.292100 1.000000 6.668894 0.962029
164.603275 1.000000 6.983264 0.974073
165.112983 1.000000 6.419021 0.959302
165.635110 1.000000 6.784116 0.968903
157.807981 1.000000 6.684434 0.965812
177.225237 1.000000 6.934166 0.975708
158.409995 1.000000 6.819193 0.965502
160.217627 1.000000 6.761914 0.967685
170.408037 1.000000 6.200721 0.965801
168.460888 1.000000 7.008840 0.969861
153.034498 1.000000 6.745557 0.968670
169.224380 1.000000 6.490785 0.967896
174.409270 1.000000 6.672823 0.962002
153.403013 1.000000 6.425096 0.969844
157.568193 1.000000 6.536778 0.972215
155.412338 1.000000 6.933658 0.968430
162.250681 1.000000 6.753185 0.966531
162.858646 1.000000 6.396478 0.972689
148.066191 1.000000 6.407398 0.966365
161.412856 1.000000 6.865629 0.966142
154.030834 1.000000 6.810135 0.961072
174.718474 1.000000 6.832167 0.969851
165.243244 1.000000 6.556012 0.965164
163.703212 1.000000 6.384995 0.967731
168.063493 1.000000 7.314305 0.975042
167.906161 1.000000 6.461110 0.964899
167.534931 1.000000 7.169000 0.964686
160.578051 1.000000 7.116794 0.965054
164.235436 1.000000 7.070761 0.967734
155.008998 1.000000 6.605111 0.962232
168.404630 1.000000 7.240202 0.968440
158.640308 1.000000 6.097162 0.963838
169.846162 1.000000 6.663648 0.958705
166.141267 1.000000 6.715065 0.967781
154.979091 1.000000 6.587455 0.964150
160.257850 1.000000 6.832932 0.960118
161.575667 1.000000 6.082574 0.970461
163.650667 1.000000 6.827709 0.960974
163.903141 1.000000 6.552121 0.968834
163.390138 1.000000 6.458401 0.966191
153.517123 1.000000 6.355234 0.961995
163.341905 1.000000 6.895637 0.973309
163.202602 1.000000 6.311523 0.967069
157.279692 1.000000 7.232584 0.961731
157.167207 1.000000 6.791197 0.968367
164.519616 1.000000 7.036681 0.966046
172.830553 1.000000 6.814535 0.967312
157.045462 1.000000 6.336946 0.961892
158.939444 1.000000 7.169263 0.966185
166.080571 1.000000 6.496220 0.965227
155.330333 1.000000 6.867713 0.964519
157.490737 1.000000 6.576886 0.962501
169.363566 1.000000 6.950175 0.962902
162.087452 1.000000 6.625318 0.965546
166.146630 1.000000 6.804640 0.959658
152.824261 1.000000 6.207275 0.965771
146.829732 1.000000 6.504319 0.968329
168.877339 1.000000 7.058063 0.961673
168.070651 1.000000 6.978154 0.972926
160.946507 1.000000 6.660465 0.966669
167.129891 1.000000 6.631728 0.963092
169.302965 1.000000 6.802213 0.969989
162.711044 1.000000 6.748184 0.958732
171.184705 1.000000 7.095008 0.972894
163.758445 1.000000 6.422240 0.968471
157.228066 1.000000 5.967757 0.961971
168.130405 1.000000 6.724928 0.963659
166.130041 1.000000 6.266466 0.967390
175.154889 1.000000 7.265977 0.964555
160.073787 1.000000 6.353818 0.963119
152.161636 1.000000 6.705373 0.965946
164.625480 1.000000 7.109392 0.961951
159.881383 1.000000 6.996777 0.962087
156.692569 1.000000 6.410235 0.967659
158.534200 1.000000 6.525644 0.970323
179.911712 1.000000 6.939200 0.965738
158.838348 1.000000 6.907806 0.960267
162.954921 1.000000 6.042676 0.962306
161.096447 1.000000 6.356647 0.967446
156.183857 1.000000 6.673931 0.968644
169.187674 1.000000 6.589015 0.969718
151.841936 1.000000 6.451470 0.965447
163.061099 1.000000 6.504542 0.966156
150.880346 1.000000 6.735311 0.965319
161.731835 1.000000 6.782007 0.968079
174.998578 1.000000 6.914712 0.967057
168.526015 1.000000 6.809653 0.970070
162.954771 1.000000 6.811249 0.961744
169.277889 1.000000 6.571980 0.964043
168.228308 1.000000 7.093452 0.966385
159.667011 1.000000 6.153428 0.962081
150.853417 1.000000 6.358823 0.960819
156.269186 1.000000 6.316291 0.967301
173.503051 1.000000 7.213451 0.969577
168.339646 1.000000 6.544971 0.965318
155.678395 1.000000 6.178977 0.962322
159.265431 1.000000 6.440922 0.966130
168.691649 1.000000 6.662432 0.964060
161.585337 1.000000 6.753415 0.964149
165.321779 1.000000 6.847173 0.960586
155.662681 1.000000 6.400734 0.964781
176.757512 1.000000 6.983299 0.969690
177.075374 1.000000 7.042213 0.961593
150.342079 1.000000 6.357507 0.969735
167.786899 1.000000 6.487679 0.962533
158.571917 1.000000 6.595534 0.961135
172.169312 1.000000 6.960459 0.961366
155.700443 1.000000 6.375258 0.961260
155.115318 1.000000 6.917587 0.965207
159.181861 1.000000 6.612921 0.963192
160.698527 1.000000 7.253825 0.968740
165.051598 1.000000 6.631770 0.968471
160.228322 1.000000 6.754440 0.965282
141.442469 1.000000 6.215643 0.961827
161.683492 1.000000 6.431265 0.967939
eigenvalue mag eigenvector mag self attention cosine similarity (without softmax)
9206.530596 1.000000 335508.409704 0.999959
153.035448 1.000000 980.576950 0.999218
146.985750 1.000000 893.156364 0.999037
160.615877 1.000000 1059.140536 0.999684
145.946463 1.000000 835.240866 0.999063
165.390938 1.000000 1056.013472 0.999341
168.939761 1.000000 1167.695788 0.999439
173.455692 1.000000 1100.162241 0.999209
164.364278 1.000000 1013.285135 0.999220
170.809366 1.000000 1184.056074 0.999533
168.725411 1.000000 1017.714913 0.999324
162.228956 1.000000 1077.815806 0.999588
152.398589 1.000000 905.883228 0.998999
152.788177 1.000000 1000.714900 0.999299
159.292100 1.000000 1016.551882 0.999486
164.603275 1.000000 1110.016882 0.999268
165.112983 1.000000 1007.651938 0.999207
165.635110 1.000000 1087.665259 0.999417
157.807981 1.000000 1015.792448 0.999045
177.225237 1.000000 1183.069916 0.999440
158.409995 1.000000 1041.341783 0.999103
160.217627 1.000000 1045.492795 0.999375
170.408037 1.000000 1006.962115 0.999552
168.460888 1.000000 1139.796276 0.999467
153.034498 1.000000 998.866526 0.999187
169.224380 1.000000 1056.073283 0.999463
174.409270 1.000000 1113.155996 0.999422
153.403013 1.000000 959.709497 0.999186
157.568193 1.000000 994.261380 0.999348
155.412338 1.000000 1039.196926 0.999620
162.250681 1.000000 1052.313715 0.999064
162.858646 1.000000 1009.010381 0.999345
148.066191 1.000000 918.977593 0.999327
161.412856 1.000000 1065.250962 0.999232
154.030834 1.000000 1006.422552 0.999484
174.718474 1.000000 1144.160700 0.999451
165.243244 1.000000 1036.244052 0.999196
163.703212 1.000000 1002.230678 0.999131
168.063493 1.000000 1194.027092 0.999732
167.906161 1.000000 1037.117655 0.999291
167.534931 1.000000 1153.219993 0.999348
160.578051 1.000000 1099.115470 0.999620
164.235436 1.000000 1119.036819 0.999459
155.008998 1.000000 983.492939 0.999347
168.404630 1.000000 1173.490961 0.999621
158.640308 1.000000 929.069986 0.998900
169.846162 1.000000 1073.715782 0.999079
166.141267 1.000000 1073.629802 0.999427
154.979091 1.000000 982.535438 0.999386
160.257850 1.000000 1044.637846 0.999435
161.575667 1.000000 950.606726 0.999074
163.650667 1.000000 1071.597437 0.999612
163.903141 1.000000 1036.558550 0.999079
163.390138 1.000000 1015.912389 0.999246
153.517123 1.000000 938.737242 0.999245
163.341905 1.000000 1087.966424 0.999131
163.202602 1.000000 990.607248 0.999282
157.279692 1.000000 1092.031681 0.999304
157.167207 1.000000 1031.914173 0.999375
164.519616 1.000000 1116.878004 0.999525
172.830553 1.000000 1127.607220 0.999212
157.045462 1.000000 949.857215 0.999393
158.939444 1.000000 1097.207891 0.999530
166.080571 1.000000 1032.473234 0.999307
155.330333 1.000000 1026.918002 0.999654
157.490737 1.000000 995.761941 0.999186
169.363566 1.000000 1126.070143 0.999389
162.087452 1.000000 1031.539309 0.999257
166.146630 1.000000 1082.673834 0.999461
152.824261 1.000000 910.017183 0.999206
146.829732 1.000000 924.393371 0.999365
168.877339 1.000000 1137.035860 0.999606
168.070651 1.000000 1129.742025 0.999584
160.946507 1.000000 1032.181823 0.999479
167.129891 1.000000 1065.179861 0.999144
169.302965 1.000000 1112.927707 0.999332
162.711044 1.000000 1054.900093 0.999504
171.184705 1.000000 1172.545820 0.999545
163.758445 1.000000 1014.453929 0.998668
157.228066 1.000000 901.574449 0.998980
168.130405 1.000000 1078.706157 0.999472
166.130041 1.000000 1002.529343 0.999341
175.154889 1.000000 1212.304183 0.999446
160.073787 1.000000 978.252126 0.999221
152.161636 1.000000 986.410238 0.999273
164.625480 1.000000 1116.785044 0.999591
159.881383 1.000000 1075.237116 0.999590
156.692569 1.000000 972.616152 0.999177
158.534200 1.000000 1000.061725 0.999020
179.911712 1.000000 1196.074154 0.999441
158.838348 1.000000 1050.462071 0.999292
162.954921 1.000000 937.901760 0.999124
161.096447 1.000000 980.331768 0.999430
156.183857 1.000000 1007.957108 0.999193
169.187674 1.000000 1069.793460 0.999277
151.841936 1.000000 946.482385 0.999365
163.061099 1.000000 1024.474030 0.999079
150.880346 1.000000 979.292387 0.999025
161.731835 1.000000 1060.704492 0.999267
174.998578 1.000000 1156.713788 0.999556
168.526015 1.000000 1104.741326 0.999258
162.954771 1.000000 1065.073759 0.999186
169.277889 1.000000 1069.862912 0.999285
168.228308 1.000000 1151.591753 0.998837
159.667011 1.000000 942.982663 0.999597
150.853417 1.000000 924.862940 0.998897
156.269186 1.000000 950.494281 0.998943
173.503051 1.000000 1202.286503 0.999509
168.339646 1.000000 1060.308303 0.998967
155.678395 1.000000 920.965998 0.999031
159.265431 1.000000 986.311731 0.999203
168.691649 1.000000 1080.569822 0.999205
161.585337 1.000000 1048.289312 0.999266
165.321779 1.000000 1077.226498 0.999056
155.662681 1.000000 959.719982 0.998696
176.757512 1.000000 1186.465938 0.999492
177.075374 1.000000 1190.868471 0.999425
150.342079 1.000000 926.105312 0.999111
167.786899 1.000000 1038.944430 0.999644
158.571917 1.000000 997.085097 0.999270
172.169312 1.000000 1147.495995 0.999657
155.700443 1.000000 951.025125 0.999222
155.115318 1.000000 1039.340709 0.999662
159.181861 1.000000 1010.550430 0.998771
160.698527 1.000000 1126.487105 0.999554
165.051598 1.000000 1051.438811 0.999469
160.228322 1.000000 1046.138799 0.999189
141.442469 1.000000 851.192952 0.999132
161.683492 1.000000 1005.032512 0.999332 
0/129 outside of cosine similarity of .95 (with softmax)
0/129 outside of cosine similarity of .99 (without softmax)

Non-Principal eigenvector vs self attention 1st vector

eigenvalue mag eigenvector mag self attention cosine similarity (with softmax)
9206.530596 1.000000 95.267962 0.831076
153.035448 1.000000 6.677844 0.827429
146.985750 1.000000 6.469487 0.854084
160.615877 1.000000 6.880530 0.818504
145.946463 1.000000 6.241726 0.809585
165.390938 1.000000 6.646201 0.846254
168.939761 1.000000 6.637303 0.821917
173.455692 1.000000 7.324602 0.838477
164.364278 1.000000 7.093303 0.817140
170.809366 1.000000 6.815885 0.817282
168.725411 1.000000 6.963146 0.855412
162.228956 1.000000 6.471354 0.849791
152.398589 1.000000 6.420290 0.806038
152.788177 1.000000 6.011698 0.803389
159.292100 1.000000 6.641088 0.828570
164.603275 1.000000 6.781595 0.805229
165.112983 1.000000 7.025318 0.817439
165.635110 1.000000 6.455346 0.817941
157.807981 1.000000 6.499539 0.841227
177.225237 1.000000 6.971489 0.845225
158.409995 1.000000 6.420343 0.849690
160.217627 1.000000 6.233390 0.817334
170.408037 1.000000 7.133776 0.799513
168.460888 1.000000 6.594924 0.832881
153.034498 1.000000 6.524873 0.841504
169.224380 1.000000 6.500595 0.806773
174.409270 1.000000 7.013595 0.815065
153.403013 1.000000 6.277223 0.819660
157.568193 1.000000 6.493295 0.794931
155.412338 1.000000 6.163024 0.808923
162.250681 1.000000 6.594340 0.796342
162.858646 1.000000 6.511601 0.843460
148.066191 1.000000 6.250871 0.812702
161.412856 1.000000 6.308377 0.815296
154.030834 1.000000 6.039407 0.810947
174.718474 1.000000 6.969898 0.796877
165.243244 1.000000 6.815741 0.820110
163.703212 1.000000 6.490107 0.806950
168.063493 1.000000 6.475724 0.845176
167.906161 1.000000 6.497066 0.821268
167.534931 1.000000 6.784336 0.816985
160.578051 1.000000 6.815965 0.830033
164.235436 1.000000 6.536982 0.815864
155.008998 1.000000 6.548110 0.821649
168.404630 1.000000 6.042017 0.801990
158.640308 1.000000 6.138811 0.837103
169.846162 1.000000 7.062570 0.802700
166.141267 1.000000 6.706509 0.827752
154.979091 1.000000 6.249225 0.847668
160.257850 1.000000 6.502338 0.803048
161.575667 1.000000 6.861466 0.808260
163.650667 1.000000 7.000538 0.826630
163.903141 1.000000 6.550479 0.843383
163.390138 1.000000 7.016451 0.805900
153.517123 1.000000 6.939118 0.806179
163.341905 1.000000 6.181780 0.853177
163.202602 1.000000 6.498106 0.795990
157.279692 1.000000 6.649323 0.815621
157.167207 1.000000 6.459216 0.801528
164.519616 1.000000 6.337793 0.802012
172.830553 1.000000 6.946143 0.840569
157.045462 1.000000 6.116538 0.792016
158.939444 1.000000 6.169372 0.809588
166.080571 1.000000 6.873749 0.832122
155.330333 1.000000 6.276361 0.816875
157.490737 1.000000 6.500678 0.819748
169.363566 1.000000 6.415871 0.831047
162.087452 1.000000 6.448137 0.839985
166.146630 1.000000 6.409095 0.827629
152.824261 1.000000 6.379470 0.826678
146.829732 1.000000 6.456515 0.860507
168.877339 1.000000 6.621731 0.829650
168.070651 1.000000 6.735037 0.824828
160.946507 1.000000 6.945548 0.795881
167.129891 1.000000 6.698101 0.828953
169.302965 1.000000 6.386418 0.833584
162.711044 1.000000 6.820714 0.841658
171.184705 1.000000 6.479948 0.814795
163.758445 1.000000 6.786087 0.876946
157.228066 1.000000 6.897502 0.834229
168.130405 1.000000 7.153319 0.829107
166.130041 1.000000 6.869857 0.824784
175.154889 1.000000 7.173491 0.828407
160.073787 1.000000 6.545292 0.800310
152.161636 1.000000 6.392094 0.808618
164.625480 1.000000 6.573605 0.824282
159.881383 1.000000 6.868605 0.837351
156.692569 1.000000 6.437777 0.792224
158.534200 1.000000 6.013324 0.836775
179.911712 1.000000 7.174269 0.814937
158.838348 1.000000 5.587373 0.798442
162.954921 1.000000 7.203439 0.826036
161.096447 1.000000 6.860645 0.796685
156.183857 1.000000 6.461804 0.829516
169.187674 1.000000 6.765574 0.822824
151.841936 1.000000 5.958579 0.817141
163.061099 1.000000 6.349641 0.806047
150.880346 1.000000 6.106754 0.803671
161.731835 1.000000 6.297961 0.844227
174.998578 1.000000 6.804727 0.820807
168.526015 1.000000 6.916555 0.853660
162.954771 1.000000 6.502600 0.790466
169.277889 1.000000 6.708884 0.829987
168.228308 1.000000 6.492005 0.843894
159.667011 1.000000 7.356210 0.835014
150.853417 1.000000 6.376264 0.837785
156.269186 1.000000 6.799447 0.818207
173.503051 1.000000 6.649155 0.864477
168.339646 1.000000 6.439181 0.849884
155.678395 1.000000 6.491497 0.837756
159.265431 1.000000 6.530884 0.807703
168.691649 1.000000 6.532615 0.827122
161.585337 1.000000 6.650253 0.817530
165.321779 1.000000 6.535029 0.785597
155.662681 1.000000 6.734863 0.818867
176.757512 1.000000 6.260079 0.825161
177.075374 1.000000 6.768350 0.823006
150.342079 1.000000 5.996217 0.804438
167.786899 1.000000 6.814211 0.839086
158.571917 1.000000 6.793648 0.809067
172.169312 1.000000 7.102100 0.806284
155.700443 1.000000 6.887268 0.847649
155.115318 1.000000 6.317686 0.818406
159.181861 1.000000 6.494668 0.826172
160.698527 1.000000 6.389890 0.823149
165.051598 1.000000 6.528850 0.809876
160.228322 1.000000 6.611481 0.781293
141.442469 1.000000 6.018621 0.803200
161.683492 1.000000 6.363545 0.832482
eigenvalue mag eigenvector mag self attention cosine similarity (without softmax)
9206.530596 1.000000 663563.299286 0.759353
153.035448 1.000000 986.932529 0.794412
146.985750 1.000000 921.390981 0.844428
160.615877 1.000000 1062.004238 0.786020
145.946463 1.000000 881.471524 0.792560
165.390938 1.000000 1060.110611 0.824138
168.939761 1.000000 1076.486190 0.786310
173.455692 1.000000 1217.497038 0.806871
164.364278 1.000000 1122.958085 0.778544
170.809366 1.000000 1117.169061 0.792614
168.725411 1.000000 1130.548081 0.817988
162.228956 1.000000 1004.520971 0.822705
152.398589 1.000000 942.872802 0.778624
152.788177 1.000000 884.139813 0.763514
159.292100 1.000000 1015.351799 0.790285
164.603275 1.000000 1078.569778 0.768503
165.112983 1.000000 1107.108619 0.780043
165.635110 1.000000 1034.053413 0.788457
157.807981 1.000000 989.866195 0.811710
177.225237 1.000000 1191.653017 0.809651
158.409995 1.000000 981.265974 0.824366
160.217627 1.000000 963.461947 0.796600
170.408037 1.000000 1161.864926 0.780410
168.460888 1.000000 1068.981161 0.812708
153.034498 1.000000 961.107531 0.795096
169.224380 1.000000 1055.491698 0.770199
174.409270 1.000000 1169.397747 0.787765
153.403013 1.000000 932.864167 0.802928
157.568193 1.000000 989.713896 0.775089
155.412338 1.000000 923.322647 0.768451
162.250681 1.000000 1028.272934 0.752371
162.858646 1.000000 1026.139767 0.829964
148.066191 1.000000 895.530530 0.780310
161.412856 1.000000 977.204049 0.779351
154.030834 1.000000 897.585165 0.788568
174.718474 1.000000 1166.419430 0.759370
165.243244 1.000000 1079.371510 0.781315
163.703212 1.000000 1019.041048 0.773263
168.063493 1.000000 1054.097491 0.822969
167.906161 1.000000 1042.329824 0.790147
167.534931 1.000000 1088.410514 0.795205
160.578051 1.000000 1054.150942 0.803976
164.235436 1.000000 1032.632653 0.786649
155.008998 1.000000 973.848494 0.790484
168.404630 1.000000 969.162695 0.755415
158.640308 1.000000 932.434099 0.817776
169.846162 1.000000 1142.183379 0.751276
166.141267 1.000000 1068.792570 0.810726
154.979091 1.000000 930.404063 0.810392
160.257850 1.000000 993.183997 0.770809
161.575667 1.000000 1073.839519 0.767971
163.650667 1.000000 1098.034117 0.786050
163.903141 1.000000 1033.038479 0.800069
163.390138 1.000000 1104.663242 0.791498
153.517123 1.000000 1029.581357 0.782122
163.341905 1.000000 972.653563 0.818319
163.202602 1.000000 1023.257487 0.767478
157.279692 1.000000 999.641199 0.776580
157.167207 1.000000 980.416235 0.776756
164.519616 1.000000 1000.514799 0.775064
172.830553 1.000000 1151.895329 0.812888
157.045462 1.000000 915.874324 0.750592
158.939444 1.000000 944.702202 0.774818
166.080571 1.000000 1096.793760 0.801077
155.330333 1.000000 940.219388 0.772656
157.490737 1.000000 979.557709 0.767341
169.363566 1.000000 1040.782782 0.808411
162.087452 1.000000 1001.808777 0.808881
166.146630 1.000000 1021.310957 0.808150
152.824261 1.000000 936.832629 0.784438
146.829732 1.000000 911.813623 0.810947
168.877339 1.000000 1067.384958 0.793722
168.070651 1.000000 1091.169738 0.790385
160.946507 1.000000 1076.632457 0.760864
167.129891 1.000000 1068.878228 0.816008
169.302965 1.000000 1043.349807 0.816336
162.711044 1.000000 1064.869687 0.811908
171.184705 1.000000 1070.166797 0.789451
163.758445 1.000000 1072.393726 0.854383
157.228066 1.000000 1046.396507 0.816342
168.130405 1.000000 1147.423876 0.779455
166.130041 1.000000 1098.639801 0.803197
175.154889 1.000000 1198.024532 0.767754
160.073787 1.000000 1002.511192 0.757024
152.161636 1.000000 940.430782 0.755833
164.625480 1.000000 1032.317496 0.791767
159.881383 1.000000 1054.191665 0.803722
156.692569 1.000000 975.302779 0.766659
158.534200 1.000000 915.931027 0.806051
179.911712 1.000000 1232.841465 0.777079
158.838348 1.000000 848.501567 0.772948
162.954921 1.000000 1126.246139 0.790454
161.096447 1.000000 1062.143340 0.752874
156.183857 1.000000 974.546894 0.806801
169.187674 1.000000 1100.147076 0.788766
151.841936 1.000000 869.090949 0.804821
163.061099 1.000000 995.329331 0.770385
150.880346 1.000000 888.211331 0.765935
161.731835 1.000000 984.603202 0.808891
174.998578 1.000000 1135.187239 0.794548
168.526015 1.000000 1126.843098 0.839097
162.954771 1.000000 1017.808123 0.745439
169.277889 1.000000 1086.866835 0.815043
168.228308 1.000000 1049.004683 0.803221
159.667011 1.000000 1132.786460 0.796088
150.853417 1.000000 924.619091 0.830806
156.269186 1.000000 1018.898690 0.781596
173.503051 1.000000 1105.225452 0.817194
168.339646 1.000000 1042.579505 0.824729
155.678395 1.000000 969.937637 0.797230
159.265431 1.000000 999.033539 0.787179
168.691649 1.000000 1056.922474 0.770239
161.585337 1.000000 1032.916585 0.765049
165.321779 1.000000 1030.084105 0.736276
155.662681 1.000000 1006.992628 0.791940
176.757512 1.000000 1060.100038 0.799826
177.075374 1.000000 1140.121328 0.808242
150.342079 1.000000 870.263814 0.770739
167.786899 1.000000 1089.167103 0.789884
158.571917 1.000000 1027.084912 0.771537
172.169312 1.000000 1164.404668 0.780515
155.700443 1.000000 1032.449923 0.823544
155.115318 1.000000 946.423578 0.799834
159.181861 1.000000 996.326855 0.791866
160.698527 1.000000 987.887217 0.804006
165.051598 1.000000 1034.203451 0.759536
160.228322 1.000000 1025.394188 0.765167
141.442469 1.000000 823.310567 0.786806
161.683492 1.000000 992.011656 0.812388 
129/129 outside of cosine similarity of .95 (with softmax)
129/129 outside of cosine similarity of .99 (without softmax)

About

Self attention based computation of principal eigenvector under certain conditions

Topics

eigenvectors lemma self-attention

Resources

Readme

License

BSD-3-Clause license
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages