reference.txt

Token IDs: [128000, 791, 6864, 315, 9822, 374]
Num tokens: 6

Embedding shape: torch.Size([1, 6, 2048])

Token 0 (id=128000) embedding first 10 values:
  [0] = 0.002685546875
  [1] = 0.003082275390625
  [2] = -0.006805419921875
  [3] = 0.0419921875
  [4] = -0.002655029296875
  [5] = -0.0003032684326171875
  [6] = 0.00102996826171875
  [7] = 0.00469970703125
  [8] = 0.006866455078125
  [9] = -0.0025634765625

Token 1 (id=791) embedding first 10 values:
  [0] = 0.007659912109375
  [1] = -0.030517578125
  [2] = 0.01458740234375
  [3] = 0.01361083984375
  [4] = -0.0093994140625
  [5] = -0.0224609375
  [6] = 0.007080078125
  [7] = 0.0439453125
  [8] = 0.007659912109375
  [9] = -0.0036468505859375

Token 2 (id=6864) embedding first 10 values:
  [0] = -0.0301513671875
  [1] = 0.0361328125
  [2] = 0.0250244140625
  [3] = 0.0576171875
  [4] = 0.0034332275390625
  [5] = -0.0242919921875
  [6] = 0.0023193359375
  [7] = -0.017333984375
  [8] = -0.00081634521484375
  [9] = 0.00848388671875

Token 3 (id=315) embedding first 10 values:
  [0] = -0.0019378662109375
  [1] = -0.0140380859375
  [2] = 0.005279541015625
  [3] = -0.02783203125
  [4] = 0.0091552734375
  [5] = -0.0038909912109375
  [6] = -0.0076904296875
  [7] = -0.004974365234375
  [8] = 0.016845703125
  [9] = 0.0159912109375

Token 4 (id=9822) embedding first 10 values:
  [0] = -0.00823974609375
  [1] = 0.02880859375
  [2] = -0.00799560546875
  [3] = -0.0147705078125
  [4] = -0.0079345703125
  [5] = 0.006103515625
  [6] = 0.004730224609375
  [7] = 0.006439208984375
  [8] = 0.0011444091796875
  [9] = 0.00628662109375

Token 5 (id=374) embedding first 10 values:
  [0] = 0.01031494140625
  [1] = 0.0196533203125
  [2] = 0.02197265625
  [3] = -0.00093841552734375
  [4] = 0.0247802734375
  [5] = -0.01361083984375
  [6] = 0.019775390625
  [7] = 0.018310546875
  [8] = -0.00390625
  [9] = 0.02197265625

RMSNorm output shape: torch.Size([1, 6, 2048])

Token 0 (id=128000) RMSNorm first 10 values:
  [0] = 0.021240234375
  [1] = 0.0289306640625
  [2] = -0.08935546875
  [3] = -0.025146484375
  [4] = -0.01904296875
  [5] = -0.002960205078125
  [6] = -0.007354736328125
  [7] = -0.03857421875
  [8] = 0.049072265625
  [9] = 0.0225830078125

Token 1 (id=791) RMSNorm first 10 values:
  [0] = 0.0703125
  [1] = -0.33203125
  [2] = 0.2216796875
  [3] = -0.0093994140625
  [4] = -0.07861328125
  [5] = -0.25390625
  [6] = -0.058837890625
  [7] = -0.41796875
  [8] = 0.0634765625
  [9] = 0.037109375

Token 2 (id=6864) RMSNorm first 10 values:
  [0] = -0.197265625
  [1] = 0.279296875
  [2] = 0.271484375
  [3] = -0.0284423828125
  [4] = 0.0205078125
  [5] = -0.1962890625
  [6] = -0.01373291015625
  [7] = 0.11767578125
  [8] = -0.004852294921875
  [9] = -0.0615234375

Token 3 (id=315) RMSNorm first 10 values:
  [0] = -0.0164794921875
  [1] = -0.1416015625
  [2] = 0.07421875
  [3] = 0.017822265625
  [4] = 0.07080078125
  [5] = -0.040771484375
  [6] = 0.05908203125
  [7] = 0.043701171875
  [8] = 0.12890625
  [9] = -0.150390625

Token 4 (id=9822) RMSNorm first 10 values:
  [0] = -0.0654296875
  [1] = 0.26953125
  [2] = -0.10498046875
  [3] = 0.00885009765625
  [4] = -0.057373046875
  [5] = 0.0595703125
  [6] = -0.033935546875
  [7] = -0.052978515625
  [8] = 0.00823974609375
  [9] = -0.05517578125

Token 5 (id=374) RMSNorm first 10 values:
  [0] = 0.091796875
  [1] = 0.20703125
  [2] = 0.32421875
  [3] = 0.00063323974609375
  [4] = 0.201171875
  [5] = -0.1494140625
  [6] = -0.1591796875
  [7] = -0.1689453125
  [8] = -0.031494140625
  [9] = -0.2158203125

Q projection shape: torch.Size([1, 6, 2048])

Token 0 (id=128000) Q projection first 10 values:
  [0] = -0.1376953125
  [1] = -0.1806640625
  [2] = 0.48828125
  [3] = -0.23828125
  [4] = -0.1474609375
  [5] = -0.08203125
  [6] = -0.05859375
  [7] = 0.05908203125
  [8] = 0.0225830078125
  [9] = -0.62109375

Token 1 (id=791) Q projection first 10 values:
  [0] = 0.52734375
  [1] = 1.3984375
  [2] = 1.8359375
  [3] = -2.03125
  [4] = -2.859375
  [5] = -0.859375
  [6] = -0.1396484375
  [7] = -1.046875
  [8] = -1.765625
  [9] = -2.453125

Token 2 (id=6864) Q projection first 10 values:
  [0] = 0.5703125
  [1] = 1.4765625
  [2] = 1.2890625
  [3] = -2.609375
  [4] = -1.8671875
  [5] = -1.1640625
  [6] = -0.8359375
  [7] = -0.6015625
  [8] = -1.375
  [9] = -1.5625

Token 3 (id=315) Q projection first 10 values:
  [0] = 0.9140625
  [1] = 2.640625
  [2] = 3.46875
  [3] = -4.46875
  [4] = -4.75
  [5] = -0.283203125
  [6] = -2.28125
  [7] = -1.171875
  [8] = -1.9375
  [9] = -2.765625

Token 4 (id=9822) Q projection first 10 values:
  [0] = 0.515625
  [1] = 1.5625
  [2] = 1.34375
  [3] = -2.46875
  [4] = -2.546875
  [5] = -1.390625
  [6] = -0.5078125
  [7] = -0.7890625
  [8] = -1.8359375
  [9] = -1.7421875

Token 5 (id=374) Q projection first 10 values:
  [0] = 1.0546875
  [1] = 3.453125
  [2] = 3.5625
  [3] = -4.6875
  [4] = -5.96875
  [5] = -0.71484375
  [6] = -2.234375
  [7] = -1.046875
  [8] = -1.859375
  [9] = -2.953125

K projection shape: torch.Size([1, 6, 512])

Token 0 (id=128000) K projection first 10 values:
  [0] = 0.17578125
  [1] = 0.126953125
  [2] = -0.010986328125
  [3] = -0.11328125
  [4] = -0.150390625
  [5] = -0.06591796875
  [6] = -0.03515625
  [7] = -0.02685546875
  [8] = -0.09716796875
  [9] = -0.138671875

Token 1 (id=791) K projection first 10 values:
  [0] = 5.5625
  [1] = 3.375
  [2] = -0.56640625
  [3] = -1.4921875
  [4] = -2.65625
  [5] = -1.71875
  [6] = -0.8203125
  [7] = -2.359375
  [8] = -1.40625
  [9] = -2.125

Token 2 (id=6864) K projection first 10 values:
  [0] = 6.4375
  [1] = 2.421875
  [2] = -0.6328125
  [3] = -2.859375
  [4] = -1.625
  [5] = -2.8125
  [6] = -1.5234375
  [7] = -2.15625
  [8] = -2.296875
  [9] = -1.78125

Token 3 (id=315) K projection first 10 values:
  [0] = 4.78125
  [1] = 3.453125
  [2] = -1.8359375
  [3] = -2.40625
  [4] = -1.5703125
  [5] = -2.375
  [6] = -0.92578125
  [7] = -2.296875
  [8] = -1.734375
  [9] = -3.0625

Token 4 (id=9822) K projection first 10 values:
  [0] = 5.71875
  [1] = 2.1875
  [2] = -0.306640625
  [3] = -2.890625
  [4] = -1.7265625
  [5] = -2.578125
  [6] = -1.4296875
  [7] = -2.375
  [8] = -1.578125
  [9] = -2.15625

Token 5 (id=374) K projection first 10 values:
  [0] = 4.75
  [1] = 3.265625
  [2] = -1.671875
  [3] = -2.4375
  [4] = -2.0
  [5] = -2.796875
  [6] = -1.390625
  [7] = -3.375
  [8] = -1.796875
  [9] = -2.140625

V projection shape: torch.Size([1, 6, 512])

Token 0 (id=128000) V projection first 10 values:
  [0] = 0.004364013671875
  [1] = 0.005523681640625
  [2] = -0.027099609375
  [3] = -0.003631591796875
  [4] = 0.005401611328125
  [5] = -0.00775146484375
  [6] = 0.00194549560546875
  [7] = -0.000164031982421875
  [8] = -4.267692565917969e-05
  [9] = 0.0013427734375

Token 1 (id=791) V projection first 10 values:
  [0] = -0.0859375
  [1] = -0.01397705078125
  [2] = 0.01953125
  [3] = 0.057373046875
  [4] = 0.06787109375
  [5] = -0.53125
  [6] = -0.006072998046875
  [7] = -0.039794921875
  [8] = 0.0177001953125
  [9] = 0.0029449462890625

Token 2 (id=6864) V projection first 10 values:
  [0] = -0.193359375
  [1] = -0.0006103515625
  [2] = 0.018798828125
  [3] = -0.10498046875
  [4] = 0.007415771484375
  [5] = 0.0791015625
  [6] = 0.0693359375
  [7] = 0.0240478515625
  [8] = 0.06982421875
  [9] = -0.06884765625

Token 3 (id=315) V projection first 10 values:
  [0] = -0.01092529296875
  [1] = 0.047119140625
  [2] = 0.00830078125
  [3] = 0.02685546875
  [4] = -0.1875
  [5] = 0.17578125
  [6] = 0.006866455078125
  [7] = 0.11572265625
  [8] = 0.048583984375
  [9] = 0.034912109375

Token 4 (id=9822) V projection first 10 values:
  [0] = 0.0030059814453125
  [1] = -0.041259765625
  [2] = 0.09814453125
  [3] = -0.042236328125
  [4] = 0.02294921875
  [5] = 0.014404296875
  [6] = 0.09130859375
  [7] = 0.059326171875
  [8] = -0.0036773681640625
  [9] = -0.03173828125

Token 5 (id=374) V projection first 10 values:
  [0] = 0.053466796875
  [1] = 0.0123291015625
  [2] = 0.001739501953125
  [3] = 0.047607421875
  [4] = 0.1513671875
  [5] = 0.036865234375
  [6] = 0.08837890625
  [7] = -0.0859375
  [8] = 0.022705078125
  [9] = -0.00823974609375

Weight shapes:
  q_proj: torch.Size([2048, 2048])
  k_proj: torch.Size([512, 2048])
  v_proj: torch.Size([512, 2048])
  o_proj: torch.Size([2048, 2048])
--------------------------------------------------

Layer 0 input first 10 values (token 0):
  [0] = 0.002685546875
  [1] = 0.003082275390625
  [2] = -0.006805419921875
  [3] = 0.0419921875
  [4] = -0.002655029296875
  [5] = -0.0003032684326171875
  [6] = 0.00102996826171875
  [7] = 0.00469970703125
  [8] = 0.006866455078125
  [9] = -0.0025634765625
  shape after layer: torch.Size([1, 6, 2048])

Layer 1 input first 10 values (token 0):
  [0] = 0.02001953125
  [1] = -0.0458984375
  [2] = 0.05126953125
  [3] = 0.0380859375
  [4] = -0.00080108642578125
  [5] = -0.009033203125
  [6] = -0.02978515625
  [7] = -0.0087890625
  [8] = -0.024169921875
  [9] = -0.02001953125
  shape after layer: torch.Size([1, 6, 2048])

Layer 2 input first 10 values (token 0):
  [0] = 0.1337890625
  [1] = -0.490234375
  [2] = 1.6015625
  [3] = -0.271484375
  [4] = 0.46875
  [5] = -0.14453125
  [6] = -0.064453125
  [7] = 0.453125
  [8] = -0.1572265625
  [9] = -0.265625
  shape after layer: torch.Size([1, 6, 2048])

Layer 3 input first 10 values (token 0):
  [0] = 0.1357421875
  [1] = -0.50390625
  [2] = 1.6015625
  [3] = -0.30078125
  [4] = 0.484375
  [5] = -0.1318359375
  [6] = -0.0556640625
  [7] = 0.46484375
  [8] = -0.12060546875
  [9] = -0.263671875
  shape after layer: torch.Size([1, 6, 2048])

Layer 4 input first 10 values (token 0):
  [0] = 0.1279296875
  [1] = -0.5078125
  [2] = 1.59375
  [3] = -0.353515625
  [4] = 0.490234375
  [5] = -0.1416015625
  [6] = -0.053955078125
  [7] = 0.474609375
  [8] = -0.0595703125
  [9] = -0.283203125
  shape after layer: torch.Size([1, 6, 2048])

Layer 5 input first 10 values (token 0):
  [0] = 0.154296875
  [1] = -0.51171875
  [2] = 1.6171875
  [3] = -0.294921875
  [4] = 0.53125
  [5] = -0.142578125
  [6] = -0.05517578125
  [7] = 0.478515625
  [8] = -0.07421875
  [9] = -0.36328125
  shape after layer: torch.Size([1, 6, 2048])

Layer 6 input first 10 values (token 0):
  [0] = 0.1376953125
  [1] = -0.453125
  [2] = 1.6171875
  [3] = -0.291015625
  [4] = 0.462890625
  [5] = -0.158203125
  [6] = -0.0615234375
  [7] = 0.498046875
  [8] = -0.11669921875
  [9] = -0.36328125
  shape after layer: torch.Size([1, 6, 2048])

Layer 7 input first 10 values (token 0):
  [0] = 0.125
  [1] = -0.41796875
  [2] = 1.5859375
  [3] = -0.2294921875
  [4] = 0.458984375
  [5] = -0.1943359375
  [6] = -0.048828125
  [7] = 0.490234375
  [8] = -0.1220703125
  [9] = -0.40234375
  shape after layer: torch.Size([1, 6, 2048])

Layer 8 input first 10 values (token 0):
  [0] = 0.1484375
  [1] = -0.35546875
  [2] = 1.578125
  [3] = -0.166015625
  [4] = 0.462890625
  [5] = -0.224609375
  [6] = -0.044921875
  [7] = 0.435546875
  [8] = -0.115234375
  [9] = -0.35546875
  shape after layer: torch.Size([1, 6, 2048])

Layer 9 input first 10 values (token 0):
  [0] = 0.140625
  [1] = -0.353515625
  [2] = 1.484375
  [3] = -0.1796875
  [4] = 0.4765625
  [5] = -0.193359375
  [6] = 0.03173828125
  [7] = 0.353515625
  [8] = -0.0947265625
  [9] = -0.33984375
  shape after layer: torch.Size([1, 6, 2048])

Layer 10 input first 10 values (token 0):
  [0] = 0.0615234375
  [1] = -0.294921875
  [2] = 1.453125
  [3] = -0.1533203125
  [4] = 0.447265625
  [5] = -0.1015625
  [6] = 0.06787109375
  [7] = 0.38671875
  [8] = -0.0888671875
  [9] = -0.36328125
  shape after layer: torch.Size([1, 6, 2048])

Layer 11 input first 10 values (token 0):
  [0] = 0.0341796875
  [1] = -0.265625
  [2] = 1.3671875
  [3] = -0.16015625
  [4] = 0.4453125
  [5] = -0.1181640625
  [6] = -0.0238037109375
  [7] = 0.271484375
  [8] = -0.078125
  [9] = -0.37890625
  shape after layer: torch.Size([1, 6, 2048])

Layer 12 input first 10 values (token 0):
  [0] = 0.031494140625
  [1] = -0.26953125
  [2] = 1.3125
  [3] = -0.111328125
  [4] = 0.478515625
  [5] = -0.1328125
  [6] = -0.09326171875
  [7] = 0.255859375
  [8] = -0.0751953125
  [9] = -0.384765625
  shape after layer: torch.Size([1, 6, 2048])

Layer 13 input first 10 values (token 0):
  [0] = 0.017822265625
  [1] = -0.28515625
  [2] = 1.28125
  [3] = -0.1259765625
  [4] = 0.61328125
  [5] = -0.17578125
  [6] = -0.080078125
  [7] = 0.19921875
  [8] = -0.041748046875
  [9] = -0.39453125
  shape after layer: torch.Size([1, 6, 2048])

Layer 14 input first 10 values (token 0):
  [0] = 0.0888671875
  [1] = -0.36328125
  [2] = 1.140625
  [3] = 0.0693359375
  [4] = 0.7890625
  [5] = -0.2421875
  [6] = -0.04833984375
  [7] = 0.1630859375
  [8] = 0.134765625
  [9] = -0.39453125
  shape after layer: torch.Size([1, 6, 2048])

Layer 15 input first 10 values (token 0):
  [0] = 0.0947265625
  [1] = -0.83203125
  [2] = 1.4296875
  [3] = 0.1298828125
  [4] = 0.76953125
  [5] = -0.462890625
  [6] = -0.08837890625
  [7] = -0.0888671875
  [8] = 0.06201171875
  [9] = -0.38671875
  shape after layer: torch.Size([1, 6, 2048])

Final hidden state first 10 values (token 0):
  [0] = 2.765625
  [1] = 0.59375
  [2] = 2.78125
  [3] = -0.6484375
  [4] = 0.10546875
  [5] = 0.009765625
  [6] = 2.328125
  [7] = 2.53125
  [8] = -1.59375
  [9] = 0.357421875

Final RMSNorm first 10 values (token 0):
  [0] = 0.97265625
  [1] = 0.1923828125
  [2] = 0.61328125
  [3] = -0.181640625
  [4] = 0.037841796875
  [5] = 0.0035552978515625
  [6] = 0.83984375
  [7] = 0.953125
  [8] = -0.478515625
  [9] = 0.1318359375
Predicted token ID: 12366
Decoded:  Paris