confusion about coordinates

[BroJunn]

Thanks for this amazing work!

I'm very confused about one thing that how MLLM is aware of the size of the image.

I looked a bit through the json, e.g. there's an example

{ "id": 3559, "image": "bridge_data_v2/bridge_data_v1/berkeley/toysink3_bww/put_pot_or_pan_from_sink_into_drying_rack/2021-06-14_18-01-38/raw/traj_group0/traj12/images0/im_0.jpg", "height": 480, "width": 640, "conversations": [ { "from": "human", "value": "<image>\nYou are currently a robot performing robotic manipulation tasks. Your task instruction: put pot or pan from sink into drying rack. Observe the image, use 2D points to mark the manipulated object-centric waypoints to guide the robot to manipulate the object.Typically, the waypoints consists of an ordered sequence of eight 2D points. The format is <point>[[x1, y1], [x2, y2], ...]</point>." }, { "from": "gpt", "value": "<Description>\nThe image shows a robotic arm positioned above a <ref>metal pot</ref><box>[[411, 452, 712, 667]]</box> in a sink. To the right of the pot is a blue cup, and above the pot is a <ref>drying rack</ref><box>[[74, 191, 554, 495]]</box>.\n</Description>\n\n<Reasoning>\nTo move the metal pot from the sink into the drying rack, start by identifying the initial position of the pot in the sink at <point>[[674, 565]]</point>. The robotic arm should lift the pot upwards and slightly to the left, moving towards the drying rack. The path should include intermediate points such as <point>[[611, 579]]</point> and <point>[[607, 530]]</point> as it moves out of the sink. Continue moving the pot towards the drying rack, passing through points like <point>[[594, 458]]</point> and <point>[[566, 396]]</point>. Finally, place the pot into the drying rack at the target location <box>[[173, 263, 406, 424]]</box>, ending the trace at <point>[[575, 301]]</point>.\n</Reasoning>\n\n<Answer>\nThe visual trace for moving the metal pot into the drying rack is <point>[[674, 565], [611, 579], [607, 530], [594, 458], [566, 396], [527, 343], [561, 304], [575, 301]]</point>.\n</Answer>" } ] },

If the size of image is (480, 640), how point like[674, 565] exists on this image? I'm quite confused here.

And if the coordinates are not normalized to [0,1] or e.g. [0, 336], how could the model be aware of the size of given image

Looking forward to you reply!

[pickxiguapi]

Hi, thanks so much for the kind words and for asking this excellent and important question!

The image coordinates are not based on the original image dimensions (e.g., 480x640). Instead, we process them as follows:

1. Padding to a Square: We first pad the original image to make it a square, maintaining the aspect ratio. For a 480x640 image, it would be padded to 640x640.

2. Normalizing to 1000: The coordinates are then represented in a normalized coordinate system that ranges from 0 to 1000 based on this new square canvas.

This is why a coordinate like [674, 565] is valid—it's located on the 1000x1000 grid, not the original 480x640 pixel space. The primary benefit is that it allows all images to share a single, unified coordinate scale, regardless of their original size or aspect ratio. This approach is identical to the one used in ASMv2.

I did mention this in the paper and also in the README here: https://github.com/pickxiguapi/Embodied-FSD?tab=readme-ov-file#-weak-to-strong-dataset

However, your question makes it very clear that this information is not prominent enough lol. You're right—this is absolutely critical information for training, evaluation, or simply using the model correctly. I will update the documentation to place this explanation in a much more visible location.

[BroJunn]

Thanks for the swift reply!

1. Another question regarding this:
   do you think it's gonna be okay if I train on e.g. 336,336 but eval on 224,224? My first guess is that it should be fine since it's considered as a scaled normalization? But it's been a bit tricky if the img_token_num changes for different solution?

2. I didn't find the path to download Visual7W, the link cannot be opened. Do you have any solution?

Thanks in advance!

[BroJunn]

I think this is the newest link for visual7w:
http://vision.stanford.edu/yukezhu/visual7w_images.zip