max_new_tokens?

[xlchen0205]

Hello,

When you evaluate LVLMs on the generative task, how do you set the parameter "max_new_tokens" or "max_length"?

Maybe it has a big influence on the final results, thank you!

[junyangwang0410]

Hello,

When you evaluate LVLMs on the generative task, how do you set the parameter "max_new_tokens" or "max_length"?

Maybe it has a big influence on the final results, thank you!

As you say, the more content generated by MLLMs, the easier it is to have hallucinations. However, this does not affect our evaluation. When evaluating, we will set up max_new_tokens as bigger as much as possible to let MLLMs generate stop token. To some extent, it can reflect the ability of the model to refuse to generate hallucination content.

In addition, the experimental results in the paper show that when we personally cut off the replies of MLLMs, it will indeed reduce hallucinations, but this also leads to a decline in the quality of recovery.

[mehrdadfazli]

Hi,

I have a follow-up question. I am trying to replicate some of your results in the generation mode and I was wondering what values you used for "max_new_tokens". I understand that you said you picked the largest value possible but can you be more specific regarding each model (particularly InstructBLIP and LLaVA)? Did you even set this parameter or you went with the default value?
Thanks

[junyangwang0410]

Thanks. We hope that the experimental results of the generative task are completely determined by the model. This means that we do not want the response to be truncated by max_new_tokens before generating the terminator token. Therefore, we use a large max_new_tokens value of 2048, which is also the default value of many models.

[mehrdadfazli]

Thank you for your response.