A bug in tokenization_codegen.py

[CBellaris]

When running the code, the following error might be encountered:

File "HKU-DASC7606-A2\tokenization_codegen.py", line 203, in get_vocab
return dict(self.encoder, **self.added_tokens_encoder)
AttributeError: 'CodeGenTokenizer' object has no attribute 'encoder'

It appears that the get_vocab() function is being called during the parent class constructor's execution, at which point self.encoder has not been initialized. To address this issue, the following segment of code should be moved to execute before super().init():

with open(vocab_file, encoding="utf-8") as vocab_handle:
    self.encoder = json.load(vocab_handle)

This change can avoid the error but I don't know if it makes sense

[hzngadieu]

This bug is due to the incorrect version of transformers
run:
pip install transformers==4.31.0