Error when run "python train.py"

[minhntm]

Preparing dialog data in /var/lib/tf_seq2seq_chatbot/data
Creating vocabulary /var/lib/tf_seq2seq_chatbot/data/vocab20000.in from data /var/lib/tf_seq2seq_chatbot/data/chat.in
Traceback (most recent call last):
File "train.py", line 15, in
tf.app.run()
File "/home/minh/opt/anaconda3/envs/ml/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "train.py", line 12, in main
train()
File "/home/minh/source/machine-learing/chatbot/tf_seq2seq_chatbot/tf_seq2seq_chatbot/lib/train.py", line 22, in train
train_data, dev_data, _ = data_utils.prepare_dialog_data(FLAGS.data_dir, FLAGS.vocab_size)
File "/home/minh/source/machine-learing/chatbot/tf_seq2seq_chatbot/tf_seq2seq_chatbot/lib/data_utils.py", line 200, in prepare_dialog_data
create_vocabulary(vocab_path, train_path + ".in", vocabulary_size)
File "/home/minh/source/machine-learing/chatbot/tf_seq2seq_chatbot/tf_seq2seq_chatbot/lib/data_utils.py", line 70, in create_vocabulary
for line in f:
File "/home/minh/opt/anaconda3/envs/ml/lib/python3.5/site-packages/tensorflow/python/platform/gfile.py", line 176, in next
return next(self._fp)
File "/home/minh/opt/anaconda3/envs/ml/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 2329: invalid start byte

[tjrileywisc]

Hi Minhbk,

I'm getting the same error, it's just as the message says. You're running into encoding problems. That character (soft hyphen apparently) isn't in the utf-8 encoding set.

Try opening with a different encoding if you can, otherwise modify the input text to use standard hyphens.

[minhntm]

I run this code with python 2.7, and it work! I don't understand why it work :D

[oradomskyi]

Solution:
Remove or replace all non-ascii characters from train data.
On Windows OS:

1. Install Notepad++ from here:
   https://notepad-plus-plus.org/download/v7.3.html

2. Open chat.in in Notepad++ and do follow:

3. Ctrl+F

4. Go to "Replace" tab

5. Change search mode to "Regular Expression"

6. Paste this regex for Non-ASCII Characters into "Find what :" field:
   [^\x00-\x7F]+

7. Leave field "Replace with : " empty

8. Push "Replace All"

9. Also, you can try Encoding -> Convert to ANSI and then Save

enjoy an be ready for loooooong time of training =)

Problem is that dataset has non ASCII characters (about 3k times ) such as 0xAD(some short -) 0x97 (long -)
0x00AD in Unicode: http://www.fileformat.info/info/unicode/char/ad/index.htm

Works on Win 10, GPU TensorFlow and Python 3.5.

[dsblr]

@FrayaMiner Thanks for your response. Since you are using the same OS configuration what I am using.
I have tested the windows10 + GPU + python 3.5 configuration on example tensorflow model and its working fantastic.
Could you please help me with the initial steps to run this chat-bot module. I am getting the following error while executing the code.

(tensorflow-gpu) C:\Users\user1>python C:\Users\user1\Downloads\tf_seq2seq_chatbot\tf_seq2seq_chatbot\train.py
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:128] successfully opened CUDA library cublas64_80.dll locally
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:128] successfully opened CUDA library cudnn64_5.dll locally
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:128] successfully opened CUDA library cufft64_80.dll locally
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:128] successfully opened CUDA library nvcuda.dll locally
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:128] successfully opened CUDA library curand64_80.dll locally
Preparing dialog data in /var/lib/tf_seq2seq_chatbot/data
Creating vocabulary /var/lib/tf_seq2seq_chatbot/data\vocab20000.in from data /var/lib/tf_seq2seq_chatbot/data\chat.in
Traceback (most recent call last):
File "C:\Users\user1\Downloads\tf_seq2seq_chatbot\tf_seq2seq_chatbot\train.py", line 15, in
tf.app.run()
File "C:\Users\user1\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\platform\app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "C:\Users\user1\Downloads\tf_seq2seq_chatbot\tf_seq2seq_chatbot\train.py", line 12, in main
train()
File "C:\Users\user1\Downloads\tf_seq2seq_chatbot\tf_seq2seq_chatbot\tf_seq2seq_chatbot\lib\train.py", line 22, in train
train_data, dev_data, _ = data_utils.prepare_dialog_data(FLAGS.data_dir, FLAGS.vocab_size)
File "C:\Users\user1\Downloads\tf_seq2seq_chatbot\tf_seq2seq_chatbot\tf_seq2seq_chatbot\lib\data_utils.py", line 200, in prepare_dialog_data
create_vocabulary(vocab_path, train_path + ".in", vocabulary_size)
File "C:\Users\user1\Downloads\tf_seq2seq_chatbot\tf_seq2seq_chatbot\tf_seq2seq_chatbot\lib\data_utils.py", line 70, in create_vocabulary
for line in f:
File "C:\Users\user1\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 162, in next
return self.next()
File "C:\Users\user1\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 156, in next
retval = self.readline()
File "C:\Users\user1\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 123, in readline
self._preread_check()
File "C:\Users\user1\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 73, in _preread_check
compat.as_bytes(self.__name), 1024 * 512, status)
File "C:\Users\user1\Anaconda3\envs\tensorflow-gpu\lib\contextlib.py", line 66, in exit
next(self.gen)
File "C:\Users\user1\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 469, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: /var/lib/tf_seq2seq_chatbot/data\chat.in : The system cannot find the path specified.

[oradomskyi]

@dsblr
Solution:

1. Copypaste file
   "tf_seq2seq_chatbot/data/train/movie_lines_selected.txt"

2. Rename it into
   "chat.in"

3. In
   ../tf_seq2seq_chatbot/configs/config.py
   Specify absolute path to "chat.in" using Windows style with special characters enclosing (dash '\' is special and need to be enclosed), for example:
   "D:\\tf_seq2seq_chatbot\\tf_seq2seq_chatbot\\data\\train\\"
   or
   "D:\\tf_seq2seq_chatbot\\tf_seq2seq_chatbot\\data\\train"
   (I'm not sure here)

4. Follow steps from my comment above, to clean "chat.in" from Non-ASCII characters

[dsblr]

@FrayaMiner Thank you for your response. I am now getting the following error. UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 2329: invalid start byte
I have cleanse the chat.in file as per the step mentioned by you.
Please help to resolve this.

[oradomskyi]

@dsblr you can try manually search and clean file, but I prefer cleaning with regex. also, you can try Encoding -> Convert to ANSI and then save

character 0xad at 2329 is a non-ascii "soft hyphen"
http://www.fileformat.info/info/unicode/char/ad/index.htm

[dsblr]

@FrayaMiner thanks once again for your response. I tried to remove the soft hyphen manually but the errpr still presist . I beleive few of the soft hype I might have missed.
Could you please help me with the cleansed chat file which you are using ? It will be so helpful.

Thanks

[dsblr]

@FrayaMiner is there a way to solve the following error. I am getting this while running the test.py "C:\Work\tf_seq2seq_chatbot\tf_seq2seq_chatbot\lib\seq2seq_model_utils.py", line 43, in get_predicted_sentence
bucket_id = min([b for b in xrange(len(BUCKETS)) if BUCKETS[b][0] > len(input_token_ids)])
NameError: name 'xrange' is not defined

The above error solved after importing xrange into seq2seq_model_utils.py.

But while executing chat.py, another error I am getting :

hello
Traceback (most recent call last):
File "chat.py", line 15, in
tf.app.run()
File "D:\Users\user1\chatbots\tensorflow\softwares\anaconda\lib\site-packages
\tensorflow\python\platform\app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "chat.py", line 12, in main
chat()
File "C:\Work\tf_seq2seq_chatbot\tf_seq2seq_chatbot\lib\chat.
py", line 27, in chat
predicted_sentence = get_predicted_sentence(sentence, vocab, rev_vocab, mode
l, sess)
File "C:\Work\tf_seq2seq_chatbot\tf_seq2seq_chatbot\lib\seq2s
eq_model_utils.py", line 65, in get_predicted_sentence
output_sentence = ' '.join([rev_vocab[output] for output in outputs])
File "C:\Work\tf_seq2seq_chatbot\tf_seq2seq_chatbot\lib\seq2s
eq_model_utils.py", line 65, in
output_sentence = ' '.join([rev_vocab[output] for output in outputs])
IndexError: list index out of range

[fastlater]

Guys,
Any idea why I am getting the error below. I have the file ssd_mobilenet_v1_pets.config in the folder but it says something like he cannot open it without specify the reason why it cannot find the file. I am working on windows, python 3.5.3, tensorflow latest version

Traceback (most recent call last):
File "C:\Program Files (x86)\Python 3.5.2\tensorflow\my codes\objectDetectionOwnTrained_Flowers\train.py", line 202, in
tf.app.run()
File "C:\Program Files (x86)\Python 3.5.2\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "C:\Program Files (x86)\Python 3.5.2\tensorflow\my codes\objectDetectionOwnTrained_Flowers\train.py", line 147, in main
model_config, train_config, input_config = get_configs_from_pipeline_file()
File "C:\Program Files (x86)\Python 3.5.2\tensorflow\my codes\objectDetectionOwnTrained_Flowers\train.py", line 104, in get_configs_from_pipeline_file
text_format.Merge(f.read(), pipeline_config)
File "C:\Program Files (x86)\Python 3.5.2\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 118, in read
self._preread_check()
File "C:\Program Files (x86)\Python 3.5.2\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 78, in _preread_check
compat.as_bytes(self.__name), 1024 * 512, status)
File "C:\Program Files (x86)\Python 3.5.2\lib\contextlib.py", line 66, in exit
next(self.gen)
File "C:\Program Files (x86)\Python 3.5.2\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: ssd_mobilenet_v1_pets.config : �t�Χ䤣�����w���ɮסC

[danwenxuan]

are you slove the problem

[ameyakale603]

Can anyone solve the below errors?

Traceback (most recent call last):
File "export_inference_graph.py", line 106, in
tf.app.run()
File "/home/ameya/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "export_inference_graph.py", line 99, in main
text_format.Merge(f.read(), pipeline_config)
File "/home/ameya/anaconda3/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 118, in read
self._preread_check()
File "/home/ameya/anaconda3/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 78, in _preread_check
compat.as_bytes(self.__name), 1024 * 512, status)
File "/home/ameya/anaconda3/lib/python3.6/contextlib.py", line 89, in exit
next(self.gen)
File "/home/ameya/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: training/to/ssd_inception_v2.config

[Rabin-Rai]

File "train.py", line 184, in
tf.app.run()
File "C:\Users\Robin\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\TensorFlow\models\research\object_detection\trainer.py", line 274, in train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "C:\TensorFlow\models\research\object_detection\trainer.py", line 59, in create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 121, in get_next
dataset_builder.build(config)).get_next()
File "C:\TensorFlow\models\research\object_detection\builders\dataset_builder.py", line 176, in build
num_additional_channels=num_additional_channels)
File "C:\TensorFlow\models\research\object_detection\data_decoders\tf_example_decoder.py", line 267, in init
use_display_name)
File "C:\TensorFlow\models\research\object_detection\utils\label_map_util.py", line 152, in get_label_map_dict
label_map = load_labelmap(label_map_path)
File "C:\TensorFlow\models\research\object_detection\utils\label_map_util.py", line 132, in load_labelmap
label_map_string = fid.read()
File "C:\Users\Robin\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 120, in read
self._preread_check()
File "C:\Users\Robin\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 80, in _preread_check
compat.as_bytes(self.__name), 1024 * 512, status)
File "C:\Users\Robin\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 519, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: C:/tensorflow1/models/research/object_detection/training/labelmap.pbtxt : The system cannot find the path specified.
; No such process

C:\TensorFlow\models\research\object_detection>

[kaaier]

Hello, I want to ask if you have solved the problem

[kaaier]

I am very worry, do you solve it/label_map??????????????/
PBTXT: ϵ ͳ \ udcd5 Ҳ \ udcbb \ udcb5 \ udcbd ָ \ udcb6 \ udca8 \ udcb5 \ udcc4 · \ udcbe \ udcb6 \ udca1 \ udca3