Trouble with unicode character encodings...

[mfeblowitz]

Encountering errors in running predict_network.py, e.g.,

File "predict_network.py", line 327, in <module>
    main()
  File "predict_network.py", line 121, in main
    tmp_note = TimeNote(tml, tml)
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/TimeNote.py", line 46, in __init__
    tokenized_text, token_to_offset, sentence_features, dependency_paths, id_to_tok = pre_processing.pre_process(self.original_text)
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/utilities/pre_processing/pre_processing.py", line 26, in pre_process
    naf_tagged_doc = pre_processor.pre_process(text) # output a string of xml doc
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/utilities/pre_processing/news_reader.py", line 68, in pre_process
    srl_text = srl.parse_dependencies(constituency_parsed_text)
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/utilities/pre_processing/news_reader.py", line 124, in parse_dependencies
    return SRLClient.parse_dependencies(naf_tokenized_pos_tagged_text)
  File "/homes/hny2/mfeb/causal/event-based-reasoning/code/notes/utilities/pre_processing/news_reader.py", line 159, in parse_dependencies
    _output, _ = srl.communicate(naf_tokenized_pos_tagged_text)
  File "/homes/hny2/mfeb/anaconda3/envs/py27/lib/python2.7/subprocess.py", line 479, in communicate
    return self._communicate(input)
  File "/homes/hny2/mfeb/anaconda3/envs/py27/lib/python2.7/subprocess.py", line 1102, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/homes/hny2/mfeb/anaconda3/envs/py27/lib/python2.7/subprocess.py", line 1166, in _communicate_with_poll
    input_offset += os.write(fd, chunk)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1787: ordinal not in range(128)

This is just one of many such errors.

Web postings indicate that the use of str() is the culprit in these cases, recommending the use of, e.g., .encode('utf-8'), instead. It's not clear, though, where the root cause might be. The trace indicates that it's writing the results of the parsing subprocess.

Any recommendations for addressing this?

[ylmeng]

Yes usually such problems are caused by encoding issues with some special characters.
On the one hand, you can do some preprocessing to convert the unicode characters to ascii characters in all training and test files.
On the other hand, you may try using encode('utf8') with all the unicode strings. For the model per se, such special characters are unimportant, so just do whatever you feel convenient.