encoding issues with native lexer and Ruby2.1

[boutil]

Hi,

when trying to rebuild the Debian package for ruby-gherkin with support for ruby2.1, I stumbled upon the following test failures in spec/gherkin/native_lexer_spec.rb

Failures:

  1) Native Lexer it should behave like a Gherkin lexer lexing tags should lex UTF-8 tags
     Failure/Error: @lexer.scan(gherkin)
       #<Gherkin::SexpRecorder:0x000000017d1328> received :tag with unexpected arguments
         expected: ("@シナリオテンプレート", 1)
              got: ("@\xE3\x82\xB7\xE3\x83\x8A\xE3\x83\xAA\xE3\x82\xAA\xE3\x83\x86\xE3\x83\xB3\xE3\x83\x97\xE3\x83\xAC\xE3\x83\xBC\xE3\x83\x88", 1)
     Shared Example Group: "a Gherkin lexer lexing tags" called from ./spec/gherkin/native_lexer_spec.rb:27
     # ./lib/gherkin/lexer/en.rb:995:in `scan'
     # ./spec/gherkin/shared/tags_group.rb:6:in `scan'
     # ./spec/gherkin/shared/tags_group.rb:21:in `block (2 levels) in <module:Lexer>'

  2) Native Lexer it should behave like a Gherkin lexer lexing tags should lex mixed tags
     Failure/Error: @lexer.scan(gherkin)
       #<Gherkin::SexpRecorder:0x00000001797308> received :tag with unexpected arguments
         expected: ("@Значения", 1)
              got: ("@\xD0\x97\xD0\xBD\xD0\xB0\xD1\x87\xD0\xB5\xD0\xBD\xD0\xB8\xD1\x8F", 1)
     Shared Example Group: "a Gherkin lexer lexing tags" called from ./spec/gherkin/native_lexer_spec.rb:27
     # ./lib/gherkin/lexer/en.rb:995:in `scan'
     # ./spec/gherkin/shared/tags_group.rb:6:in `scan'
     # ./spec/gherkin/shared/tags_group.rb:27:in `block (2 levels) in <module:Lexer>'

  3) Native Lexer it should behave like a Gherkin lexer lexing rows should allow utf-8
     Failure/Error: ]
       expected: [[:row, ["ůﻚ", "2"], 1], [:eof]]
            got: [[:row, ["\xC5\xAF\xEF\xBB\x9A", "2"], 1], [:eof]] (using ==)
     Shared Example Group: "a Gherkin lexer lexing rows" called from ./spec/gherkin/native_lexer_spec.rb:29
     # ./spec/gherkin/shared/row_group.rb:51:in `block (2 levels) in <module:Lexer>'

  4) Native Lexer it should behave like a Gherkin lexer lexing rows should allow utf-8 using should_receive
     Failure/Error: @lexer.scan(gherkin)
       #<Gherkin::SexpRecorder:0x0000000142e450> received :row with unexpected arguments
         expected: (r ["繁體中文  而且", "並且", "繁體中文  而且", "並且"], 1)
              got: (r ["\xE7\xB9\x81\xE9\xAB\x94\xE4\xB8\xAD\xE6\x96\x87  \xE8\x80\x8C\xE4\xB8\x94", "\xE4\xB8\xA6\xE4\xB8\x94", "\xE7\xB9\x81\xE9\xAB\x94\xE4\xB8\xAD\xE6\x96\x87  \xE8\x80\x8C\xE4\xB8\x94", "\xE4\xB8\xA6\xE4\xB8\x94"], 1)
     Shared Example Group: "a Gherkin lexer lexing rows" called from ./spec/gherkin/native_lexer_spec.rb:29
     # ./lib/gherkin/lexer/en.rb:1054:in `scan'
     # ./spec/gherkin/shared/row_group.rb:6:in `scan'
     # ./spec/gherkin/shared/row_group.rb:56:in `block (2 levels) in <module:Lexer>'

Finished in 0.10166 seconds
90 examples, 4 failures, 2 pending

Failed examples:

rspec ./spec/gherkin/shared/tags_group.rb:19 # Native Lexer it should behave like a Gherkin lexer lexing tags should lex UTF-8 tags
rspec ./spec/gherkin/shared/tags_group.rb:24 # Native Lexer it should behave like a Gherkin lexer lexing tags should lex mixed tags
rspec ./spec/gherkin/shared/row_group.rb:46 # Native Lexer it should behave like a Gherkin lexer lexing rows should allow utf-8
rspec ./spec/gherkin/shared/row_group.rb:54 # Native Lexer it should behave like a Gherkin lexer lexing rows should allow utf-8 using should_receive
/usr/bin/ruby2.1 -S rspec ./spec/gherkin/native_lexer_spec.rb failed

Those tests pass with Ruby2.0.

Looks like an encoding issue, but I couldn't see what changed from ruby2.0 to ruby2.1 that could cause that.

(I extended the regexp in spec/spec_helper.rb to set the value of Encoding.default_* to Ruby2.1, but this had no (positive) effect).

[johnvoloski]

@boutil

you still have problem? or this issue can be closed?
I added the ruby's versions 2.1.* on travis-ci in this Pull Request.

[boutil]

Hi @johnvoloski,
the problem disappeared with the changes you merge. You can close the issue. Thanks!