how do I use TiParser lexer?

[david-sail]

Question

how do I call TiParser lexer for sql word split just like vitess token not ast parser, the sample code is as follows

var tokens []Token
tkn := sqlparser.NewStringTokenizer(sql)
typ, val := tkn.Scan()

[tangenta]

Here is an example:

cat example_test.go

package parser

import (
	"fmt"
	. "github.com/pingcap/check"
)

var _ = Suite(&testExampleSuite{})

type testExampleSuite struct {
}

func (s *testExampleSuite) TestExample(c *C) {
	l := NewScanner("create table t (a int);")
	var v yySymType
	for {
		result := l.Lex(&v)
		if result == 0 || result == invalid {
			break
		}
		fmt.Printf("%d\n", result)
	}
}

go test -v github.com/pingcap/parser -check.f TestExample

=== RUN   TestT
57382
57531
57346
40
57346
57446
41
59
PASS: example_test.go:13: testExampleSuite.TestExample  0.001s

The meaning of these numbers can be found in parser.go.

[zjcxc]

yySymType is a private struct, how can i use it in my program?