Code

hotfix#18

Code

Closed

Saibo-creator wants to merge 34 commits intoepfl-dlab:mainepfl-dlab/transformers-CFG:mainfrom

Saibo-creator:dev_unicode_1Saibo-creator/transformers-CFG:dev_unicode_1Copy head branch name to clipboard

Commits

Commits on Feb 29, 2024

• add unicode test

  Saibo-creator

  committed
  191aa3a

  Copy full SHA for 191aa3a

• support parsing unicode

  Saibo-creator

  committed
  1036680

  Copy full SHA for 1036680

• use dynamical char_acceptance_at_element implementation

  Saibo-creator

  committed
  ab2f2bd

  Copy full SHA for ab2f2bd

• change rule_offsets from dict to array, aligning with llama-cpp

  Saibo-creator

  committed
  f1b6e7e

  Copy full SHA for f1b6e7e

• more flexible GrammarRecognizer construct, aligning with llama-cpp

  Saibo-creator

  committed
  79511b5

  Copy full SHA for 79511b5

• add utf8 decoding utils

  Saibo-creator

  committed
  d0c2ccf

  Copy full SHA for d0c2ccf

• add a bit more on decode utf8, including two NotImplementedError:

  Show description for 1f3807f
  Saibo-creator

  committed
  1f3807f

  Copy full SHA for 1f3807f

• unicode acceptance is working

  Saibo-creator

  committed
  783b1bc

  Copy full SHA for 783b1bc

• fix repetition_penalty issue; add tests for running examples

  Saibo-creator

  committed
  704e5a5

  Copy full SHA for 704e5a5

• v0.1.2

  Saibo-creator

  committed
  bb7225f

  Copy full SHA for bb7225f

• add github workflow, add badges on README and skip example testing locally

  Saibo-creator

  committed
  ebc6c20

  Copy full SHA for ebc6c20

• install pytest dependency in github workflow

  Saibo-creator

  committed
  369cf8c

  Copy full SHA for 369cf8c

• update requirements.txt

  Saibo-creator

  committed
  7d6767d

  Copy full SHA for 7d6767d

• set numpy version to 1.24.2

  Saibo-creator

  committed
  9900dda

  Copy full SHA for 9900dda

• del workflow

  Saibo-creator

  committed
  15f154f

  Copy full SHA for 15f154f

• support parsing unicode

  Saibo-creator

  committed
  b762873

  Copy full SHA for b762873

• use dynamical char_acceptance_at_element implementation

  Saibo-creator

  committed
  af65c70

  Copy full SHA for af65c70

• add utf8 decoding utils

  Saibo-creator

  committed
  c8e9ee6

  Copy full SHA for c8e9ee6

• add a bit more on decode utf8, including two NotImplementedError:

  Show description for a524120
  Saibo-creator

  committed
  a524120

  Copy full SHA for a524120

• unicode acceptance is working

  Saibo-creator

  committed
  15c446d

  Copy full SHA for 15c446d

• fix repetition_penalty issue; add tests for running examples

  Saibo-creator

  committed
  e11c442

  Copy full SHA for e11c442

• add github workflow, add badges on README and skip example testing locally

  Saibo-creator

  committed
  da1afb7

  Copy full SHA for da1afb7

• add test_accept_unicode_bytes.py

  Show description for 90b96f5
  Saibo-creator

  committed
  90b96f5

  Copy full SHA for 90b96f5

• add ByteEncoding

  Saibo-creator

  committed
  1d146e5

  Copy full SHA for 1d146e5

• unicode is working, but very slow and the last stack element has problem

  Saibo-creator

  committed
  222ddf8

  Copy full SHA for 222ddf8

• unicode is working and fast: reset_utf8_decode is reduces the speed from 40s to 15s

  Saibo-creator

  committed
  1ded206

  Copy full SHA for 1ded206

• freeze generate_json.py, repetition_penalty=1.1 is critical

  Saibo-creator

  committed
  be527cf

  Copy full SHA for be527cf

• add generate_c_code.py and fix test_grammar_constrained_decoding.py

  Saibo-creator

  committed
  d22c432

  Copy full SHA for d22c432

• Two modes: unicode(slower) or non-unicode(fast)

  Show description for 6373590
  Saibo-creator

  committed
  6373590

  Copy full SHA for 6373590

• add more languages

  Saibo-creator

  committed
  d2f0ebf

  Copy full SHA for d2f0ebf

• add _accept_string as opposed to _accept_prefix

  Saibo-creator

  committed
  90e066d

  Copy full SHA for 90e066d

• fix unicode mapping issue

  Saibo-creator

  committed
  a7f5e01

  Copy full SHA for a7f5e01

• update README.md and set version to v0.2.0

  Saibo-creator

  committed
  67c7b50

  Copy full SHA for 67c7b50

• remove github workflow for now

  Saibo-creator

  committed
  2320a0b

  Copy full SHA for 2320a0b