Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: LANGUAGE encoding types #29

Merged
merged 60 commits into from
Feb 18, 2025
Merged

feat: LANGUAGE encoding types #29

merged 60 commits into from
Feb 18, 2025

Conversation

lukaszkolodziejczyk
Copy link
Contributor

No description provided.

radurogojanumai and others added 30 commits February 5, 2025 14:12
@radurogojanumai
feat: add support for language categorical encoding and analysis, wip1
2aff54a
@lukaszkolodziejczyk
wip
1c7d547
@lukaszkolodziejczyk
language_numeric
a16a6a1
@lukaszkolodziejczyk
language_numeric
827679a
@lukaszkolodziejczyk
language_numeric
2e10629
@lukaszkolodziejczyk
language_numeric
e4e0e4f
@andre-mostly
add test and beginnings of numeric
a9b681d
@lukaszkolodziejczyk
language_numeric
ab5ce5a
@lukaszkolodziejczyk
fix
52c8303
@lukaszkolodziejczyk
remove unnecessary
ce61fed
@andre-mostly
change test to AMD, fix max_decimal
e2e23a4
@andre-mostly
fix decimal
988d3c8
@lukaszkolodziejczyk
max_scale
e3dc810
@andre-mostly
encode string/numeric
926ec2b
@andre-mostly
placeholder datetime
abda5d5
@andre-mostly
add dt test asserts
337003c
@lukaszkolodziejczyk
datetime
c668e67
@andre-mostly
add coercion
8d5e764
@andre-mostly
fix invalid to nan for numeric
5913da9
@andre-mostly
coerce datetimes to valid dates, add datetime grammar, change test to…
87fb34f 
… use gpt2 for now. temporarily disable categorical test due to error
@andre-mostly
comments and refactor
548f11a
@michdr
tiny refactor
31fea13
@andre-mostly
add back categorical test
6228782
@michdr
Merge branch 'main' into msd-965-language-enctypes
fea28fb
@lukaszkolodziejczyk
Simpler categorical analyze / analyze reduce; rare category protection (
386d976 
#32)
@lukaszkolodziejczyk
expose rare_category_replacement_method for LANGUAGE
6afe58f
@shuangwu5
MSD-XXX: add initial alphabets to untrained tokenizer if needed (#33)
f7948bc
@lukaszkolodziejczyk
simplify numeric
ac2c1cc
@shuangwu5
fix tests (#34)
65b9dd6
@shuangwu5
refactor temp_formatron.py (#35)
a1bdade
lukaszkolodziejczyk and others added 27 commits February 11, 2025 17:01
@lukaszkolodziejczyk
extreme value protection for datetimes
7456760
@shuangwu5
build: uv run without re-syncing the environment (#37)
28516f6
@michdr
temp fix for datetime validation
5d8011f
@michdr
enhance test_categorical_numeric_datetime
4eccc0d
@michdr
enable all models in test_categorical_numeric_datetime
d52772e
@michdr
fix _decode_datetime
e343ace
@andre-mostly
feat: constrain numeric and simplify datetime (#38)
9453080
@michdr
refactor: move language decode functions + tabular encoding types uni…
d66670d 
…t tests
@michdr
add decode_text
ccbb1e0
@michdr
ruff
c643d25
@michdr
Merge branch 'main' into msd-965-language-enctypes
8002963
@andre-mostly
restrict number to correct number of decimal points in grammar, negat…
88b5772 
…ive/positive, and number of digits, and whether 0 can be a member
@andre-mostly
fix datetime pattern
8a3506c
@michdr
add unit tests + improve decode_numeric
6ee92e1
@andre-mostly
fix monkey patch
9c20ee2
@andre-mostly
remove temp_formatron.py and move code into formatron_utils.py
089cf62
@lukaszkolodziejczyk
fix numeric training
6692258
@andre-mostly
make max5, min5 maintain numeric dtype (int for int and float for flo…
da28b5c 
…at) rather than always float
@mplatzer
added description for new enc types
7b4bc2f
@michdr
re-add disabled models in test_categorical_numeric_datetime
6e00e98
@andre-mostly
fix comments
743ec3e
@lukaszkolodziejczyk
refactor analyze
77e55c0
@lukaszkolodziejczyk
LANGUAGE CATEGORICAL (#43)
33a3b1e
@lukaszkolodziejczyk
n_jobs
7875908
@lukaszkolodziejczyk
kill examples/language_encoding_types.ipynb
be41a73
@lukaszkolodziejczyk
datetime tests
d0452c4
@lukaszkolodziejczyk
numeric tests
2548431
@lukaszkolodziejczyk lukaszkolodziejczyk marked this pull request as ready for review February 18, 2025 09:03
@mplatzer mplatzer merged commit 5f3b54b into main Feb 18, 2025
7 checks passed
@mplatzer mplatzer deleted the msd-965-language-enctypes branch February 18, 2025 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@lukaszkolodziejczyk @mplatzer @radurogojanumai @andre-mostly @michdr @shuangwu5