Replace TNC with Phupha word frequency dataset filtered by ORST words #1284
+62,440
−8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Created filtered Phupha dataset with only thai_orst_words (38,160 words) - Added pythainlp/corpus/phupha.py module with word_freqs() and unigram_word_freqs() - Updated pythainlp/spell/pn.py to use Phupha instead of TNC - Updated documentation to reference Phupha dataset Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>
- Changed to use full Phupha dataset (62,264 words) in corpus file - Added filtering logic in pythainlp/spell/pn.py to filter by thai_orst_words - This allows the full Phupha dataset to be available for other uses - Updated tests to verify filtering works correctly - Spell checker now filters 38,160 ORST words from full Phupha dataset Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>
- Added test_issue_680_orst_filtering test case - Documents that compound words not in ORST won't be suggested - Verifies spell checker only uses ORST words - Tests specific case from issue #680: ปลาอินทรีย์ -> ปลาอินทรี - Explains expected behavior when words aren't in ORST dictionary Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>
- Changed 'is A Thai' to 'is a Thai' (lowercase 'a') - Fixed in module docstring and both function docstrings - Addressed code review feedback Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>
|
Copilot
AI
changed the title
[WIP] Add Phupha Thai Word Frequency Dataset to corpus
Replace TNC with Phupha word frequency dataset filtered by ORST words
Feb 5, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do
Replaces Thai National Corpus (TNC) word frequencies in the spell checker with Phupha dataset filtered by Royal Society of Thailand (ORST) word list.
What was wrong
TNC word frequency dictionary captured real-world language usage, including misspelled words. The spell checker trained on this data could suggest incorrect spellings.
How this fixes it
Added Phupha corpus module:
pythainlp/corpus/phupha.py- word frequency accessor functionspythainlp/corpus/phupha_word_freqs.txt- 62,264 words from Common Crawl CorpusModified spell checker:
pythainlp/spell/pn.py- filters Phupha bythai_orst_wordsat initializationExample:
Tests:
test_phupha()- verifies corpus module functionstest_issue_680_orst_filtering()- validates ORST filtering behaviortest_norvig_spell_checker()- checks dictionary size constraintsLimitation: Compound words not in ORST (e.g., "ปลาอินทรีย์") won't be suggested. This is expected when restricting to authoritative dictionary.
Your checklist for this pull request
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.