Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Be robust against invalid utf-8 byte sequences and surrogateescape them when en- or decoding #144

Merged
merged 2 commits into from
Oct 3, 2024
Merged

Be robust against invalid utf-8 byte sequences and surrogateescape them when en- or decoding #144

merged 2 commits into from
Oct 3, 2024

Conversation

pyrco
Copy link
Contributor

@pyrco pyrco commented Sep 30, 2024

This commit also takes the opportunity to remove Python 2 string compatibility code.

It will also remove the final left-over Python 2 compatibility in the test cases.

@pyrco pyrco linked an issue Sep 30, 2024 that may be closed by this pull request
Surrogate escape strings #143
Closed
@codecov Codecov
Copy link

codecov bot commented Sep 30, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 68.18182% with 7 lines in your changes missing coverage. Please review.

Project coverage is 83.89%. Comparing base (8d6fe37) to head (426eadf).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
flow/record/utils.py 63.63% 4 Missing ⚠️
flow/record/adapter/elastic.py 0.00% 1 Missing ⚠️
flow/record/adapter/xlsx.py 50.00% 1 Missing ⚠️
flow/record/fieldtypes/__init__.py 50.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #144      +/-   ##
==========================================
+ Coverage   83.70%   83.89%   +0.19%     
==========================================
  Files          34       34              
  Lines        3485     3458      -27     
==========================================
- Hits         2917     2901      -16     
+ Misses        568      557      -11
Flag Coverage Δ
unittests 83.89% <68.18%> (+0.19%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yunzheng
Copy link
Member

yunzheng commented Sep 30, 2024 •
edited
Loading

has probably nothing to do with the actual changes in the PR, but i'm not sure why pypy3.9 keep failing in the CI.

@pyrco pyrco mentioned this pull request Sep 30, 2024
Merged
@pyrco
Copy link
Contributor Author

pyrco commented Oct 1, 2024 •
edited
Loading

has probably nothing to do with the actual changes in the PR, but i'm not sure why pypy3.9 keep failing in the CI.

There seems to be an issue with a sub-dependency (cramjam) of the avro dependency, which fails building because the installed version of nasm and/or yasm is too old apparently.

Probably a new version got released without wheels for pypy3.9

@pyrco
Copy link
Contributor Author

pyrco commented Oct 1, 2024

Probably a new version got released without wheels for pypy3.9

Jup, cramjam stopped building wheels for pypy3.9

@pyrco pyrco requested a review from Miauwkeru October 1, 2024 11:11
yunzheng
yunzheng requested changes Oct 1, 2024
View reviewed changes
flow/record/stream.py Outdated Show resolved Hide resolved
tests/test_adapter_line.py Outdated Show resolved Hide resolved
tests/test_adapter_text.py Outdated Show resolved Hide resolved
tests/test_record.py Outdated Show resolved Hide resolved
@pyrco
Be robust against invalid utf-8 byte sequences and surrogateescape th…
d019974 
…em when en- or decoding

This commit also takes the opportunity to remove Python 2 string
compatibility code.

It will also remove the final left-over Python 2 compatibility
in the test cases.
@pyrco pyrco requested a review from yunzheng October 3, 2024 06:48
@yunzheng yunzheng merged commit 4e1a285 into main Oct 3, 2024
19 of 20 checks passed
@yunzheng yunzheng deleted the surrogate-escape-strings branch October 3, 2024 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yunzheng yunzheng yunzheng approved these changes

@Miauwkeru Miauwkeru Awaiting requested review from Miauwkeru

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Surrogate escape strings
3 participants
@pyrco @yunzheng @Miauwkeru