Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

event_by_corp_filing errors processing records from LEAR #323

Open
WadeBarnes opened this issue Jun 20, 2023 · 13 comments
Open

event_by_corp_filing errors processing records from LEAR #323

WadeBarnes opened this issue Jun 20, 2023 · 13 comments

Comments

@WadeBarnes
Copy link
Member

Pipeline Logs:

INFO:bcreg.bcreg_core:Caching data for parties and events ...
INFO:bcreg.bcreg_core:Caching data for corporations ...
INFO:bcreg.bcreg_core:Caching data for code tables ...
INFO:bcreg.bcreg_lear:>>> get corp info for: FM0794306,False
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:>>> get corp info for: FM0794306,True
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:    get filings etc ...
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:    get filings etc ...
INFO:bcreg.bcreg_lear:    sort version records for: FM0794306
INFO:bcreg.bcreg_lear:    done.
INFO:bcreg.bcreg_lear:>>> get corp info for: PA0000002,False
INFO:bcreg.bcreg_lear:No corp rec found for PA0000002
ERROR:bcreg.bcreg_lear:year 0 is out of range
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcreg_lear.py", line 89, in default
    ret = o.astimezone(pytz.utc).isoformat()
ValueError: year 0 is out of range
ERROR:bcreg.bcreg_lear:None
ERROR:bcreg.eventprocessor:year 0 is out of range
Traceback (most recent call last):
  File "/data-pipeline/bcreg/eventprocessor.py", line 1929, in process_corp_event_queue_internal
    corp_info_json = bc_registries.to_json(corp_info)
  File "/data-pipeline/bcreg/bcreg_lear.py", line 906, in to_json
    ret = json.dumps(data, cls=CustomJsonEncoder)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/data-pipeline/bcreg/bcreg_lear.py", line 94, in default
    raise error
  File "/data-pipeline/bcreg/bcreg_lear.py", line 89, in default
    ret = o.astimezone(pytz.utc).isoformat()
ValueError: year 0 is out of range
ERROR:bcreg.eventprocessor:None
INFO:bcreg.bcreg_lear:>>> get corp info for: FM0794306,False
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:>>> get corp info for: FM0794306,True
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:    get filings etc ...
INFO:bcreg.bcreg_lear:    got corp rec: 607858,FM0794306,SP
INFO:bcreg.bcreg_lear:    get filings etc ...
INFO:bcreg.bcreg_lear:    sort version records for: FM0794306
INFO:bcreg.bcreg_lear:    done.
INFO:bcreg.bcreg_lear:>>> get corp info for: PA0000002,False
INFO:bcreg.bcreg_lear:No corp rec found for PA0000002
ERROR:bcreg.bcreg_lear:year 0 is out of range
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcreg_lear.py", line 89, in default
    ret = o.astimezone(pytz.utc).isoformat()
ValueError: year 0 is out of range
ERROR:bcreg.bcreg_lear:None
ERROR:bcreg.eventprocessor:year 0 is out of range
Traceback (most recent call last):
  File "/data-pipeline/bcreg/eventprocessor.py", line 1929, in process_corp_event_queue_internal
    corp_info_json = bc_registries.to_json(corp_info)
  File "/data-pipeline/bcreg/bcreg_lear.py", line 906, in to_json
    ret = json.dumps(data, cls=CustomJsonEncoder)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/data-pipeline/bcreg/bcreg_lear.py", line 94, in default
    raise error
  File "/data-pipeline/bcreg/bcreg_lear.py", line 89, in default
    ret = o.astimezone(pytz.utc).isoformat()
ValueError: year 0 is out of range
ERROR:bcreg.eventprocessor:None

Pipeline Status Records:

[
    {
        'record_id': 10639773, 
        'system_type_cd': 'BCREG_LEAR', 
        'corp_num': 'FM0794306', 
        'prev_event_id': 1648671, 
        'prev_event_date': datetime.datetime(2023, 6, 12, 22, 44, 23, 777035), 
        'last_event_id': 1648680, 
        'last_event_date': datetime.datetime(2023, 6, 12, 23, 1, 19, 175234), 
        'entry_date': datetime.datetime(2023, 6, 12, 23, 15, 1, 96251), 
        'process_date': datetime.datetime(2023, 6, 20, 14, 50, 34, 438742), 
        'process_success': 'N', 
        'process_msg': 'year 0 is out of range'
    }, 
    {
        'record_id': 10639628, 
        'system_type_cd': 'BCREG_LEAR', 
        'corp_num': 'FM0794306', 
        'prev_event_id': 1648645, 
        'prev_event_date': datetime.datetime(2023, 6, 12, 22, 10, 41, 927270), 
        'last_event_id': 1648671, 
        'last_event_date': datetime.datetime(2023, 6, 12, 22, 44, 23, 777035), 
        'entry_date': datetime.datetime(2023, 6, 12, 22, 45, 1, 217412), 
        'process_date': datetime.datetime(2023, 6, 20, 14, 50, 34, 423968), 
        'process_success': 'N', 
        'process_msg': 'year 0 is out of range'
    }
]

cc @esune, @amanji, @ianco
@swcurran
Copy link
Contributor

This is a problem with 2 records? Are the rest processing OK? Any other information beyond the logs to provide, @WadeBarnes ?

BC Registries has in the past used a date of 0 to trigger magic handling, so my guess is this is an instance of that.

@thorwolpert — could you please raise a flag with other BC Registries folks about this? I don’t know/can’t find the GitHub ID of folks like Dwayne and David Roberts.

@WadeBarnes
Copy link
Member Author

Yes, just an issue with the two records. All others are fine. No additional information yet. I have not looked at the data or associated code yet.

@swcurran
Copy link
Contributor

OK — I think the next step is to have BC Reg look at this entity (it’s one sole prop) on their side and see what is magic about it. Their registration is in OrgBook BC — https://orgbook.gov.bc.ca/entity/FM0794306

I don’t see any new records associated with the entity, which is what is implied with these process failures.

@WadeBarnes
Copy link
Member Author

WadeBarnes commented Jun 20, 2023 •
edited
Loading

BTW, this issue is unrelated to bcgov/aries-vcr#751 which was causing credential processing (posting) errors.

@swcurran
Copy link
Contributor

Email sent to Registries to ask them to look into the issue.

@WadeBarnes
Copy link
Member Author

Code that is failing:

von-bc-registries-agent/data-pipeline/bcreg/bcreg_lear.py

Lines 85 to 94 in ee1aa1e

if isinstance(o, datetime.datetime):
try:
# tz_aware = timezone.localize(o)
# ret = tz_aware.astimezone(pytz.utc).isoformat()
ret = o.astimezone(pytz.utc).isoformat()
return ret
except (Exception) as error:
LOGGER.error(error)
LOGGER.error(traceback.print_exc())
raise error

@WadeBarnes
Copy link
Member Author

WadeBarnes commented Jun 20, 2023 •
edited
Loading

Based on feedback from BC Registries PA0000002 has all it's dates set to 0001-01-01 which cause the error we're seeing. It's a little strange though, that the code is flagging the records for FM0794306 and not PA0000002 with the error.

We are waiting for a response from BC Registries as to why all the dates are set to 0001-01-01 on PA0000002.

@esune
Copy link
Member

esune commented Jul 7, 2023

@WadeBarnes @swcurran any updates on this? Can we close the issue if the problem is with the data, or do we still have something we need to be looking into?

@WadeBarnes
Copy link
Member Author

WadeBarnes commented Jul 7, 2023 •
edited
Loading

I forwarded you some additional information. Some decisions need to be made as to how we handle these situations. It looks like the BC Reg code may handle these issues, but the LEAR code does not contain the same logic.

@WadeBarnes
Copy link
Member Author

@esune, have we looked into fixing this issue yet?

@esune
Copy link
Member

esune commented Aug 30, 2023

I believe this was being tackled by @amanji together with the other fixes to the pipeline - see https://app.zenhub.com/workspaces/bcgov-ditp-products-team-63c987121278d5001ecb177b/issues/zh/41

I believe the changes have been merged and are waiting for validation and promotion.

@WadeBarnes
Copy link
Member Author

@amanji, is there a PR associated to this ticket?

@esune
Copy link
Member

esune commented Nov 2, 2023

Checked-in with @amanji, this is still an issue. We might want to deal with it as part of the rewrite in #334

@swcurran swcurran moved this to Assignment Ready in CDT Enterprise Apps Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
CDT Enterprise Apps
Status: Assignment Ready
Milestone
No milestone
Development

No branches or pull requests
3 participants
@swcurran @esune @WadeBarnes