TypeError: can't pickle _cffi_backend.CTypeDescr objects

[jayvdb]

First use fails

> privacyscanner update_dependencies
Traceback (most recent call last):
  File "/usr/bin/privacyscanner", line 11, in <module>
    load_entry_point('privacyscanner==0.8.0', 'console_scripts', 'privacyscanner')()
  File "/usr/lib/python3.7/site-packages/privacyscanner/scanner.py", line 289, in main
    args.func(args)
  File "/usr/lib/python3.7/site-packages/privacyscanner/scanner.py", line 97, in scan_site
    config = load_config(args.config)
  File "/usr/lib/python3.7/site-packages/privacyscanner/scanner.py", line 44, in load_config
    config = deepcopy(defaultconfig.__dict__)
  File "/usr/lib64/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python3.7/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib64/python3.7/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib64/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle _cffi_backend.CTypeDescr objects

This occurs because the module __dict__ can include lots of interesting things added by hooks during the module loading process. In my environment, __dict__ contains.

{'__cffi_backend_extern_py': {140569175620256: (<ctype 'int(*)(void *, int)'>, <bound method AbstractCallbacks.python_callback of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>, b'\x00\x00\x00\x00\x00\x00\x00\x00', <bound method AbstractCallbacks.unhandled_onerror of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>), 140569175620192: (<ctype 'void(*)(void *, int)'>, <bound method AbstractCallbacks.python_handle_error of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>, b'\x00\x00\x00\x00\x00\x00\x00\x00', <bound method AbstractCallbacks.unhandled_onerror of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>), 140569175620128: (<ctype 'void(*)(void *)'>, <bound method AbstractCallbacks.python_stop of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>, b'\x00\x00\x00\x00\x00\x00\x00\x00', <bound method AbstractCallbacks.unhandled_onerror of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>), 140569175620224: (<ctype 'void(*)(struct ev_loop *, void *, int)'>, <bound method _Callbacks.python_check_callback of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>, b'\x00\x00\x00\x00\x00\x00\x00\x00', <bound method AbstractCallbacks.check_callback_onerror of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>), 140569175620160: (<ctype 'void(*)(struct ev_loop *, void *, int)'>, <bound method _Callbacks.python_prepare_callback of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>, b'\x00\x00\x00\x00\x00\x00\x00\x00', <bound method AbstractCallbacks.check_callback_onerror of <gevent.libev.corecffi._Callbacks object at 0x7fd8cfcbbeb8>>), 140569175620288: (<ctype 'void(*)(char *)'>, <function _syserr_cb at 0x7fd8d1019d08>, b'\x00\x00\x00\x00\x00\x00\x00\x00', None)}}}

Worth nothing that trying to deepcopy a module is not supported, and using its __dict__ is not any saner.

>>> import privacyscanner.defaultconfig
>>> from copy import deepcopy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle module objects