Sinking course data fails with an exception

[snglth]

I ran tutor k8s do dump-data-to-clickhouse --options "--object course_overviews --force" which created a K8s job. The job failed spewing following exception:

Traceback (most recent call last):
  File "./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/management/commands/dump_data_to_clickhouse.py", line 198, in handle
    dump_target_objects_to_clickhouse(
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/management/commands/dump_data_to_clickhouse.py", line 77, in dump_target_objects_to_clickhouse
    sink.dump(objects_to_submit, many=True)
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/base_sink.py", line 187, in dump
    nested_sink.dump_related(
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/course_overview_sink.py", line 48, in dump_related
    self.dump(
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/base_sink.py", line 176, in dump
    serialized_item = self.serialize_item(item_id, many=many, initial=initial)
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/course_overview_sink.py", line 113, in serialize_item
    items = self.get_xblocks_recursive(course_block, detached_xblock_types, initial)
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/course_overview_sink.py", line 63, in get_xblocks_recursive
    self.serialize_xblock(
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/course_overview_sink.py", line 166, in serialize_xblock
    course_key = item.scope_ids.usage_id.course_key
AttributeError: 'NoneType' object has no attribute 'scope_ids'

[Ian2012]

Hi @snglth, can you leave details of the following information:

• Open edx version (branch)
• Tutor version
• Aspects version + platform-plugin-aspects version

[snglth]

Sure:

• Open edx: open-release/quince.3
• Tutor: 17.0.6
• Aspects: 1.1.0
• platform-plugin-aspects: 0.10.0

[Ian2012]

Does it happens with all courses or fails only with some?

[snglth]

I think the error occurs on one specific course. Here's the full job log:

2024-08-27 19:02:57,879 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:177 - Now dumping 19 Course Overview to ClickHouse
2024-08-27 19:02:58,271 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:181 - Completed dumping 19 Course Overview to ClickHouse
2024-08-27 19:02:58,443 WARNING 8 [py.warnings] [user None] [ip None] warnings.py:109 - /openedx/edx-platform/xmodule/mako_block.py:49: DeprecationWarning: Use of runtime.render_template is deprec
ated. Use MakoService.render_template or a JavaScript-based template instead.
  if getattr(self.runtime, 'render_template', None) is None:

2024-08-27 19:02:58,672 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:177 - Now dumping 153 XBlock to ClickHouse
2024-08-27 19:02:58,692 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:181 - Completed dumping 153 XBlock to ClickHouse
2024-08-27 19:02:58,857 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:177 - Now dumping 165 XBlock to ClickHouse
2024-08-27 19:02:58,875 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:181 - Completed dumping 165 XBlock to ClickHouse
2024-08-27 19:02:58,909 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:177 - Now dumping 29 XBlock to ClickHouse
2024-08-27 19:02:58,922 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:181 - Completed dumping 29 XBlock to ClickHouse
2024-08-27 19:02:58,941 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:177 - Now dumping 12 XBlock to ClickHouse
2024-08-27 19:02:58,952 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:181 - Completed dumping 12 XBlock to ClickHouse
2024-08-27 19:02:59,174 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:177 - Now dumping 182 XBlock to ClickHouse
2024-08-27 19:02:59,192 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:181 - Completed dumping 182 XBlock to ClickHouse
2024-08-27 19:02:59,346 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:177 - Now dumping 129 XBlock to ClickHouse
2024-08-27 19:02:59,363 INFO 8 [platform_plugin_aspects.management.commands.dump_data_to_clickhouse] [user None] [ip None] base_sink.py:181 - Completed dumping 129 XBlock to ClickHouse
Traceback (most recent call last):
  File "./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/management/commands/dump_data_to_clickhouse.py", line 198, in handle
    dump_target_objects_to_clickhouse(
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/management/commands/dump_data_to_clickhouse.py", line 77, in dump_target_objects_to_clickhouse
    sink.dump(objects_to_submit, many=True)
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/base_sink.py", line 187, in dump
    nested_sink.dump_related(
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/course_overview_sink.py", line 48, in dump_related
    self.dump(
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/base_sink.py", line 176, in dump
    serialized_item = self.serialize_item(item_id, many=many, initial=initial)
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/course_overview_sink.py", line 113, in serialize_item
    items = self.get_xblocks_recursive(course_block, detached_xblock_types, initial)
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/course_overview_sink.py", line 63, in get_xblocks_recursive
    self.serialize_xblock(
  File "/openedx/venv/lib/python3.8/site-packages/platform_plugin_aspects/sinks/course_overview_sink.py", line 166, in serialize_xblock
    course_key = item.scope_ids.usage_id.course_key
AttributeError: 'NoneType' object has no attribute 'scope_ids'

[Ian2012]

Seems like it's trying to serialize a block that's empty or has been deleted, can you try to find the course that it's trying to serialize and check anything strange?

If possible, you can share the XML the course and we can try to debug the issue

[snglth]

It was caused by course overview that was linked to nonexistent course. Deleting that overview resolved the issue.