Shopify acceptance tests fail and block the PRs

[rpopov]

Running the CAT tests locally on:

• Oracle Linux 9.5
• Oracle Linux 8.7
• Fedora Workstation 41

Observations

• Running the CDK acceptance tests locally with:

cd airbyte/airbyte-integrations/bases/connector-acceptance-test
poetry run pytest -p connector_acceptance_test.plugin --acceptance-test-config=../../connectors/source-shopify --timeout=300

fails with the container (Docker image: registry.dagger.io/engine v0.15.1) errors:

# expect more open files due to per-client SQLite databases
# many systems default to 1024 which is far too low
ulimit -n 1048576 || echo "cannot increase open FDs with ulimit, ignoring"
... ... ...
buildkitd: failed to create engine: failed to create network providers: CNI setup error: plugin type="bridge" failed (add): failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 3: iptables v1.8.10 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
iptables v1.8.10 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

• Namely:

  • The internally executed /usr/sbin/iptables -t nat -S --wait command fails with exit code 3 which does not allow the container to initialize and terminates it.
  • man iptables states: "Errors which indicate an incompatibility between kernel and user space cause an exit code of 3".
  • ulimit -n 1048576 used to fail when used podman instead of docker.

• The same failure seems to happen in the pipelines of the Pull requests. Examples:

  • fix(declarative): Pass extra_fields in global_substream_cursor #195
  • chore(deps): bump openai from 0.27.9 to 1.59.2 #196
  • feat: Publish the response to process in the context of the record's transformation and filtering steps #193

• The logs report the same:

...
=========================== short test summary info ============================
SKIPPED [1] ../app/connector_acceptance_test/plugin.py:65: Skipping TestConnectorAttributes.test_streams_define_primary_key: not found in the config.
SKIPPED [1] ../app/connector_acceptance_test/plugin.py:65: Skipping TestConnectorDocumentation.test_prerequisites_content: not found in the config.
SKIPPED [1] ../app/connector_acceptance_test/tests/test_core.py:142: Config is not provided
SKIPPED [1] ../app/connector_acceptance_test/tests/test_core.py:121: The previous and actual specifications are identical.
SKIPPED [1] ../app/connector_acceptance_test/tests/test_core.py:803: This tests currently leads to too much failures. We need to fix the connectors at scale first.
SKIPPED [2] ../app/connector_acceptance_test/tests/test_core.py:1269: Skipping the test for supported file types as it is only applicable for certified file-based connectors.
ERROR test_core.py::TestDiscovery::test_streams_have_valid_json_schemas[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - Val...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - Val...
ERROR test_core.py::TestDiscovery::test_primary_keys_data_type[inputs0] - Val...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - ValueError: No catalo...
ERROR test_core.py::TestBasicRead::test_read[inputs1] - ValueError: No catalo...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: Co...
FAILED test_core.py::TestConnection::test_check[inputs2] - AssertionError: Co...
FAILED test_core.py::TestConnection::test_check[inputs4] - AssertionError: Co...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - IndexError: list...
== 4 failed, 28 passed, 7 skipped, 8 warnings, 16 errors in 114.62s (0:01:54) ==

• In contrast, the following PRs do not report the error:
  • feat: add CustomSchemaNormalization #194
  • fix: Async Retriever change url path for download retriever #192

Suggestion

• Please investigate the issue of running the Dagger container locally. It maybe causes the mass errors reported.
• Please investigate if there is dependency between the host/OS the container is run on. That could explain why those tests may succeed (if run on hosts with different OS).
• Please update the documentation, specifically:
  • Acceptance Tests Reference
  • Connector Acceptance Tests
    with instructions how to resolve those issues locally.

[rpopov]

The problem is caused by the fact that I used:

• Oracle Linux 9.5
• Oracle Linux 8.7
• Fedora Workstation 41
  All of them are RHEL versions. Dagger - Podman integration and Dagger Troubleshooting advises executing:

# just for the current session, until restart
sudo modprobe iptable_nat

# include the nat module to survive restart
echo iptable_nat | sudo tee -a /etc/modules-load.d/modules

This solves the problem.

[rpopov]

The tests still fail. These are:

• test_image_labels in airbyte/airbyte-integrations/bases/connector-acceptance-test/connector_acceptance_test/tests/test_core.py:
  • This test is commented as:

# This test should not be part of TestSpec because it's testing the connector's docker image content, not the spec itself
# But it's cumbersome to declare a separate, non configurable, test class
# See https://github.com/airbytehq/airbyte/issues/15551

• It checks if the last image of the specific connector is referred in the test configuration with the actual version. In the example of shopify, it compares image versionconnector_image: airbyte/source-shopify:dev in airbyte/airbyte-integrations/connectors/source-shopify/acceptance-test-config.yml (dev) matches the version set in dockerImageTag field in airbyte/airbyte-integrations/connectors/source-shopify/metadata.yaml (2.5.15).
• This test logic depends only in the specific connector's code in the root project and does not depend on the changes in the CDK porject.
• As shown in the shopify example, the "dev" version does not match "2.5.15", so this test fails.
• test_check in airbyte/airbyte-integrations/bases/connector-acceptance-test/connector_acceptance_test/tests/test_core.py
  • This test verifies that the number of connection messages in success/failure status are exactly 1, extracting them from the logs.
  • This test highly depends on the infrastructure's performance and container caching (when run locally).
• test_airbyte_trace_message_on_failure to "Ensure that a trace message is emitted when the connector crashes". It fails when run locally.

All these tests seem to be flaky, dependent on the infrastructure and the development practices. They seem to be not related to the CDK changes, but blocking them.