Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: use new --use-local-cdk method for testing connectors #40

Merged
merged 22 commits into from
Nov 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 46 additions & 9 deletions .github/workflows/connector-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@ jobs:
fail-fast: false
matrix:
include:
- connector: source-faker
cdk_extra: n/a
- connector: source-shopify
cdk_extra: n/a
# Currently not passing CI (unrelated)
Expand All @@ -87,31 +89,66 @@ jobs:
if: ${{ matrix.cdk_extra != 'n/a' && needs.cdk_changes.outputs[matrix.cdk_extra] == 'false' }}
run: |
echo "Aborting job as specified extra not changed: ${{matrix.cdk_extra}} = ${{ needs.cdk_changes.outputs[matrix.cdk_extra] }}"
echo "> Skipped '${{matrix.connector}}' (no relevant changes)" >> $GITHUB_STEP_SUMMARY
echo "status=cancelled" >> $GITHUB_OUTPUT
exit 1
exit 0
continue-on-error: true
# Get the monorepo so we can test the connectors
- name: Checkout CDK
if: steps.no_changes.outcome != 'failure'
if: steps.no_changes.outputs.status != 'cancelled'
uses: actions/checkout@v4
with:
path: airbyte-python-cdk
- name: Checkout Airbyte Monorepo
uses: actions/checkout@v4
if: steps.no_changes.outcome != 'failure'
if: steps.no_changes.outputs.status != 'cancelled'
with:
repository: airbytehq/airbyte
ref: master
# TODO: Revert to `master` after Airbyte CI released:
ref: aj/airbyte-ci/update-python-local-cdk-code
path: airbyte
- name: Test Connector
if: steps.no_changes.outcome != 'failure'
if: steps.no_changes.outputs.status != 'cancelled'
timeout-minutes: 90
env:
GCP_GSM_CREDENTIALS: ${{ secrets.GCP_GSM_CREDENTIALS }}
POETRY_DYNAMIC_VERSIONING_BYPASS: "0.0.0"
# TODO: Revert below to use `tools.airbyte-ci-binary.install` after Airbyte CI released:
run: |
cd airbyte
make tools.airbyte-ci-binary.install
airbyte-ci connectors \
make tools.airbyte-ci-dev.install
airbyte-ci-dev connectors \
--name ${{matrix.connector}} \
test
--global-status-check-context='Connectors Test: ${{matrix.connector}}'
--use-local-cdk \
test \
--fail-fast \
--skip-step qa_checks \
--skip-step connector_live_tests

# Upload the job output to the artifacts
- name: Upload Job Output
id: upload_job_output
if: always() && steps.no_changes.outputs.status != 'cancelled'
uses: actions/upload-artifact@v4
with:
name: ${{matrix.connector}}-job-output
path: airbyte/airbyte-ci/connectors/pipelines/pipeline_reports

- name: Evaluate Job Output
if: always() && steps.no_changes.outputs.status != 'cancelled'
run: |
# save job output json file as ci step output
json_output_file=$(find airbyte/airbyte-ci/connectors/pipelines/pipeline_reports -name 'output.json' -print -quit)
job_output=$(cat ${json_output_file})
success=$(echo ${job_output} | jq -r '.success')
failed_jobs=$(echo ${job_output} | jq -r '.failed_steps')
run_duration=$(echo ${job_output} | jq -r '.run_duration')
echo "## Job Output for ${{matrix.connector}}" >> $GITHUB_STEP_SUMMARY
echo "- Success: ${success}" >> $GITHUB_STEP_SUMMARY
echo "- Test Duration: $(printf "%.0f" ${run_duration})s" >> $GITHUB_STEP_SUMMARY
echo "- Failed Checks: ${failed_jobs}" >> $GITHUB_STEP_SUMMARY
echo -e "\n[Download Job Output](${{steps.upload_job_output.outputs.artifact-url}})" >> $GITHUB_STEP_SUMMARY
if [ "${success}" != "true" ]; then
# Throw failure if tests failed
exit 1
fi
14 changes: 10 additions & 4 deletions airbyte_cdk/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -282,12 +282,18 @@
"StreamSlice",
]

__version__ = _dunamai.get_version(
"airbyte-cdk",
third_choice=_dunamai.Version.from_any_vcs,
).serialize()
__version__: str
"""Version generated by poetry dynamic versioning during publish.

When running in development, dunamai will calculate a new prerelease version
from existing git release tag info.
"""

try:
__version__ = _dunamai.get_version(
"airbyte-cdk",
third_choice=_dunamai.Version.from_any_vcs,
fallback=_dunamai.Version("0.0.0+dev"),
).serialize()
except:
__version__ = "0.0.0+dev"
Loading