Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: use new --use-local-cdk method for testing connectors #40

Merged
merged 22 commits into from
Nov 13, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 38 additions & 5 deletions .github/workflows/connector-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -101,17 +101,50 @@ jobs:
if: steps.no_changes.outcome != 'failure'
with:
repository: airbytehq/airbyte
ref: master
# TODO: Revert to `master` after Airbyte CI released:
ref: aj/airbyte-ci/update-python-local-cdk-code
path: airbyte
- name: Test Connector
if: steps.no_changes.outcome != 'failure'
timeout-minutes: 90
env:
GCP_GSM_CREDENTIALS: ${{ secrets.GCP_GSM_CREDENTIALS }}
POETRY_DYNAMIC_VERSIONING_BYPASS: "0.0.0"
# TODO: Revert below to use `tools.airbyte-ci-binary.install` after Airbyte CI released:
run: |
cd airbyte
make tools.airbyte-ci-binary.install
airbyte-ci connectors \
make tools.airbyte-ci-dev.install
airbyte-ci-dev connectors \
--name ${{matrix.connector}} \
test
--global-status-check-context='Connectors Test: ${{matrix.connector}}'
--use-local-cdk \
test \
--fail-fast \
--skip-step qa_checks

# Upload the job output to the artifacts
- name: Upload Job Output
id: upload_job_output
if: always()
uses: actions/upload-artifact@v4
with:
name: ${{matrix.connector}}-job-output
path: airbyte/airbyte-ci/connectors/pipelines/pipeline_reports

- name: Evaluate Job Output
if: always()
run: |
# save job output json file as ci step output
json_output_file=$(find airbyte/airbyte-ci/connectors/pipelines/pipeline_reports -name 'output.json' -print -quit)
job_output=$(cat $json_output_file)
echo "job_output=$job_output" >> $GITHUB_OUTPUT
succes=$(echo $job_output | jq -r '.success')
failed_jobs=$(echo $job_output | jq -r '.failed_steps')
echo -e "succes=${success}\nfailed_jobs=${failed_jobs}" >> $GITHUB_OUTPUT
echo "## Job Output for ${{matrix.connector}}" >> $GITHUB_STEP_SUMMARY
echo "- Success: ${success}" >> $GITHUB_STEP_SUMMARY
echo "- Failed Jobs: ${failed_jobs}" >> $GITHUB_STEP_SUMMARY
if [ "$succes" != "true" ]; then
echo "status=failed" >> $GITHUB_OUTPUT
exit 1
fi
aaronsteers marked this conversation as resolved.
Show resolved Hide resolved
echo -e "\n[Download Job Output](${{steps.upload_job_output.outputs.artifact-url}})" >> $GITHUB_STEP_SUMMARY
16 changes: 11 additions & 5 deletions airbyte_cdk/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
# Imports should also be placed in `if TYPE_CHECKING` blocks if they are only used as type
# hints - again, to avoid circular dependencies.
# Once those issues are resolved, the below can be sorted with isort.
import dunamai as _dunamai
# import dunamai as _dunamai

from .destinations import Destination
from .models import AirbyteConnectionStatus, AirbyteMessage, ConfiguredAirbyteCatalog, Status, Type, FailureType, AirbyteStream, AdvancedAuth, DestinationSyncMode, ConnectorSpecification, OAuthConfigSpecification, OrchestratorType, ConfiguredAirbyteStream, SyncMode, AirbyteLogMessage, Level, AirbyteRecordMessage
Expand Down Expand Up @@ -282,12 +282,18 @@
"StreamSlice",
]

__version__ = _dunamai.get_version(
"airbyte-cdk",
third_choice=_dunamai.Version.from_any_vcs,
).serialize()
__version__: str
"""Version generated by poetry dynamic versioning during publish.

When running in development, dunamai will calculate a new prerelease version
from existing git release tag info.
"""

# try:
# __version__ = _dunamai.get_version(
# "airbyte-cdk",
# third_choice=_dunamai.Version.from_any_vcs,
# fallback=_dunamai.Version("0.0.0+dev"),
# ).serialize()
# except:
__version__ = "0.0.0+dev"
Loading
Loading