Airbyte - Error: Response too large to return while syncing large datasets

[jenisha0512]

Description:
I am facing an issue while trying to sync a large dataset of approximately 50GB using Airbyte. The sync process is failing, and I receive the following error message: Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/troubleshooting-errors.

Additionally, I observed that when I try to sync only 10 or fewer columns from the dataset, the sync runs smoothly without any errors. However, when attempting to sync more than 10 columns, the same error message is encountered.

Additional Information:
Source: Bigquery:0.3.0
Destination: Postgresql:0.4.0

Steps to Reproduce:

1. Configure Airbyte to sync a dataset of around 50GB in size.
2. Initiate the synchronization process.
3. Observe the error message: Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/troubleshooting-errors.
4. Retry the sync with a reduced number of columns (10 or fewer).
5. Verify if the sync completes successfully without any errors.

Expected Behavior:
I expected the Airbyte sync process to handle large datasets seamlessly, irrespective of the number of columns being synchronized.

Actual Behavior:
Airbyte throws the error Response too large to return when syncing a dataset of 50GB, especially when the number of columns exceeds 10.

Is anyone else experiencing a similar issue or has found a solution? Please share your insights and suggestions. Any help would be appreciated.