Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid loading everyting into a DataTable #21

Closed
yorek opened this issue Sep 22, 2020 · 2 comments · Fixed by #26
Closed

Avoid loading everyting into a DataTable #21

yorek opened this issue Sep 22, 2020 · 2 comments · Fixed by #26

Comments

@yorek
Copy link
Contributor

yorek commented Sep 22, 2020 •
edited
Loading

At line 105 of SqlAsyncCollector.cs, everything is loaded in memory to allow DataTable to figure out if a row should be INSERTed or UPDATEd. This is ok for a very small table, but with bigger table it will probably crash the runtime as the entire target SQL table will be loaded in memory.

Usage of Table-Values Parameters + MERGE statment is recommended to push data to the Azure SQL database and avoiding to load all data in memory
@yorek
Copy link
Contributor Author

yorek commented Sep 22, 2020

Issue is somehow related to #4

@yorek
Copy link
Contributor Author

yorek commented Sep 22, 2020

using TVP will help for #6

@glennamanns glennamanns linked a pull request Oct 7, 2020 that will close this issue
Output Binding: Switch to MERGE + JSON query model #26
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Output Binding: Switch to MERGE + JSON query model Azure/azure-functions-sql-extension
1 participant
@yorek