Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make extract_top_level_blocks() faster #189

Closed
wants to merge 3 commits into from
Closed

Make extract_top_level_blocks() faster #189

wants to merge 3 commits into from

Conversation

fredriv
Copy link

@fredriv fredriv commented Sep 10, 2024 •
edited
Loading

Resolves dbt-labs/dbt-core#9037

Description

This PR improves the performance of parsing docs files.
Duplicates the change from dbt-labs/dbt-core#9045 into dbt-common

Have verified the performance improvement in our local dbt project (~2300 models). Full parse time on my M1 Mac went from 2m20s to 41s.

Checklist

@fredriv fredriv requested a review from a team as a code owner September 10, 2024 13:36
@cla-bot cla-bot bot added the cla:yes label Sep 10, 2024
This was referenced Sep 10, 2024
Closed
Closed
Closed
@fredriv
Copy link
Author

fredriv commented Sep 10, 2024

Credit should really go to @peterallenwebb BTW for the original PR in dbt-core (dbt-labs/dbt-core#9045) which I've just re-created here 🙂

@fredriv fredriv force-pushed the fredriv/CT-3361-improve-docs-parsing-performance branch from 7b6978b to 14cef17 Compare September 24, 2024 07:24
@fredriv fredriv force-pushed the fredriv/CT-3361-improve-docs-parsing-performance branch from 14cef17 to 21a1c31 Compare October 7, 2024 08:33
@fredriv
Copy link
Author

fredriv commented Oct 7, 2024

Hi @dbeatty10! Do you know if there is a chance to get this merged into the next release of dbt-common? We're eager to get this fix into our systems to improve dbt performance 🙂

I just rebased my branch on the latest main branch so should be ready to merge I think. Let me know if there is anything else I need to do!

@codecov-commenter
Copy link

codecov-commenter commented Oct 7, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.48%. Comparing base (ed11c6c) to head (a45798c).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #189      +/-   ##
==========================================
+ Coverage   68.33%   68.48%   +0.14%     
==========================================
  Files          52       52              
  Lines        3373     3389      +16     
==========================================
+ Hits         2305     2321      +16     
  Misses       1068     1068
Flag Coverage Δ
unit 68.48% <100.00%> (+0.14%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@fredriv fredriv force-pushed the fredriv/CT-3361-improve-docs-parsing-performance branch from a70f349 to a45798c Compare October 15, 2024 18:32
@fredriv
Copy link
Author

fredriv commented Oct 15, 2024

Updated my branch with latest changes in main now. Any chance of merging this PR @gshank @dbeatty10 @peterallenwebb? 🙏 Please let me know if there is anything more I should do 🙂

@gshank
Copy link
Contributor

gshank commented Oct 15, 2024

Closed in favor of #205

@gshank gshank closed this Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gshank gshank gshank approved these changes

Assignees
No one assigned
Labels
cla:yes performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CT-3361] Improve Docs Parsing Performance
4 participants
@fredriv @codecov-commenter @gshank @dbeatty10