Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not use infinite slow write timers #1670

Merged
merged 1 commit into from
Jul 8, 2023
Merged

Do not use infinite slow write timers #1670

merged 1 commit into from
Jul 8, 2023

Conversation

adriansmares
Copy link
Contributor

@adriansmares adriansmares commented Jul 4, 2023

Summary

References #1629

This PR removes the use of 'infinite' slow write timers (i.e. timers which expire at the end of time). Previously these slow write timers would be leaked and slowly buildup over time as connections are recycled.

Background

This 'bug' is a bit of a weird feature - the perpetual reference to the net.Conn ensures that the connection is not closed in case the library users don't correctly close the connection, and in turn this is visible when the connection limit is hit.

We discovered this while debugging why some of our tests were failing after #1629. The commit that triggered our issues was cd2986e, so we started to look into what could cause this issue in these changes.

The slowWriteTimer keeps a perpetual reference to the net.Conn (via the background reader). This in turn basically ensures that the net.Conn is never garbage collected, which in turn means that the finalizer is never run. The finalizer for a net.Conn under Unix closes the file descriptor.

What we were doing wrong in our code was that we did not close some rows returned by QueryContext, but we did stop referencing every related to the *sql.DB (after calling Close on it). However *sql.DB works with a reference counting mechanism, and because we did not close the sql.Rows, the underlying connection was never properly closed.

Our application code works correctly both with and without the fix proposed in this PR, but I would still suggest not using these timers, as they are inherently leaky.

Changes

  • Instead of using timers that expire at the end of time, simply Stop the timers when they should not fire, and Reset otherwise.

Notes for reviewers

I use gofumpt which is a bit more exigent than gofmt and it picked up some whitespace changes. Let me know if I should remove these.

@adriansmares adriansmares changed the title Do not use infinite timers Do not use infinite slow write timers Jul 4, 2023
@adriansmares adriansmares marked this pull request as ready for review July 4, 2023 14:34
@drakkan
Copy link
Contributor

drakkan commented Jul 4, 2023

Hello,

I'm investigating a similar issue. Unfortunately I cannot reproduce it myself, it was reported by a SFTPGo user. I'll share profiling results later.

Maybe is enough to add

pgConn.bgReader.Stop()
pgConn.slowWriteTimer.Stop()

before return pgConn.conn.Close() here.

The memory leak seems related to the update to pgx 5.4.x. I was using v5.3.2-0.20230520135323-70a200cff4d4 with no issues before.

In my case I always call rows.Close()

@ujihisa
Copy link

ujihisa commented Jul 5, 2023

I also observed memory leak in our work app, and I did a quick git bisect against pgx and found that 26c79eb introduced the memory leak. Not sure if it's the same or not, but I share here just FYI.

@drakkan drakkan mentioned this pull request Jul 6, 2023
@jackc jackc merged commit 2bf5a61 into jackc:master Jul 8, 2023
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants