You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to read a table with a where statement. In my case the result is about 144MB:
r$> object.size(a)/1024/1024144.7bytes
If I run the following command:
a=DBI::dbGetQuery(con, "select * from schema.table where column = 'value'")
a few times, I see memory usage jump sharply on the first connection (around 1.5GB), then it keeps increasing moderately as I keep rerunning that statement, until stabilizing at ~2GB total RAM.
However, if I wrap the call in a function:
test_db_con=function(con){
ret=dbGetQuery(con, "select * from schema.table where column = 'value'")
return(ret)
}
has a different behavior. No matter how many times I run the last line, memory usage never peaks. Instead it keeps increasing, by increments larger than the size of the data (around 500MB), eventually leading to OOM. To me, this hints at a memory leak problem, but I don't know why this would happen only in a package. FWIW, I am running this on an AWS EC2 instance, and the issue also happens when using PostgreSQL (although the memory explosion there seems much more pronounced). Any help appreciated !
Thanks. Minor nitpick: RPostgres::Redshift() is preferred over RPostgres::Postgres() to connect to the database, but I don't think this will change the outcome here.
Do you have a way of reproducing this on a toy RedShift instance where you could share credentials?
I am connecting to a Redshift Serverless deployment using RPostgres. Let's call that connection
con
:I am trying to read a table with a where statement. In my case the result is about 144MB:
If I run the following command:
a few times, I see memory usage jump sharply on the first connection (around 1.5GB), then it keeps increasing moderately as I keep rerunning that statement, until stabilizing at ~2GB total RAM.
However, if I wrap the call in a function:
and put it in a package, the following:
has a different behavior. No matter how many times I run the last line, memory usage never peaks. Instead it keeps increasing, by increments larger than the size of the data (around 500MB), eventually leading to OOM. To me, this hints at a memory leak problem, but I don't know why this would happen only in a package. FWIW, I am running this on an AWS EC2 instance, and the issue also happens when using
PostgreSQL
(although the memory explosion there seems much more pronounced). Any help appreciated !Version info:
The text was updated successfully, but these errors were encountered: