Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop #3

Merged
merged 16 commits into from
Jul 29, 2024
Merged

Develop #3

merged 16 commits into from
Jul 29, 2024

Conversation

davidmscholz
Copy link
Contributor

No description provided.

Copy link
Member

@Threated Threated left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good mostly some style comments.
One general thing you can import anyhow::Result which is a type alias for Result<T, anyhow::Error> which makes the return types look a bit nicer.

Comment on lines +6 to +59
pub async fn get_pg_connection_pool(pg_url: &str, num_attempts: u32) -> Result<PgPool, anyhow::Error> {
info!("Trying to establish a PostgreSQL connection pool");

let mut attempts = 0;
let mut err: Option<anyhow::Error> = None;

while attempts < num_attempts {
info!("Attempt to connect to PostgreSQL {} of {}", attempts + 1, num_attempts);
match PgPoolOptions::new()
.max_connections(10)
.connect(&pg_url)
.await
{
Ok(pg_con_pool) => {
info!("PostgreSQL connection successfull");
return Ok(pg_con_pool)
},
Err(e) => {
error!("Failed to connect to PostgreSQL. Attempt {} of {}: {}", attempts + 1, num_attempts, e);
err = Some(anyhow!(e));
}
}
attempts += 1;
tokio::time::sleep(std::time::Duration::from_secs(5)).await; //@todo: move param somewhere else?
}
Err(err.unwrap_or_else(|| anyhow!("Failed to connect to PostgreSQL")))
}


pub async fn check_blaze_connection(blaze_base_url: &str, num_attempts: u32) -> Result<bool, anyhow::Error> {
info!("Attempting to connect to Blaze");

let mut attempts = 0;
let mut err: Option<anyhow::Error> = None;
let client = Client::new();

while attempts < num_attempts {
info!("Attempt to connect to Blaze {} of {}", attempts + 1, num_attempts);
match client.get(format!("{}/health", blaze_base_url)).send().await {
Ok(_) => {
info!("Blaze connection successfull");
return Ok(true)
},
Err(e) => {
error!("Failed to connect to Blaze. Attempt {} of {}: {}", attempts + 1, num_attempts, e);
err = Some(anyhow!(e));
}
}
attempts += 1;
tokio::time::sleep(std::time::Duration::from_secs(5)).await; //@todo: move param somewhere else?
}
Err(err.unwrap_or_else(|| anyhow!("Failed to connect to Blaze")))

}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to consider using tryhard as we ended up doing in focus as well for the retries.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For focus the goal was to implement exponential backoff, right. That makes sense to me. In this case here a scheduled job is supposed to run that may take up considerable resources from the Bridgehead servers. For the time being I'd prefer to have a very easy to understand window of time when sync_blaze_2_pg is supposed to run.

use tracing::info;

pub async fn wait_for_signal() {
use tokio::signal::unix::{signal,SignalKind};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will only compile on unix now because of the import. I usually just use tokio::signal::ctrl_c which sadly does not work in docker for some reason so I set STOPSIGNAL sigint in the Dockerfile.

src/models.rs Outdated Show resolved Hide resolved
src/main.rs Show resolved Hide resolved
src/main.rs Outdated Show resolved Hide resolved
Comment on lines +301 to 305
if let Some(next_url) = next_url {
url = next_url;
} else {
break;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of a loop with this I would use a while let loop. Although you might need to fight a bit with the compiler maybe its not so simple.

src/main.rs Show resolved Hide resolved
src/main.rs Outdated Show resolved Hide resolved
src/main.rs Outdated Show resolved Hide resolved
src/main.rs Show resolved Hide resolved
Copy link
Member

@enola-dkfz enola-dkfz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to admit I still don't understand why you use % operator here but I trust you that it works
if update_counter > 0 && update_counter % batch_size == 0 {
update_helper(pg_con_pool, &update_batch, &table_name).await?;
update_batch.clear();

        } else if insert_counter > 0 && insert_counter % batch_size == 0 {
            insert_helper(pg_con_pool, &insert_batch, &table_name).await?;
            insert_batch.clear();
        }

src/db_utils.rs Outdated
info!("Creating PostgreSQL tables");

let create_tables_queries = vec![
"CREATE TABLE IF NOT EXISTS patients (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would extract the DDL into a separate file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. These queries should be rewritten at some point.

@davidmscholz davidmscholz merged commit f687f27 into main Jul 29, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants