Computing considerations for large data set

[victoriaserelli]

Hi Keegan,
Hoping you can help shed some insight on my issue. I have 94 samples (emseq) in my dataset, and I am trying to run dmrseq with the test covariate being a continuous variable. I always get stuck for a very long time at the first step: "Detecting candidate regions with coefficient larger than 0.1 in magnitude. Chromosome chr1" .. and does not progress any further.

Also experience the same scenario when i try to do multiple (more than 2) group comparisons (categorical test covariate with 3 or 4 levels).
On the same machine, same data set, however I am able to run 2-group comparisons successfully.

Any thoughts on why this may be so? is this a matter of compute power?

Thanks in advance,
vic

[kdkorthauer]

Hi Vic,

Thanks for reaching out. I'm not able to think of a reason that points to the nature of the type of covariate used causing very different run times. A few questions:

• Can you give me some intuition about how comparable the 2-group covariate you've compared against is to the continuous in terms of expected differential methylation signal? In other words, is by chance the continuous covariate expected to be associated with differential methylation whereas the 2-group covariate is not (or less so)? The reason I ask is because the computation time increases for larger candidate regions.
• When you say it gets stuck for a very long time - how long did you let it run? You don't see progression to chr 2, etc?

Thanks

[victoriaserelli]

Hi Keegan Thanks for your prompt response. My continuous variable is correlated with my categoricals – i.e., the categories were created based on the continuous. I’ve tried twice letting it run overnight (~10-12 hours), and yes, it does not progress to chr2. Same scenario for multiple group comparisons, so I had to subset my bsseq object to 2 groups and do pairwise comparisons for 2 groups at a time (I have 4 categories). Not sure how this approach will affect the interpretation of the statistics.. Best, vic From: Keegan Korthauer ***@***.***> Date: Saturday, March 1, 2025 at 8:00 To: kdkorthauer/dmrseq ***@***.***> Cc: Victoria Serelli Lee ***@***.***>, Author ***@***.***> Subject: [EXTERNAL] Re: [kdkorthauer/dmrseq] Computing considerations for large data set (Issue #70) Hi Vic, Thanks for reaching out. I'm not able to think of a reason that points to the nature of the type of covariate used causing very different run times. A few questions: Can you give me some intuition about how comparable the categorical Hi Vic, Thanks for reaching out. I'm not able to think of a reason that points to the nature of the type of covariate used causing very different run times. A few questions: * Can you give me some intuition about how comparable the categorical covariates you've compared against are to the continuous in terms of expected differential methylation signal? In other words, is by chance the continuous covariate expected to be associated with differential methylation whereas the categorical covariates are not (or less so)? The reason I ask is because the computation time increases for larger candidate regions. * When you say it gets stuck for a very long time - how long did you let it run? You don't see progression to chr 2, etc? Thanks — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/kdkorthauer/dmrseq/issues/70*issuecomment-2691686468__;Iw!!A_kSX19cfVk!AuEy1abZnSY0Tg8lysDvVnnDxv_2It99KH8A01K7vhVQUR7qxwfoVSkS-jEiVndVKecl1_9luGBu1UkpqYzoSDKLT_c$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVU2S3FTCQX2HBEIFKYZXND2SDS7VAVCNFSM6AAAAABYBREJDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJRGY4DMNBWHA__;!!A_kSX19cfVk!AuEy1abZnSY0Tg8lysDvVnnDxv_2It99KH8A01K7vhVQUR7qxwfoVSkS-jEiVndVKecl1_9luGBu1UkpqYzofHCQYns$>. You are receiving this because you authored the thread.Message ID: ***@***.***> [kdkorthauer]kdkorthauer left a comment (kdkorthauer/dmrseq#70)<https://urldefense.com/v3/__https:/github.com/kdkorthauer/dmrseq/issues/70*issuecomment-2691686468__;Iw!!A_kSX19cfVk!AuEy1abZnSY0Tg8lysDvVnnDxv_2It99KH8A01K7vhVQUR7qxwfoVSkS-jEiVndVKecl1_9luGBu1UkpqYzoSDKLT_c$> Hi Vic, Thanks for reaching out. I'm not able to think of a reason that points to the nature of the type of covariate used causing very different run times. A few questions: * Can you give me some intuition about how comparable the categorical covariates you've compared against are to the continuous in terms of expected differential methylation signal? In other words, is by chance the continuous covariate expected to be associated with differential methylation whereas the categorical covariates are not (or less so)? The reason I ask is because the computation time increases for larger candidate regions. * When you say it gets stuck for a very long time - how long did you let it run? You don't see progression to chr 2, etc? Thanks — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/kdkorthauer/dmrseq/issues/70*issuecomment-2691686468__;Iw!!A_kSX19cfVk!AuEy1abZnSY0Tg8lysDvVnnDxv_2It99KH8A01K7vhVQUR7qxwfoVSkS-jEiVndVKecl1_9luGBu1UkpqYzoSDKLT_c$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVU2S3FTCQX2HBEIFKYZXND2SDS7VAVCNFSM6AAAAABYBREJDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJRGY4DMNBWHA__;!!A_kSX19cfVk!AuEy1abZnSY0Tg8lysDvVnnDxv_2It99KH8A01K7vhVQUR7qxwfoVSkS-jEiVndVKecl1_9luGBu1UkpqYzofHCQYns$>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[kdkorthauer]

Thanks for the additional information. So the two group comparison includes a smaller subset of the samples - this could be a clue. But I still wouldn't expect such a drastic difference in behaviour. To troubleshoot this, I would try the comparison of the two group and the four group analyses on just a small subset of chromosome 1 (maybe start with 10% of it) and examine the difference in runtime. If you are willing and able to share a deidentified subset of your object that reproduces the behaviour (either chr 1 or a portion thereof), I'd be happy to dig into this and try to find what is the cause - here is a file upload link.

Best,
Keegan

[victoriaserelli]

Hi Keegan, Thank you for the insights! I will heed your advice and try my runs on a smaller subset of the data. Unfortunately I am not able to share the data for troubleshooting, but thank you for offering! Best, vic From: Keegan Korthauer ***@***.***> Date: Tuesday, March 4, 2025 at 2:00 To: kdkorthauer/dmrseq ***@***.***> Cc: Victoria Serelli Lee ***@***.***>, Author ***@***.***> Subject: [EXTERNAL] Re: [kdkorthauer/dmrseq] Computing considerations for large data set (Issue #70) Thanks for the additional information. So the two group comparison includes a smaller subset of the samples - this could be a clue. But I still wouldn't expect such a drastic difference in behaviour. To troubleshoot this, I would try the comparison Thanks for the additional information. So the two group comparison includes a smaller subset of the samples - this could be a clue. But I still wouldn't expect such a drastic difference in behaviour. To troubleshoot this, I would try the comparison of the two group and the four group analyses on just a small subset of chromosome 1 (maybe start with 10% of it) and examine the difference in runtime. If you are willing and able to share a deidentified subset of your object that reproduces the behaviour (either chr 1 or a portion thereof), I'd be happy to dig into this and try to find what is the cause - here is a file upload link<https://urldefense.com/v3/__https:/www.dropbox.com/request/cGKxeJmOvlR943LGXDTJ__;!!A_kSX19cfVk!BXn3uifbOkCOB0M5CKqgXRha-F1Hv6KEjqNd2QwOUD6JLbCGX_1C3dhHaAsI6kK88nQBqh00lR6vJmvc9iLmh4T93kw$>. Best, Keegan — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/kdkorthauer/dmrseq/issues/70*issuecomment-2695018164__;Iw!!A_kSX19cfVk!BXn3uifbOkCOB0M5CKqgXRha-F1Hv6KEjqNd2QwOUD6JLbCGX_1C3dhHaAsI6kK88nQBqh00lR6vJmvc9iLmy5Nh8rg$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVU2S3G62LIV4P3PSZ7JWMT2SSDA5AVCNFSM6AAAAABYBREJDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJVGAYTQMJWGQ__;!!A_kSX19cfVk!BXn3uifbOkCOB0M5CKqgXRha-F1Hv6KEjqNd2QwOUD6JLbCGX_1C3dhHaAsI6kK88nQBqh00lR6vJmvc9iLmb-FECXQ$>. You are receiving this because you authored the thread.Message ID: ***@***.***> [kdkorthauer]kdkorthauer left a comment (kdkorthauer/dmrseq#70)<https://urldefense.com/v3/__https:/github.com/kdkorthauer/dmrseq/issues/70*issuecomment-2695018164__;Iw!!A_kSX19cfVk!BXn3uifbOkCOB0M5CKqgXRha-F1Hv6KEjqNd2QwOUD6JLbCGX_1C3dhHaAsI6kK88nQBqh00lR6vJmvc9iLmy5Nh8rg$> Thanks for the additional information. So the two group comparison includes a smaller subset of the samples - this could be a clue. But I still wouldn't expect such a drastic difference in behaviour. To troubleshoot this, I would try the comparison of the two group and the four group analyses on just a small subset of chromosome 1 (maybe start with 10% of it) and examine the difference in runtime. If you are willing and able to share a deidentified subset of your object that reproduces the behaviour (either chr 1 or a portion thereof), I'd be happy to dig into this and try to find what is the cause - here is a file upload link<https://urldefense.com/v3/__https:/www.dropbox.com/request/cGKxeJmOvlR943LGXDTJ__;!!A_kSX19cfVk!BXn3uifbOkCOB0M5CKqgXRha-F1Hv6KEjqNd2QwOUD6JLbCGX_1C3dhHaAsI6kK88nQBqh00lR6vJmvc9iLmh4T93kw$>. Best, Keegan — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/kdkorthauer/dmrseq/issues/70*issuecomment-2695018164__;Iw!!A_kSX19cfVk!BXn3uifbOkCOB0M5CKqgXRha-F1Hv6KEjqNd2QwOUD6JLbCGX_1C3dhHaAsI6kK88nQBqh00lR6vJmvc9iLmy5Nh8rg$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVU2S3G62LIV4P3PSZ7JWMT2SSDA5AVCNFSM6AAAAABYBREJDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJVGAYTQMJWGQ__;!!A_kSX19cfVk!BXn3uifbOkCOB0M5CKqgXRha-F1Hv6KEjqNd2QwOUD6JLbCGX_1C3dhHaAsI6kK88nQBqh00lR6vJmvc9iLmb-FECXQ$>. You are receiving this because you authored the thread.Message ID: ***@***.***>