Problems during model training.

[amingc3]

I have trained the new two genomes, but during the training Pearsonr has been NAN and r2 is inf. and the model automatically stops training at the 12th time. Can you please help me to solve this problem?

[davek44]

Hi, this isn’t enough information to help you debug your training run. Usually this indicates that the training data itself is numerically unstable. I’d convert the tfrecords back to numpy arrays and verify their integrity and the appropriateness of your loss function.

…

On Fri, Apr 5, 2024 at 11:42 PM amingc3 ***@***.***> wrote: I have trained the new two genomes, but during the training Pearsonr has been NAN and r2 is inf. and the model automatically stops training at the 12th time. Can you please help me to solve this problem? image.png (view on web) <https://github.com/calico/basenji/assets/165865744/843ac042-4d5e-4bcd-8f85-da986580d74e> — Reply to this email directly, view it on GitHub <#192>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABKFEG7QKOTHW77HEYXCL3Y36KNFAVCNFSM6AAAAABF2GSJ3CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZDSMJQHAYTONQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[amingc3]

嗨，这还不足以帮助您调试训练运行。 通常，这表明训练数据本身是数值上的 稳定。我会将 tfrecords 转换回 numpy 数组并验证它们的 损失函数的完整性和适当性。
...
2024 年 4 月 5 日星期五晚上 11：42 amingc3 @.>写道： 我已经训练了新的两个基因组，但在训练期间，Pearsonr 已经 been NAN 和 r2 是 inf.，模型自动停止训练 第12次。你能帮我解决这个问题吗？ image.png （在网络上查看） [https://github.com/calico/basenji/assets/165865744/843ac042-4d5e-4bcd-8f85-da986580d74e](https://github.com/calico/basenji/assets/165865744/843ac042-4d5e-4bcd-8f85-da986580d74e) — 直接回复此邮件，在 GitHub 上查看 <#192>，或取消订阅 https://github.com/notifications/unsubscribe-auth/AABKFEG7QKOTHW77HEYXCL3Y36KNFAVCNFSM6AAAAABF2GSJ3CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZDSMJQHAYTONQ . 您收到此消息是因为您订阅了此线程。消息 编号： @.>

Thank you for your reply, I have looked at the existing tfr file and there are a lot of warnings during the generation of the tfr file that the corresponding fragment is not found in the cool file, I don't understand what the problem is, I hope you can help me to see if it's a parameter setting error!

[davek44]

I think you're using Akita for Hi-C prediction. Is that right? Your sequence length 6,291,456 is much larger than anything we've tried before. I recommend starting with something like 1,048,576. If you share the dataset construction warnings, we might be able to help.

[amingc3]

I think you're using Akita for Hi-C prediction. Is that right? Your sequence length 6,291,456 is much larger than anything we've tried before. I recommend starting with something like 1,048,576. If you share the dataset construction warnings, we might be able to help.

I changed the input sequence to 1
mb, but still get a warning like the one in the picture below.

It then tells us that the model automatically sets these fragments to 0.

I checked by putting the hic file in the juice box and the segments are there and not 0 either.

[amingc3]

I'm finding problems with the split tfr file, there are problems with the test_inputs generated, here's the test_inputs generated by the example tfr,

here's the ones I've generated,

which are obviously very different.

[davek44]

The last portion seems to encounter a switch we made a few years ago to specify the one hot encoded nucleotides with a single integer, rather than a length 4 array of actual one hot encodings. You might follow the code around that to see if you can identify the problem.

But beyond that, you're just not sharing a clear enough description of the steps you're taking for me to help.

[amingc3]

The model I trained predicts results in a very strange form. And the trained model pearsonr ultimately stopped at around 0.06, I don't understand where the problem lies. Could it be a problem with my data processing? I am using the cool file you provided.