AFHMM and AFHMM SAC: limit number of processes

[klemenjak]

Hi there,

this is not an issue but rather something I was wondering about.

While running AFHMM and AFHMM + SAC in my experiments, I recognized that those approaches really occupy a lot of CPU power. Even our simulation servers struggle with those algorithms sometimes.

Have you thought about providing a single-thread implementation of disaggregate_chunk ? Is it possible to limit the number of created threads in an easy manner?

Feel free to ignore this :)

Best,
C

[Rithwikksvr]

Hey @klemenjak

If you increase the 'time_period' parameter value of the AFHMM or AFHMM-SAC, you can decrease the number of threads.

First, we break the input sequence into multiple chunks of length 'time_period'. Each of those chunks is given a thread.

So, by increasing the value of 'time_period', you can decrease the threads.

It takes more time to solve a thread if you increase the 'time_period' value.

For our buildsys-experiments, we used a powerful CPU system with many threads. We were using 'time_period':720, and even then it took a lot of time.

Hope this was helpful!

[klemenjak]

Hi, thank you so much! I will give it a shot! Cheers, C

…

Am 05.05.2020 um 08:27 schrieb Rithwik Kukunuri ***@***.***>: Hey @klemenjak <https://github.com/klemenjak> If you increase the 'time_period' parameter value of the AFHMM or AFHMM-SAC, you can decrease the number of threads. First, we break the input sequence into multiple chunks of length 'time_period'. Each of those chunks is given a thread. So, by increase the value of 'time_period', you can decrease the threads. It takes more time if you increase the 'time_period' value. For our buildsys-experiments, we used a powerful CPU system with many threads. We were using 'time_period':720, and even then it took a lot of time. Hope this was helpful! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AINC6E42666RPY5MXKPZB4LRP6WWNANCNFSM4MYTHEFA>.

[Wusir2018]

Hi @klemenjak
Could you give me a example with AFHMM and AFHMM_SAC? I can't run them correctly.
Best,
Wu

[klemenjak]

Hi @Wusir2018,

I haven't tested that so far. Sorry.

[Wusir2018]

Hey @klemenjak
It doesn't matter. Thank you!

[Rithwikksvr]

HI @Wusir2018

What exactly you aren't able to run?

[Wusir2018]

Hey @Rithwikksvr ,
Thank you for your reply !
I can run mean, but I can't run AFHMM and AFHMM SAC. This is my code:

But it stopped here:

[Rithwikksvr]

Hey @Wusir2018

The AFHMM and AFHMM-SAC take significantly longer times for generating the testing predictions.

It takes a couple of hours to compute predictions for a single day

Also, us use a system with more CPUs

[Wusir2018]

Hey @Rithwikksvr ,
Thank you for your reply. It helps me a lot. It's my fault and my computer is poor.
I have a problem about RNN,WindowGRU with AMPDs data.
I can run REDD data with RNN,WindowGRU.
But I can't run AMPDs data, I think the sampling frequency of AMPDs data causes this problem. How can I solve this problem?

This is my code:

This is my error

[Rithwikksvr]

I'll run it once and get back to you by tomorrow

[Wusir2018]

Hi @Rithwikksvr ,
Thank you very much !
Best wishes !

[chaosimple]

Hi @Rithwikksvr,

I am new to this area, so I don't quite clear about what exactly does the AFHMM algorithm refer to?
Is it refer to the "Additive FHMM" or the "Approximate FHMM"?

[MingjunZhong]

@chaosimple I think it was referring to the additive FHMM model and the method solving it was an approximate method employing Second-Order Cone Programming (SOCP).

[PMeira]

@chaosimple Check the README on the front page of the repository. The paper linked there should list the references: https://github.com/nilmtk/nilmtk-contrib/blob/master/README.md

(right now, the code is really missing some basic info as docstrings or comments)

[levaphenyl]

Note: processes of AFHMM and AFHMM+SAC should be daemonic. In case the OS kills the main process, e.g. because of memory pressure, the current implementation leaves tons of wild orphaned processes behind.

[Rithwikksvr]

But @levaphenyl , do you know any fix we can use for this?

[levaphenyl]

But @levaphenyl , do you know any fix we can use for this?

Yes, @Rithwikksvr. I use multiprocessing.Pool in my PR.

@klemenjak, I refactored AFHMM+SAC to use all CPUs - 1. Is it compatible with your original intent?