CheapPower module #22529
CheapPower module #22529
Conversation
|
Thanks. Is it valid only for Finland? If so, I believe it should show in the name of the file like "cheap_power_Finland" |
The current version is useful only for Finland, but it could be fairly easily extended to cover at least the entire Nord pool area. The data is produced by ENTSO-E, which based on https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show seems to cover most of Europe, including Türkiye and Ukraine. Some of these countries might not implement dynamic pricing for end consumers yet, but I believe that it will come. Ideally, someone would run a public service that is based on some code like https://github.com/oysteinjakobsen/fetch-day-ahead-price or https://github.com/JaccoR/hass-entso-e so that end users can save themselves the trouble of registering and configuring an API key with ENTSO-E. Besides, this API could be too resource intensive to implement in Tasmota. The hypothetical service would deliver the known prices starting from the currently active slot in a uniform format. The command would take a parameter to the command to specify the price area (such as an area of Norway): This could be translated into a simple URL like The second parameter could also be a full URL, for example pointing to server in the LAN, which could run an ENTSO-E interface and cache to serve multiple devices. This would in no means be limited to ENTSO-E or Europe. Another thinkable enhancement would be a third parameter to specify the desired number of slots to choose per day, like this: Some heating could need to run for multiple hours per day. There are some plans to narrow the price slots from 60 to 15 minutes in the future. In that case, even my deployment would require 2 to 4 such slots per day. A quick search turned up some further open JSON data sources:
These could be implemented fairly easily, after asking the operators if this kind of automated access is okay with them. The format of the URL and the data would likely vary between any area-specific JSON data sources. I realize that to reduce the memory footprint, it could make sense to split this interface into separate modules that would be loaded on demand. For Estonia, I only found https://elektrihind.ee/borsihind/ which does not seem to include any public JSON based interface. The German Fraunhofer-Institut für Solare Energiesysteme is running https://energy-charts.info/charts/price_spot_market/chart.htm with a nice country selection, but apparently without any raw data interface that is suitable for this kind of use. There is a CSV export function that seems to spit out data for the current week. I think that this needs to start somewhere. If it helps, I can make the country parameter mandatory in the first version, and reject anything else than |
|
I was in contact with the provider of the Swedish price data, and now there is a simpler URL https://mgrey.se/espot?format=json&domain=SE1&date=2024-12-13 that will return the price for a single zone for the given day. Unfortunately, when I tried accessing this HTTPS server in the Berry console of Tasmota 14.1.0, I got an error, I suppose due to some TLS or SSL incompatibility. It is not possible to enable plain HTTP support on this server. I could make the URL pattern configurable, so that this service could be reached via (say) |
|
Hmmm. I probed I will evaluate the impact of supporting |
|
Please try with the latest version which includes #22649 I have now enabled ECDSA: wc = webclient()
print(wc.begin('https://mgrey.se/espot?format=json&domain=SE1&date=2024-12-13'))
print(wc.GET())
print(wc.get_string())
print(wc.close())Output shows: |
|
Thank you a lot, @s-hadinger! I upgraded to a development snapshot (b3b9699 is 1 commit ahead of the merge 615c676 of #22649): I am glad to see that also sahkotin.fi now is accessible with https. My upcoming update will revise that URL as well. |
|
FTR, most example projects with Tasmota+Berry are published on Github repositories owned by the creator. I've collected a list of repositories including Berry code: |
|
That URL is timing out for me. Yes, I was uncertain if this would be the appropriate place. I was expecting to a link to a package directory at https://tasmota.github.io/docs/Tasmota-Application/. I spent quite a bit of time debugging today, trying to figure out what I am doing wrong when implementing support for a second data source. It turns out that the problem is directly caused by switching As soon as I change the code to use Unfortunately, there is no serial console connection to my only Tasmota equipped device, so I’m afraid I am unable to debug this deeper. When I was experimenting with the Berry console, it seemed that var data = json.load(wc.get_string())
wc.close()could run out of memory (end up with var data = wc.get_string()
wc.close()
data = json.load(data)might allow the Berry garbage collector to free some memory earlier. However, when I tried to revise the program like this, it would still not work (cause the device to be reset or to lose the WLAN connection). I would appreciate it if you could check if the anomaly is reproducible for you. |
|
With the URL above Once loaded the JSON takes 3.5KB which is not huge but still significant |
|
Well, of course using I had no trouble getting past the point of fetching the data, using: My test was using a recent build of tasmota 14.4.1.1 (a bit newer than yours), on an ESP32-S3 with PSRAM available. Switching to an ESP32-C3 with less RAM (and no PSRAM) made no difference, it still had no trouble https-fetching the data. While deferring the Of course, not knowing or replicating your test case, I can't be sure if my test got far enough for whatever issue you had. In general, I like to reduce scope of test cases to "zoom in". As long as I can prune to code to be shorter I know that errors still occurring has to be within what's left. |
|
I had tried to add several I will try to narrow down this "test case" for reproducing the anomaly. That will take a few days, though. I think I should use an |
|
Print goes nowhere |
|
Of course, If the failure is first time in a code path, you could also create a runtime error at a certain point in the code, if you get that, you know that it was safe that far. If the failure is not first time in the code path, it might be a memory leak, which you can check by following free memory. If it keeps depleting, something is rotten..... |
|
I think that this anomaly will occur on the first time after restart or reboot. Which Berry code would you recommend for injecting a runtime error? |
|
The Berry language does allow such: |
Sorry |
|
I debugged this a little. I removed the --- tasmota/berry/modules/cheap_power/cheap_power.be 2024-12-15 21:00:30.860176636 +0200
+++ tasmota/berry/modules/cheap_power/cp.be 2024-12-24 09:25:48.127720826 +0200
@@ -1,9 +1,6 @@
import webserver
import json
-var cheap_power = module("cheap_power")
-
-cheap_power.init = def (m)
class CheapPower
var prices # future prices for up to 48 hours
var times # start times of the prices
@@ -20,7 +17,7 @@
"<td style='width:25%'><button onclick='la(\"&op=2\");'>🔄</button></td>"
"<td style='width:25%'><button onclick='la(\"&op=3\");'>⏭</button></td>"
"</tr></table>"
- static var URL0 = 'http://sahkotin.fi/prices?start=', URL1 = '&end='
+ static var URL0 = 'https://sahkotin.fi/prices?start=', URL1 = '&end='
static var URLTIME = '%Y-%m-%dT%H:00:00.000Z'
def init()
@@ -29,15 +26,9 @@
end
def start(idx)
- if idx == nil || idx < 1 || idx > tasmota.global.devices_present
- tasmota.log(f"CheapPower{idx} is not a valid Power output")
- tasmota.resp_cmnd_failed()
- else
self.channel = idx - 1
tasmota.add_driver(self)
self.update()
- tasmota.resp_cmnd_done()
- end
end
def power(on) tasmota.set_power(self.channel, on) end
@@ -45,9 +36,8 @@
# fetch the prices for the next 24 to 48 hours
def update()
var wc = webclient()
- var rtc = tasmota.rtc()
- self.tz = rtc['timezone'] * 60
- var now = rtc['utc']
+ self.tz = 120 * 60
+ var now = 1734822000
var url = self.URL0 +
tasmota.strftime(self.URLTIME, now) + self.URL1 +
tasmota.strftime(self.URLTIME, now + 172800)
@@ -149,6 +139,6 @@
tasmota.web_send_decimal(status)
end
end
-return CheapPower()
-end
-return cheap_power
+var cheap_power = CheapPower()
+cheap_power.start(1)
+print(cheap_power.chosen)Right after rebooting the Shelly Pro 2 into Tasmota 14.4.0.1 (b3b9699-tasmota32), when I input the code with the above modification to the Berry Console, the To my surprise, the code would run just fine, even though I’m now using |
|
AFAICT, the "culprit" was the call hierarchy of
The thing is that issuing commands can be quite problematic when being invoked from an existing Tasmota core-related callback (especially a command callback), and In many cases, you can use a workaround of breaking the command hierarchy, using something like |
|
@sfromis Thank you for the advice; I will try that. I was thinking of For the record, f7fc732 is my current development, implementing an interface for Swedish prices. With Edit: Yes, it was this simple. Great! diff --git a/tasmota/berry/modules/cheap_power/cheap_power.be b/tasmota/berry/modules/cheap_power/cheap_power.be
index 238fa733a..42b4c531d 100644
--- a/tasmota/berry/modules/cheap_power/cheap_power.be
+++ b/tasmota/berry/modules/cheap_power/cheap_power.be
@@ -42,7 +42,7 @@ class CheapPower
if !payload
tasmota.log(f"CheapPower{idx}: a price zone name is expected")
elif payload == 'FI'
- self.p_url = 'http://sahkotin.fi/prices?start='
+ self.p_url = 'https://sahkotin.fi/prices?start='
self.p_kWh = '¢'
elif re.match('^SE[1-4]$', payload)
self.p_url = 'https://mgrey.se/espot?format=json&domain=' + payload +
@@ -58,7 +58,7 @@ class CheapPower
self.channel = idx - 1
self.p_zone = payload
tasmota.add_driver(self)
- self.update()
+ tasmota.set_timer(0, /->self.update())
tasmota.resp_cmnd_done()
end
|
|
It would be nice if this could be made to support Octopus Energy in the UK as well. Octopus's API returns larger text results, though - looks like about 15KB. |
|
@ryancdotorg Great idea. I hope that implementing support for more markets would make this pull request more ‘eligible’ to be merged to the Tasmota repository. I searched the web, came across https://gist.github.com/TerryE/55e413ce59b40a7233df9d76ad5821e6 and checked the output of https://api.octopus.energy/v1/products/. I successfully ran the following in the Tasmota Berry console of my Shelly Pro 2: import json
var wc=webclient()
wc.begin("https://api.octopus.energy/v1/products/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-B/standard-unit-rates/")
var rc=wc.GET()
var s=wc.get_string()
wc.close()
print(s)
s=json.load(s)
print(s)The URL in the above snippet seems to return the price information starting from the furthest available time in the future, in descending order of time, at 30-minute intervals. The interface could conveniently use a fixed URL and simply assume that it will return all data down to the current timestamp, and some time in the past (which our parser would ignore). The |
|
I'd say that support for multiple data sources would probably best be implemented by some sort of plug-in architecture, with the core functionality for controlling energy usage by optimum time would have a standardized interface to data source plugins. Maybe something like a near future list of time stamps (typically but not necessarily hourly) and price. Anyone could create their own data gathering function for their local market(s). Of course, mixing in PV and buffer batteries (like from an EV) can quickly be used to make the situation more complicated, if that's what you want.... Etc. |
|
I think that more complicated control solutions, such as those involving PV, battery banks or EV charging, are better implemented in a larger environment, such as Home Assistant. This control only makes sense for rather simple use cases. I’d like a plugin architecture where the Another thing that I need to figure out a simple enough GUI and logic for allowing multiple cheap slots per day to be used. Nord Pool is supposed to move from 60-minute to 15-minute slots in the near future. |
|
In Berry, when you pass objects (including lists), it is always by reference, with the callee getting full access. The caller should not depend on it being unmodified. A more generalized approach, not even depending on it being on the same device, could be to pass it as JSON. Yes, using |
|
Regarding Agile Octopus, yes, the tariff region would need to be manually selected. In theory, it's possible to determine based on lat/lon but that seems like too much magic to me.
There's another endpoint that can be used to discover the current correct tariff identifier, but it's a bit of a faff.
FWIW can see both sides of the argument as to whether this makes sense to have this built into Tasmota.
…On December 30, 2024 3:54:49 AM PST, "Marko Mäkelä" ***@***.***> wrote:
@ryancdotorg Great idea. I hope that implementing support for more markets would make this pull request more ‘eligible’ to be merged to the Tasmota repository.
I searched the web, came across https://gist.github.com/TerryE/55e413ce59b40a7233df9d76ad5821e6 and checked the output of https://api.octopus.energy/v1/products/. I successfully ran the following in the Tasmota Berry console of my Shelly Pro 2:
```berry
import json
var wc=webclient()
wc.begin("https://api.octopus.energy/v1/products/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-B/standard-unit-rates/")
var rc=wc.GET()
var s=wc.get_string()
wc.close()
print(s)
s=json.load(s)
print(s)
```
The URL in the above snippet seems to return the price information starting from the furthest available time in the future, in descending order of time, at 30-minute intervals. The interface could conveniently use a fixed URL and simply assume that it will return all data down to the current timestamp, and some time in the past (which our parser would ignore).
The `B` in the URL would be the tariff zone. That as well as the two occurrences of `AGILE-24-10-01` would have to be a parameter to the Tasmota command. Something like `CheapPower UK AGILE-24-10-01 B`? Is this what you had in mind? How often would the identifier such as `AGILE-24-10-01` have to be updated? Every time the electricity contract is renewed?
|
|
As I see what was suggested, it was adding Berry sample code for users to pick up, not having it "built into" Tasmota. And lots of users have published their Berry projects via Github, without them being submitted to the common Tasmota repository. |
|
Ah, thanks for clarifying. |
dr-m
force-pushed
the
development
branch
2 times, most recently
from
50861e5 to
13da61c
Compare
|
Thank you @sfromis, I will check your encapsulation example and see if that could lead to a better solution. I figured out how to make modules and class inheritance work. The base class is in a module on its own, and each subclass (specific to a data source) derives from the base class. I understand that this could lead to wasting memory for one singleton object of the base class; the derived class would use another singleton object. The singleton of the base class is only being used for To avoid code bloat on the end device, I ended up creating area specific I fixed the interface to the Swedish data source so that we can fetch the prices for the current day and the next day (in 2 requests). There was a bug in I tested this for all 3 data sources (Finland, Sweden, Octopus Energy in the UK) on my Shelly Pro 2. |
|
The example http://sfromis.strangled.net/tasmota/berry/examples/slider.be defines one function pointer, which is being assigned to an anonymous function that is defined when an object is instantiated. That is a good solution for overloading a single function, in scenarios where one would use a lambda expression in C++11 or later. For overloading multiple functions (in this case: It looks like I could rather easily include interfaces for Norway and Denmark, possibly using a unified parser, because the format only differs in the field name ( |
|
For the multi-function case, I'd actively want to not use a derived class, but just a class without any inheritance hierarchy, and then pass an instance of that class. Not much different from the case of a single function. This is the generally recommended principle of "composition over inheritance" now recognized as a cleaner solution in most cases. Easier to work with when no requirements for tight coupling. Trivial example to illustrate: |
|
Thank you. Currently, there is some tight coupling in the overloaded member function |
|
Sorry, I had introduced a bug when I refactored some code into |
|
According to Nordpool, the "single day-ahead coupling" of the EU electricity market will switch to 15-minute market time unit (MTU) on 2025-06-11. That is, the value of Another thing that may be relevant here is the billing based on peak power usage that is already used by some last-mile electricity distribution companies. It could become expensive to concentrate all dynamic consumption on the cheapest time slots if you have to pay transfer fees based on the maximum momentary power during the year. This could be implemented as an extra input, indicating that low-priority load needs to yield. For example, whenever the total momentary power reported by the HAN port of a smart meter exceeds a user-specified limit, immediately switch off the load and try to use the remaining next cheapest slots to maintain the temperature of the hot water tank. |
|
I found a nice CSV data source for Estonia, Finland, Latvia and Lihuania with timestamps in the POSIX for line: string.split(s,'\n')
for column:string.split(line,';')
print(column)
end
end |
dr-m
force-pushed
the
development
branch
2 times, most recently
from
eec6e53 to
61c4d31
Compare
|
I implemented a price graph, with the chosen time slot highlighted in red. The time unit is now determined automatically based on the difference of the first two timestamps in the downloaded data. For now, this remains limited to one time slot per day. I came up with a design: Add ➕ and ➖ buttons for changing the maximum number of slots, and let the ⏮ and ⏭ buttons move the first available slot. The logic would choose up to the specified amount of slots (limited by the available future price information), starting from the specified first slot, or from the current time. |
|
For many Central European countries, I found an electricity price data source in CSV format https://www.smard.de/home/downloadcenter/download-marktdaten/ that apparently requires POST requests. The usage conditions https://www.smard.de/home/datennutzung seem to be acceptable. I came across some documentation https://github.com/bundesAPI/smard-api that explains how to use GET requests for some of the data, but I couldn’t find any parameters for retrieving price data. |
dr-m
force-pushed
the
development
branch
2 times, most recently
from
4cd684e to
e536ffd
Compare
|
e536ffd implements the choice of multiple time slots per day, employing a reusable implementation of a binary heap both for choosing the N cheapest time slots and for sorting them by time, to keep the scheduling logic simple. The duration of a time slot will be updated automatically when downloading the data. The price graph displays the currently scheduled time slots with a red background. In other words, this should be ready for the upcoming 15-minute MTU transition of the EU electricity market. |
|
I figured out the minimal POST request format for smard.de. var wc = webclient()
wc.begin('https://www.smard.de/nip-download-manager/nip/download/market-data')
var rc=wc.POST('{"request_form":[{"format":"CSV","moduleIds":[8004169],"region":"DE","timestamp_from":1738018800000,"timestamp_to":1738191600000,"type":"discrete","language":"de"}]}')
var data = rc == 200 ? wc.get_string() : nil
wc.close()All these parameters seem to be mandatory. The timestamps are 1000 times the POSIX
I Implementing this data source would require a little refactoring so that the POST method can be supported. @s-hadinger, do you think that covering the UK and most of the EU would justify the choice of a generic name |
|
I tested d0072eb with a few of the https://smard.de price sources AT, BE, CH, CZ, DE, FR, HU, IT, LU, NL, PL, SI. The currency unit in the main menu status display is a hyperlink to the data source, now also for |
|
Relying on exceptions for bounds checking in |
|
I implemented one last cleanup of import cheap_power
import binary_heap
var make_heap = binary_heap.make_heap, remove_heap = binary_heap.remove_heapAfter these steps, one can paste a slightly edited copy of |
This fetches electricity prices and chooses the cheapest future time slots. Currently, the following price data sources have been implemented: * cheap_power_dk_no.tapp: Denmark, Norway (Nord Pool) * cheap_power_elering.tapp: Estonia, Finland, Latvia, Lithuania * cheap_power_fi.tapp: Finland (Nord Pool) * cheap_power_se.tapp: Sweden (Nord Pool); assuming Swedish time zone * cheap_power_smard.tapp: https://smard.de (Central Europe; local time) * cheap_power_uk_octopus.tapp: United Kingdom (Octopus Energy) See cheap_power/README.md for more details. To use: * copy the cheap_power_*.tapp for your data source to the file system * Invoke BrRestart or restart the entire firmware * Invoke the Tasmota command CheapPower1, CheapPower2, … to * download prices for some time into the future * automatically choose the cheapest future time slots (default: 1) * to schedule Power1 ON, Power2 ON, … at the chosen slots * to install a Web UI in the main menu In case the prices cannot be downloaded, the download will be retried in 1, 2, 4, 8, 16, 32, 64, 64, 64, … minutes until it succeeds. The user interface in the main menu consists of the following buttons: ⏮ moves the first time slot earlier (or wraps from the beginning to the end) ⏭ moves the first time slot later (or wraps from the end to the beginning) ⏯ pauses (switches off) or chooses the optimal slots 🔄 requests the prices to be downloaded and the optimal slots to be chosen ➖ decreases the number of slots (minimum: 1) ➕ increases the number of slots (maximum: currently available price slots) The status output above the buttons may also indicate that the output is paused until further command or price update: (0≤1)0.299‥2.38 ¢/kWh ⭘ Or it may indicate the price and duration until the next active slot: (1≤1)0.299‥2.38 ¢/kWh (0.299) ⭙ 5:05 Or it may indicate the price of the currently active time slot: (1≤1)0.299‥2.38 ¢/kWh (0.299) ⭙ The first number indicates the number of active or scheduled time slots, and the second number indicates the maximum number of time slots per day (default 1). The two quantities around the ‥ are the minimum and maximum known prices from the current time onwards. The scheduled slots are also indicated by red bars in the graph of the available current and future prices.
Description:
CheapPower module
This fetches electricity prices and chooses the cheapest future time slots. Currently, the following price data sources have been implemented:
cheap_power_dk_no.tapp: Denmark, Norway (Nord Pool)cheap_power_elering.tapp: Estonia, Finland, Latvia, Lithuania (Nord Pool via Elering, ex VAT)cheap_power_fi.tapp: Finland (Nord Pool)cheap_power_se.tapp: Sweden (Nord Pool); assuming Swedish time zonecheap_power_smard.tapp: https://smard.de (Central Europe; local time)cheap_power_uk_octopus.tapp: United Kingdom (Octopus Energy)See
cheap_power/README.mdfor more details.To use:
cheap_power_*.tappfor your data source to the file systemCheapPower1,CheapPower2, … toPower1 ON,Power2 ON, … at the chosen slotsIn case the prices cannot be downloaded, the download will be retried in 1, 2, 4, 8, 16, 32, 64, 64, 64, … minutes until it succeeds.
The user interface in the main menu consists of the following buttons:
⏮ moves the first time slot earlier (or wraps from the beginning to the end)
⏭ moves the first time slot later (or wraps from the end to the beginning)
⏯ pauses (switches off) or chooses the optimal slots
🔄 requests the prices to be downloaded and the optimal slots to be chosen
➖ decreases the number of slots (minimum: 1)
➕ increases the number of slots (maximum: currently available price slots)
The status output above the buttons may also indicate that the output is paused until further command or price update:
(0≤1)0.299‥2.38 ¢/kWh ⭘
Or it may indicate the duration until the next time slot and its price:
(1≤1)0.299‥2.38 ¢/kWh (0.299) ⭙ 5:05
The first number indicates the number of active or scheduled time slots, and the second number indicates the maximum number of time slots per day (default 1).
The two quantities around the ‥ are the minimum and maximum known prices from the current time onwards.
The scheduled slots are also indicated by red bars in the graph of the available current and future prices.
Checklist: