how to run the images in standalone mode, ie out of home-assistant ? #5

alexandrecuer · 2023-10-05T11:04:12Z

alexandrecuer
Oct 5, 2023
Maintainer

See also on https://emoncms-docker.github.io/

All the following run commands include the --rm option, so that the container will not persist after closing, it will be destroyed. These are just starting points to get familiar with the container approach...

The most basic launch, with the default parameters coming from the Dockerfile (mosquitto log level fixed to error and emoncms log level fixed to 2) :

docker run --rm -it -p 8081:80 -p 7883:1883 alexjunk/emoncms:alpine3.18

To have more verbosity in the emoncms and mosquitto logs :

docker run --rm -it -p 8081:80 -p 7883:1883 -e MQTT_LOG_LEVEL=notice -e EMONCMS_LOG_LEVEL=1 alexjunk/emoncms:alpine3.18

For mosquitto, you can add more than one log level like that :

docker run --rm -it -p 8081:80 -p 7883:1883 -e MQTT_LOG_LEVEL="error warning information notice" alexjunk/emoncms:alpine3.18

To disable feedwriter and go out of low-write mode :

docker run --rm -it -p 8081:80 -p 7883:1883 -e MQTT_LOG_LEVEL=notice -e EMONCMS_LOG_LEVEL=1 -e REDIS_BUFFER=0 alexjunk/emoncms:alpine3.18

Dont forget adjusting the image name to your needs, especially if you proceed to a local build

Answered by alexandrecuer

Oct 16, 2023

@gadgetbazza : I just did a manual restore as you intend to do and it works

On the host machine, I have a /data folder belonging to the current user

start the container :

docker run --rm -v /data:/data -p 8081:80 -p 7883:1883 -it alexjunk/emoncms:alpine3.16

it will create the /data/emoncmsfolder

open a shell session in the folder where you have an emoncms tar.gz backup :

sudo cp emoncms-siegecerema-2021-2022.tar.gz /data/emoncms/backup/uploads/

check the container id :

docker container ls
CONTAINER ID   IMAGE                        COMMAND   CREATED         STATUS         PORTS                                                                              NAMES
0ae9165de664   alexjunk/emo…

View full answer

alexandrecuer · 2023-10-05T18:14:42Z

alexandrecuer
Oct 5, 2023
Maintainer Author

But using docker compose is a smartest option than all those docker run commands

docker compose is an orchestrator written in go.

To install on linux : https://github.com/docker/compose

It is an easy process as you just have to download a binary file for your OS/arch

Then you have to create a compose.yaml file which look like that :

services:
  emoncms:
    image: alexjunk/emoncms:alpine3.18
    environment:
      - TZ=Europe/Paris
      - REDIS_BUFFER=0
      - MQTT_LOG_LEVEL=error
      - EMONCMS_LOG_LEVEL=2
    ports:
      - 8081:80
      - 7883:1883
    restart: always

To run all that, a simple command :

docker-compose up

This is nice, supposing you stop the orchestrator with Ctrl-C, the container stops. It will anyway be restarted when you will restart your computer, due to the restart policy always.

A more convenient way is to use the -d option :

docker-compose up -d

You won't see anymore the container log on the command line. To check it : docker service logs emoncms

At this stage, you do not achieve a consistent data persistancy. If for some reason, you recreate the container because a new image with new features is available, all monitoring data is lost.

So let's create a folder named data on the host and start emoncms with a volume.

The compose file has to be modified like that :

services:
  emoncms:
    image: alexjunk/emoncms:alpine3.18
    volumes:
      - /data:/data
    environment:
      - TZ=Europe/Paris
      - REDIS_BUFFER=0
      - MQTT_LOG_LEVEL=error
      - EMONCMS_LOG_LEVEL=2
      - MYSQL_DATABASE=emoncms
      - MYSQL_USER=mysqlusername
      - MYSQL_PASSWORD=myqlpassword
    ports:
      - 8081:80
      - 7883:1883
    restart: always

Adjust MYSQL_USER and MYSQL_PASSWORD to the custom values of your taste. Please note you can only do that at the creation of the mariadb database.

For the MQTT broker, you can change the credentials values when you want :

services:
  emoncms:
    image: alexjunk/emoncms:alpine3.18
    volumes:
      - /data:/data
    environment:
      - TZ=Europe/Paris
      - REDIS_BUFFER=0
      - MQTT_LOG_LEVEL=error
      - EMONCMS_LOG_LEVEL=2
      - MYSQL_DATABASE=emoncms
      - MYSQL_USER=mysqlusername
      - MYSQL_PASSWORD=myqlpassword
      - MQTT_USER=emonpi
      - MQTT_PASSWORD=emonpimqtt2016
    ports:
      - 8081:80
      - 7883:1883
    restart: always

9 replies

alexandrecuer Oct 16, 2023
Maintainer Author

You should not copy the file, you can use the import backup button
it puts the file at the right place. the file must be a tar.gz and not a zip

gadgetbazza Oct 16, 2023

I can’t use the browser import, it’s too big at 2.64Gb and I’m following the instructions on the import page for a manual import.

alexandrecuer Oct 16, 2023
Maintainer Author

OK, understood, so you first connect to the running container and run the restore script from the container or from the host ?

the line at which you are stuck in the script is here :
https://github.com/emoncms/backup/blob/9cfc5b8552eefb3c988f05b813aad1f5d67e88b8/emoncms-import.sh#L60C24-L60C42

if the tar.gz file is in /data/emoncms/backup/uploads, I dont see why it does not work

if you launch the command cd /var/emoncms/backup/uploads && ls -t *.tar.gzonce connected to the running container, what does it return ?

extra thought : the max file size is 3 Gb :

emoncms/emoncms_pre.sh

Line 111 in f3dd59e

echo "upload_max_filesize=3G" >> $PHP_CONF/backup.ini

if you are on a local network, the browser import should work

alexandrecuer Oct 16, 2023
Maintainer Author

@gadgetbazza : I just did a manual restore as you intend to do and it works

On the host machine, I have a /data folder belonging to the current user

start the container :

docker run --rm -v /data:/data -p 8081:80 -p 7883:1883 -it alexjunk/emoncms:alpine3.16

it will create the /data/emoncmsfolder

open a shell session in the folder where you have an emoncms tar.gz backup :

sudo cp emoncms-siegecerema-2021-2022.tar.gz /data/emoncms/backup/uploads/

check the container id :

docker container ls
CONTAINER ID   IMAGE                        COMMAND   CREATED         STATUS         PORTS                                                                              NAMES
0ae9165de664   alexjunk/emoncms:alpine3.16   "/init"   2 minutes ago   Up 2 minutes   0.0.0.0:8081->80/tcp, :::8081->80/tcp, 0.0.0.0:7883->1883/tcp, :::7883->1883/tcp   flamboyant_bohr

and open a shell session to the running container :

docker exec -it 0ae9165de664 sh

check you you are :

whoami
root

Browse to the /data/emoncms/backup/uploadsand give the file to the apache user....anyway, I think this should not be necessary as you are root (through the web browser, you are apache and not root)

/data/emoncms/backup/uploads # ls
emoncms-siegecerema-2021-2022.tar.gz
/data/emoncms/backup/uploads # ls -al
total 1396
drwxr-xr-x    2 apache   root          4096 Oct 16 18:03 .
drwxr-xr-x    3 apache   root          4096 Oct 16 17:58 ..
-rw-r--r--    1 root     root       1418092 Oct 16 18:03 emoncms-siegecerema-2021-2022.tar.gz
/data/emoncms/backup/uploads # chown apache emoncms-siegecerema-2021-2022.tar.gz
/data/emoncms/backup/uploads # ls -al
total 1396
drwxr-xr-x    2 apache   root          4096 Oct 16 18:03 .
drwxr-xr-x    3 apache   root          4096 Oct 16 17:58 ..
-rw-r--r--    1 apache   root       1418092 Oct 16 18:03 emoncms-siegecerema-2021-2022.tar.gz
/data/emoncms/backup/uploads # cd /opt/emoncms/modules/backup/

then go to /opt/emoncms/modules/backup and run the script :

/opt/emoncms/modules/backup # ./emoncms-import.sh 
=== Emoncms import start ===
2023-10-16-18:05:55
Backup module version:
cat: can't open './backup/module.json': No such file or directory
EUID: 0
Reading ./config.cfg....
Location of data databases: /data/emoncms
Location of emonhub.conf: 
Location of Emoncms: /var/www/emoncms
Backup destination: /data/emoncms/backup
Backup source path: /data/emoncms/backup/uploads
Starting import from /data/emoncms/backup/uploads to /data/emoncms/backup...
./emoncms-import.sh: line 26: systemctl: command not found
./emoncms-import.sh: line 27: systemctl: command not found
./emoncms-import.sh: line 28: systemctl: command not found
./emoncms-import.sh: line 29: systemctl: command not found
ls: /boot: No such file or directory
old image
Backup found: emoncms-siegecerema-2021-2022.tar.gz starting import..
Read MYSQL authentication details from settings.php
Decompressing backup..
emoncms.sql
settings.ini
phpfina/
phpfina/5.meta
phpfina/3.dat
phpfina/12.dat
phpfina/2.meta
phpfina/16.dat
.............................................
phpfina/9.dat
phpfina/41.meta
phpfina/9.meta
phpfina/54.meta
phpfiwa/
phptimeseries/
Removing compressed backup to save disk space..
Stopping services..
Emoncms MYSQL database import...
Import feed meta data..
Restore phpfina and phptimeseries data folders...
chown: unknown user/group www-data:root
chown: unknown user/group www-data:root
OK
Updating Emoncms Database..
["ALTER TABLE `postprocess` ADD `processid` int(11) NOT NULL auto_increment PRIMARY KEY","ALTER TABLE `postprocess` ADD `status` varchar(255)","ALTER TABLE `postprocess` ADD `status_updated` int(11)","ALTER TABLE `postprocess` ADD `status_message` varchar(255)","ALTER TABLE `postprocess` ADD `params` text"]
2023-10-16-18:05:58
=== Emoncms import complete! ===
./emoncms-import.sh: line 196: systemctl: command not found

Answer selected by alexandrecuer

gadgetbazza · 2023-10-19T21:57:58Z

gadgetbazza
Oct 19, 2023

Hi, solved the restore problems... The export download I produced was uncompressed courtesy of Safari when I downloaded it. Re-doing the backup and downloading using chrome kept it as a .tar.gz and this worked as expected.

Up and running now with the restore, but I see no EmonHub module. Any reason why this isn't visible as I still want to use it to retrieve from modbus?

2 replies

alexandrecuer Oct 20, 2023
Maintainer Author

@gadgetbazza : emonhub should be in another container. I've created another container for radio things (but not modbus yet) : https://github.com/Open-Building-Management/emontx_sniffer
Actually this is an emoncms docker image not an emonpi and it should not integrate things related to hardware input output, such as /dev/tty*

I've not planned to work on an emonhub image yet but I haven't really thought about it seriously, and indeed it's an interesting option. The only point is that emonhub is pretty hairy, with radio things and wired things like modbus are not supposed to work the same way

gadgetbazza Oct 20, 2023

Hi, but EmonHub isn't specifically an emonpi component... Sure, one of its capabilities is for the radio interface, but it also offers other interfaces, in particular, I am interested in Modbus data feeds to be pulled into emoncms.

For my use case, I am looking to have a few emonpi's located near emonTx devices connected over USB and then have those emonpi's push their feed data to a central emoncms running on the docker host using the ability for emoncms to publish it's content between hosts.

I also have other data sources using Modbus over TCP/IP, I was hoping to use the docker emoncms to pull those feeds and have the docker emoncms as the central location for all data. I could use one of the emonpi's to fetch that and proxy it over to the docker emoncms, but I was trying to avoid this as this introduces one more point of dependancy. In addition, I also have experienced issues where I have updated an emonpi to a later version of emoncms and broken the modbus interface, the fix was to bring the base image up to a newer version. Ideally, I don't want to change these outlying devices very much, just set them up, have them send the data centrally and then forget about maintaining the software on them and focus on the central emoncms.

Hope that makes sense. I really appreciate the work you have done here though as it's interesting to me. I don't understand the nuts and bolts of docker and the changes you have had to make to facilitate this, it's beyond my capabilities, so I am very grateful.

gadgetbazza · 2023-10-19T23:34:56Z

gadgetbazza
Oct 19, 2023

To follow on, I see a few of the components are not in this emon installation compared to my emonpi. I recognise that not all are relevant, but is it possible for me to add in EmonHub, EmonHubConfig and DemandShaper?

9 replies

gadgetbazza Oct 20, 2023

Incidentally, I don't think the emonhub service is running, probably needs modifying as you have done with other components, the UI is working and I'm able to manage the configuration, but there is no log output and nothing is being read from ModBus currently for me.

alexandrecuer Oct 21, 2023
Maintainer Author

@gadgetbazza : yes indeed, it is not possible that emonhub works just by adding the source.

By the way, you just want modbus TCP isn'it, not RTU and not radio 433 Mhz ? emonhub is not the best tool for modbus, the modbus interfacer was just experimental.

I have got a standalone modbus TCP sniffer which can send data payloads to a MQTT broker. You could run it in another container. But there is a bit of work to adapt as there is no UI to manage the conf. it can address as many TCP server as you want....

gadgetbazza Oct 21, 2023

Yes modbus tcp. Never had any trouble with the emon one myself. I guess an alternate container / MQTT approach may be sensible .

alexandrecuer Oct 23, 2023
Maintainer Author

Yes indeed, the emon interfacer works. But is it easy to interrogate different IP and different servers ? I dont remember...

alexandrecuer May 1, 2024
Maintainer Author

@gadgetbazza : did you update to a recent version ? for example Latest version or alpine3.19_emoncms11.5.0

You should because I had problem with the apache log growing inside the container, as logrotate was not installed. With recent version, the apache log is routed to the docker log, so the user can use docker log rotation which is handy compare to logrotate. Anyway, it's not that dramatic, I still have older versions running on some spots, and even if the apache log is getting big, the machine is running fine.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to run the images in standalone mode, ie out of home-assistant ? #5

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 20 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

how to run the images in standalone mode, ie out of home-assistant ? #5

alexandrecuer Oct 5, 2023 Maintainer

See also on https://emoncms-docker.github.io/

Replies: 3 comments · 20 replies

alexandrecuer Oct 5, 2023 Maintainer Author

alexandrecuer Oct 16, 2023 Maintainer Author

gadgetbazza Oct 16, 2023

alexandrecuer Oct 16, 2023 Maintainer Author

alexandrecuer Oct 16, 2023 Maintainer Author

gadgetbazza Oct 19, 2023

alexandrecuer Oct 20, 2023 Maintainer Author

gadgetbazza Oct 20, 2023

gadgetbazza Oct 19, 2023

gadgetbazza Oct 20, 2023

alexandrecuer Oct 21, 2023 Maintainer Author

gadgetbazza Oct 21, 2023

alexandrecuer Oct 23, 2023 Maintainer Author

alexandrecuer May 1, 2024 Maintainer Author

alexandrecuer
Oct 5, 2023
Maintainer

Replies: 3 comments 20 replies

alexandrecuer
Oct 5, 2023
Maintainer Author

alexandrecuer Oct 16, 2023
Maintainer Author

alexandrecuer Oct 16, 2023
Maintainer Author

alexandrecuer Oct 16, 2023
Maintainer Author

gadgetbazza
Oct 19, 2023

alexandrecuer Oct 20, 2023
Maintainer Author

gadgetbazza
Oct 19, 2023

alexandrecuer Oct 21, 2023
Maintainer Author

alexandrecuer Oct 23, 2023
Maintainer Author

alexandrecuer May 1, 2024
Maintainer Author