Logs Database History Storage

Hi , I have extended the history log files from 200000 to 800000
(logDBMaxEntries=200 000)
Now my question is, if its possible to store these log files on to an external SD card because I want to keep the data for six months.

Thanks !

Yes, you can specify the file name in the settings file as described here:

https://nymea.io/documentation/users/usage/advanced-configuration#section-logs

Thank you for your reply.
Can you tell me please how to calculate the log history time (200000)
eg what is the correct number for 6months

Hmm. So the number is the amount of entries. How much time that will be depends on how many devices you have, how much data they’re logging etc.

If I remember correctly, 200000 entries would create a log db with a maximum size of approximately 30MB.

So if you have a lot of storage, you can increase that number by a lot.
You may even set it to -1, which should disable removing old entries completely. Obviously you’d have to make sure you won’t hit the storage limit yourself.

Ok understood , I will test it…

You all have done an excellent job on creating this application. I’m using it in the factory for almost one year now for monitoring the fridges temperature. It is so stable , reliable with very useful and modern graphics.
Congratulations !

2 Likes

Currently my whole setup is run from an SD card in an Raspberry Pi. I would like to separate the Logs database storage onto an USB stick, while everything else remains on the SD card in hope, the continious writing on the card will be reduced, and to be able to store a bigger number of events. Above @mzanetti mentioned that the database location can be defined in the configuration file.
I just want to ask, if there are experiences with this kind of setup and also would like to know what generally happens with a running or restarting system, if for example, the log database is not available for some reasons?

Moving the database to a different location via config file is working just fine. If, after a restart, the database is not available, nymea would try to create a new one. In case it fails for whatever reason, it would warn about that in the debug log output and continue without logging. Some features relying on the logs (e.g. charts, notification history etc) would then not work and simply show empty lists.

If by accident a second log db is created, they can be merged again, by manually exporting entries from the logEntries table from one db and importing them into the other with the sqlite tool of your choice.

Note that energy logs in the energy view are different. Those are stored in /var/lib/nymea/energylogs.sqlite and currently can’t be moved via a config file. In that case it would be required to mount the USB stick or whatever to /var/lib/nymea/ which would also move some other data (like scripts) to the external drive. Generally that’s not a problem either, but you really want to make sure it is available when nymea starts as messing this up will likely lead to problems in the configuration. I suppose I’d recommend to also mount /etc/nymea to the same external media in that case, so if something goes wrong, they stay in sync at least and would continue to work once the drive is plugged back in.

Also, I am not sure if a USB stick is necessarily a better solution than the SD card as they aren’t really made for continuous access either. There are SD cards available which are said to be more robust, labelled “max endurance” or “industrial” and the likes, I haven’t ever used one so I can’t say much about them. I suppose the best and easiest (not the cheapest) solution for large and extensively used systems (50+ devices) would be to install the whole system on an SSD drive. Booting from USB works just fine from Raspberry Pi 3 and up.

@mzanetti thank you very much for your detailed reply. As the energy view is separated from the normal database and together with the scripts, it does not make sense to separate the parts in order to reduce write operations on the SD card. I guess the energy logs are written to the medium as often as the other database.
I hoped, that if something would fail by too much writing (even the USB stick), then the main system would continue to operate just without charts on a separate storage medium like the SD card. So I will stick with the SD card for the moment and maybe move to an SSD drive eventually.