Lost connection. Unable to connect to nymea:core

I was trying to reconfigure my nymea:core playing with the internal mqtt client and suddenly my nymea:app has lost the connection with nymea:core.
I tried to reboot the nymea:core with no good result.
Now, when I start my nymea:app, it tries to connect to nymea:core with no result.
Untitled

I can ssh to my nymea:core.
What can I do? Did I lost my entire nymea:core?

Also, I’ve noticed the following logs:



Hmm… you may have caused a situation that crashes the core… This obviously would be a bug.

To get going again, you may log in via SSH and uninstall the plugin you suspect to be the crashing one. e.g. sudo apt-get remove nymea-plugin-mqttclient if you think this happens because of that.

Would be great if you could try to find out what exactly caused this and report it to us so we can fix it.

You may also install systemd-coredump (sudo apt-get install systemd-coredump) which would collect information on the crash when it happens again, which you can then inspect with sudo coredumpctl dump and post it here.

You were right
As soon as I removed the mqttclient plugin the nymea:core worked again.
I reinstalled the mqttclient plugin, just for fun and the same crash of losing connection.
I’ll try to get the logs and post them here.
The coredumpctl is not wworking for me. I’ve got this erorr:
Refusing to dump core to tty (use shell redirection or specify --output).

hmm, is that via Putty? perhaps try to use shell redirection then… like this

sudo coredumpctl dump > somefile.txt

and then see if the file contains anything

yes, via putty
I tried with somefile.txt, but the content is encrypted. Is there a possibility to catch the error in nymea debug interface/server live logs?

perhaps the logs could help too, yes.

but given it crashes instantly, you can’t access the web interface. You can watch the logs via ssh using

sudo journalctl -u nymead -f

Enabling the mqttclient logs may help by editing /etc/nymea/nymead.conf and adding

Mqttclient.debug=true

into the [LoggingRules] section. (Create the section if it does not exist)

also, try sudo coredumpctl info instead of dump. perhaps that works

Here is the log lines captured by sudo journalctl -u nymead -f >somefile.txt

*Nov 18 07:56:40 nymea nymead[23623]: Translations: Could not find nymead translations for "en_GB"*
*Nov 18 07:56:40 nymea nymead[23623]: "/usr/translations"*
*Nov 18 07:56:40 nymea nymead[23623]: "/usr/share/nymea/translations"*
*Nov 18 07:56:40 nymea nymead[23623]:  I | Application: =====================================*
*Nov 18 07:56:40 nymea nymead[23623]:  I | Application: nymead 1.5.1+202211071950~bullseye+rpi1 started as root.*
*Nov 18 07:56:40 nymea nymead[23623]:  I | Application: =====================================*
*Nov 18 07:56:41 nymea nymead[23623]:  W | nymea.mqtt.server: Error listening on port 1883*
*Nov 18 07:56:41 nymea nymead[23623]:  W | Mqtt: Error starting MQTT server on port 1883*
*Nov 18 07:56:41 nymea nymead[23623]:  D | Mqtt: Policy for client "tdev1" added.*
*Nov 18 07:56:41 nymea nymead[23623]:  D | Mqtt: Policy for client "tdev2" added.*
*Nov 18 07:56:41 nymea nymead[23623]:  I | MacAddressDatabase: Database initialized successfully "/usr/share/nymea/nymead/mac-addresses.db"*
*Nov 18 07:56:41 nymea nymead[23623]:  I | NetworkDeviceDiscovery: Created successfully*
*Nov 18 07:56:41 nymea nymead[23623]:  I | NetworkDeviceDiscovery: Loading cached network device information from "/var/cache/nymea/network-device-discovery.cache"*
*Nov 18 07:56:41 nymea nymead[23623]:  I | NetworkDeviceDiscovery: Loaded 0 network device infos from cache.*
*Nov 18 07:56:41 nymea nymead[23623]:  I | Shelly: Joined CoIoT multicast group*
*Nov 18 07:56:43 nymea nymead[23623]:  W | Mqtt: Unable to find a matching MQTT server port for internal client*
*Nov 18 07:56:43 nymea nymead[23623]:  W | default: QObject::connect(MqttClient, ThingSetupInfo): invalid nullptr parameter*
*Nov 18 07:56:43 nymea nymead[23623]:  W | default: QObject::connect(MqttClient, IntegrationPluginMqttClient): invalid nullptr parameter*
*Nov 18 07:56:43 nymea nymead[23623]:  W | default: QObject::connect(MqttClient, ThingSetupInfo): invalid nullptr parameter*
*Nov 18 07:56:43 nymea nymead[23623]:  W | default: QObject::connect(MqttClient, IntegrationPluginMqttClient): invalid nullptr parameter*
*Nov 18 07:56:46 nymea systemd[1]: nymead.service: Main process exited, code=dumped, status=11/SEGV*
*Nov 18 07:56:46 nymea systemd[1]: nymead.service: Failed with result 'core-dump'.*
*Nov 18 07:56:46 nymea systemd[1]: nymead.service: Consumed 2.966s CPU time.*
*Nov 18 07:56:46 nymea systemd[1]: nymead.service: Scheduled restart job, restart counter is at 63.*
*Nov 18 07:56:46 nymea systemd[1]: Stopped nymead - IoT server.*
*Nov 18 07:56:46 nymea systemd[1]: nymead.service: Consumed 2.966s CPU time.*
*Nov 18 07:56:46 nymea systemd[1]: Started nymead - IoT server.*

These lines are repeating over and over. It seams to be a mqqtclient problem as you very well knew.

My MQTT server is running on another raspberry pi and when I configure the mqtt server in nymea I got this log:
W | nymea.mqtt.server: Error listening on port 1883
W | Mqtt: Error starting MQTT server on port 1883

The core dump would really be the most useful thing now… That would tell us immediately which line of code is crashing.

Ok, but I can’t get any info from core dump. All I got is this:

nymea@nymea:/etc/nymea $ sudo coredumpctl info
           PID: 4311 (nymead)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Fri 2022-11-18 12:31:31 EET (12min ago)
  Command Line: /usr/bin/nymead -n
    Executable: /usr/bin/nymead
 Control Group: /system.slice/nymead.service
          Unit: nymead.service
         Slice: system.slice
       Boot ID: b0a96237d2204178b2ad237166f0016c
    Machine ID: 6723526528214ce389b0ee288087c968
      Hostname: nymea
       Storage: /var/lib/systemd/coredump/core.nymead.0.b0a96237d2204178b2ad237166f0016c.4311.1668767491000000.zst
       Message: Process 4311 (nymead) of user 0 dumped core.

                Stack trace of thread 4311:
                #0  0x00000000b643ca50 _ZltRK7QStringS1_ (libQt5Core.so.5 + 0x113a50)

or this (when I redirect > somefile.txt):


           PID: 4907 (nymead)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Fri 2022-11-18 12:46:38 EET (4s ago)
  Command Line: /usr/bin/nymead -n
    Executable: /usr/bin/nymead
 Control Group: /system.slice/nymead.service
          Unit: nymead.service
         Slice: system.slice
       Boot ID: b0a96237d2204178b2ad237166f0016c
    Machine ID: 6723526528214ce389b0ee288087c968
      Hostname: nymea
       Storage: /var/lib/systemd/coredump/core.nymead.0.b0a96237d2204178b2ad237166f0016c.4907.1668768398000000.zst
       Message: Process 4907 (nymead) of user 0 dumped core.

                Stack trace of thread 4907:
                #0  0x00000000b5b417a4 _ZNK10MqttClient11isConnectedEv (libnymea-mqtt.so.1 + 0x267a4)

The last one actually already helps. Will look into it. Thank you!

1 Like

Ok. I fixed the crash in this pull request: MQTT client: Fix a crash when failing to connect to the internal broker by mzanetti · Pull Request #632 · nymea/nymea-plugins · GitHub

This will stop it from crashing, however, will not make the actual MQTT client thing work for you. As your logs show here:

*Nov 18 07:56:41 nymea nymead[23623]:  W | nymea.mqtt.server: Error listening on port 1883*
*Nov 18 07:56:41 nymea nymead[23623]:  W | Mqtt: Error starting MQTT server on port 1883*

there is something preventing nymea from starting the MQTT broker. Most likely you already have another MQTT broker running (mosquito perhaps) which is blocking port 1883. You have 2 options:
Either

a) stop the other MQTT broker and then restart nymea. This would use nymeas integrated MQTT broker and everything should start working

or

b) Disable nymeas internal broker in nymeas system settings and use the other. In this case you can’t use the “Internal mqtt client” thing but instead need to use the regular “mqtt client” thing, providing clientId, username and password as configured in the other MQTT broker to connect to it.

1 Like

Thank you :pray:
I have an external MQTT broker that’s running on a different machine and all my home sensors are connected to that broker and communicate to each other. I’d like to integrate my nymea:core and connect it to that external broker and have the published messages displayed in nymea’s things (temps sensors for example) .
In nymea’ System Settings-> MQTT broker → Server configuration → Interface, I put my external broker’s IP address (local IP address 192.168.1.10)
So in the next step I’ll remove nymea’s internal mqtt broker and I’ll install a regular mqtt client.
I hope it will work :smiley:

oh, I see… Well, the system settings for MQTT will not make nymea connect to an external broker. This would be a new features which is not implemented yet.
In this case you can just disable nymeas internal broker (or keep the default and just not use it) and use the MqttClient thing to connect to the external one, yes (No need to install anything, just when you set up the mqtt client thing, pick the “MQTT client” instead of the “Internal MQTT client” which will allow entering the connection information for the other host

1 Like

Great. I will try this tomorrow.
Thank you Michael

Hi. Since one or two days, I have a similar behaviour, but I’m not using this mqtt plugin you suspect.
The nymea app is always reconnecting to nymea core without any reason. Sometimes it happens when I click on a thing or a button or a group, sometimes it just happens out of a sudden. I use the latest experimental builds but I can not tell you if this strange behaviour started one or two days ago. So I will now try to use my backup from two days ago - first test if the backup functionality I use is working - and then can tell you later if it helped.

well, we’d be thankful if any crashes would be reported so we can fix them. Same is valid, if installing systemd-coredump, the system would start collecting crash dumps whenever something crashes.

The amount and times of crashes can be viewed with sudo coredumpctl list and individual crash dumps can be shown with sudo coredumpctl info [PID]. If no PID is given, the last one would be shown…

Installing libnymea1-dgbsym libnymea-core-dbgsym and - if the crash happens in a plugin - nymea-plugin-...-dbgsym helps to get better core dumps.

1 Like