New MQTT Gateway crashes

16 replies [Last post]
BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010

Hi Brett,

I just noticed the new gateway is crashing my plugboard if it can't connect to the configured broker.

I'm going to try to fix in my rusty Lua but wanted to highlight issue.

 

brett
Offline
Providence, United States
Joined: 9 Jan 2010
xap/mqtt gw lua applet is not fault tolerant

Thanks Bodge.  I did throw it together in a hurry so I'm not suprised.
There is no error handling or retry logic if the MQTT server is either not present or disappears and comes back.  Yep thats something that needs to be added.

Brett

UPDATE: fixed

BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010
MQTT - xAP Gateway - Node-Red
Hi Brett, Thanks for fixing that so quickly. I've had to make some mods on the script to get the Gateway working but I'm wondering if I have misunderstood the scope of the Gateway functionality. I'm using Node-Red to hook into MQTT and send messages. When the Gateway starts up it runs ok but the first time it sees a message it crashes. I tracked the issue down to the gateway not being able to "xap.Frame" the message payload. i.e. payload shows xap message as text, so all the frame elements are empty and so 'source' variable doesn't get filled. I don't think you're using Node-Red yet but wondered how I should be formatting the message to MQTT so that the script can handle it properly. I'm assuming a text message is not the same as a "raw xAP message" as discussed in Wiki for xap.Frame function.
brett
Offline
Providence, United States
Joined: 9 Jan 2010
The gateway should be

The gateway should be transparent it just slurps xap packets and forwards them "as is" into the MQTT broker.

Can you give me an example of an xap packet that crashed it and the resultant stack trace.

Brett

BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010
Hi Brett,I didn't think it

Hi Brett,

I didn't think it was doing anything too clever, that's why I thought I could use it with Node-Red.

Here is the message I have created in Nodered as a text string being sent out through the MQTT node as dbzoo/hal/nodered/xap :

[[xap-hbeat { v=12 hop=1 uid=FF00D820 class=xap-hbeat.alive source=dbzoo.hal.nodered:xap interval=60 }]]

here is the trace(I added a print(payload):

Running...
[[xap-hbeat { v=12 hop=1 uid=FF00D820 class=xap-hbeat.alive source=dbzoo.hal.nodered:xap interval=60 }]]
lua: /usr/share/lua/5.1/xap/init.lua:375: attempt to index local 'value' (a nil value)
stack traceback:
/usr/share/lua/5.1/xap/init.lua:375: in function 'add'
/etc/plugboard/mqttxapGWApplet.lua:60: in function 'callback'
/usr/share/lua/5.1/mqtt_library.lua:588: in function 'parse_message_publish'
/usr/share/lua/5.1/mqtt_library.lua:464: in function 'parse_message'
/usr/share/lua/5.1/mqtt_library.lua:353: in function 'handler'
/etc/plugboard/mqttxapGWApplet.lua:94: in function 'callback'
/usr/share/lua/5.1/xap/init.lua:181: in function 'dispatch'
/usr/share/lua/5.1/xap/init.lua:133: in function 'fun'
/usr/share/lua/5.1/pl/List.lua:392: in function 'foreach'
/usr/share/lua/5.1/xap/init.lua:131: in function 'process'
/etc_ro_fs/plugboard/plugboard.lua:68: in main chunk
[C]: ?

I think the nil value occurs here:

/etc/plugboard/mqttxapGWApplet.lua:60: in function 'callback'

If I try print(f:getValue("xap-header","source"))  I get nil.

Trying wthout the [[]] also brought same.  Then my rudimentary skills broke down and I wondered if the gateway was even supposed to do what I thought it would do.

I understand this was designed to pickup xap messages from the ESP8266 and splurge them into MQTT but I can see it's doing some collecting too...yes?

 

brett
Offline
Providence, United States
Joined: 9 Jan 2010
I take it you have munged the

I take it you have munged the xap text when you pasted into this forum?

Because like that it would be invalid.

[[xap-hbeat { v=12 hop=1 uid=FF00D820 class=xap-hbeat.alive source=dbzoo.hal.nodered:xap interval=60 }]]

Having said that I guess there is a parser bug in that its crashes out trying to handle it, I'll look into that.

1. [[ and ]] at the start/end?  I think you are confusing those with a lua string block delimiter

We do this in lua.

xap = [[
xap-hbeat
{
...
]]

The [[ and ]] mean take literally everything between those markers, they are not part of the xap message.   That also include the \n terminator for each line.

2. Each line must be terminated with \n

In javascript you would need to put those in when building the message.

xap = "xap-hbeat" + chr(10) + "{" + chr(10) etc....

The message should look like this including the \n so each is on its own line.

xap-hbeat
{
v=12
hop=1
uid=FF00D820
class=xap-hbeat.alive
source=dbzoo.hal.nodered
interval=60
}

3. The source= is incorrect and should not have :xap on it for a heartbeat as its invalid.

brett
Offline
Providence, United States
Joined: 9 Jan 2010
Javascript multiblock string
In Javascript you might find it easier to build the xAP message like this:

var
xap= [ "xap-hbeat", "{", "v=12",
"hop=1",
"key=value etc..",
"}"
].join("\n");

BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010
OK, I'll work with that, so

OK thanks, I'll work with that, so often it seems to be I need to know what I don't know I don't know!

Just as additional info incase there is some other issue, I noticed this on the Mosquitto server when the HAH crashes:

1427195029: New connection from 172.16.10.100.
1427195029: New client connected from 172.16.10.100 as dbzoo.hal.Plugboard.
1427195030: Socket read error on client dbzoo.hal.Plugboard, disconnecting.
BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010
I think it's the message node

I think it's the message node in node-red that's causing me the issue.  I can use the function node and enter the above as JS.

brett
Offline
Providence, United States
Joined: 9 Jan 2010
MQTT broker server restarts

MQTT broker server restarts and the mqtt/xap gw crashing I have fixed for that.  You don't have the latest source.  Livebox build 317 does not have this version its coming in 318 as a patch, grab it manually.

https://github.com/dbzoo/hah/blob/master/userapps/hah/xap-plugboard/samp...

This also has a fix in it so that it won't process bad xAP packets and crash, which you did earlier.

Brett

BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010
So far so good Brett.  My

So far so good Brett.  My MQTT gateway has been rock steady for a while.

I've loaded the new patched Gateway...version number was same although I could spot the differences from previous version.

I got my JS message configured properly in Node-Red so the gateway picks it up. Thanks for help on that.  Still a bit of work to do as the message doesn't quite work.  ie command to turn on rf doesn't work direct from Node-Red but when I resend the message using XFXViewer, it does.  So I know I'm close.  Probably source addressing or something.  A lot easier to work on that now the gateway is stable.

Should I be seeing a BSC endpoint in the viewer for the Node-Red output or is the Gateway supposed to be silent?

BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010
OK after a bit of fiddling I

OK after a bit of fiddling I am now able to control RF through Node_red.

var xap = [
  "xap-header",
  "{",
  "v=12",
  "hop=1",
  "class=xAPBSC.cmd",
  "target=dbzoo.hal.Controller:rf.3",
  "source=dbzoo.hal.nodered:xap",
  "uid=FF00D807",
  "}",
  "output.state.1",
  "{",
  "state=toggle",
  "ID=A2",
  "}"
].join("\n");
msg.payload = xap;
 
return msg;
brett
Offline
Providence, United States
Joined: 9 Jan 2010
You should most definately be

You should most definately be able to see the Node-Red endpoint in the xFxViewer.  If you can't then your message may not be well formed.   The viewer should tell you why its rejecting it.

I noticed that you are using ID=A2 in your message I find it easier to just use ID=* and do away with any direct UID endpoint addressing as this can mess things up if you don't get it absolutely right.

Also as a source you don't send from an ENDPOINT like this.

  "source=dbzoo.hal.nodered:xap",
  "uid=FF00D807",

You would use this:

"source=dbzoo.hal.nodered", "uid=FF00D800",

Even if your NODE-RED has a dbzoo.hal.nodered:xap endpoint you never transmit as it.
As you say your addressing is somewhat off which is probably causing you issues.

What you want is something like this on your NODE-RED:
msg.payload = [
  "xap-header",
  "{",
  "v=12",
  "hop=1",
  "class=xAPBSC.cmd",
  "target=dbzoo.hal.Controller:rf.3",
  "source=dbzoo.hal.nodered",  // Use the top level address not an endpoint.
  "uid=FF00D800",  // 00 for the unit a whole, 07 would be an endpoint of the unit.
  "}",
  "output.state.1",
  "{",
  "state=toggle",
  "ID=*", // You can use * here its easier and works the same.
  "}"
].join("\n");
 
return msg;

Brett
BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010
Hi Brett,of course it works

Hi Brett,

of course it works now!

The only thing I notice is that I get a message in xFXVeiewer, about devices having multiple addresses.

All seems to work though, so I assume I can ignore that.  It also says that for the CurrentCost device too.  I am using one of those little dev boards.

 

Thanks as ever for getting me on the right track.  Off now to get incoming xAP messages.

kevin
Offline
Huddersfield, United Kingdom
Joined: 17 May 2010
NODE-RED

Hi Bodge .. sounds good.. what hardware are you running NODE-RED on and have you added it to a HAH setup ?

Have you a xAP node written or perhaps intended for NODE-RED or are you going via the MQTT route ?

I'm using NODE-RED (ThingBox) and have been originating heartbeats and my xAP messages directly, and receiving /parsing xAP in a very crude fashion, but it is working OK .   Playing with MQTT too , but not yet the HAH one but I'll get there.

Re The xAP errors in xFX Viewer I would really encourage people not to drift from the specification.  If Viewer shows an error then its worth fixing, as other apps might balk e.g. iServer - I can point you in the right direction xAP spec wise if you post a problem message.  Sounds like you may have varying source addresses and fixed UID or vv.

The ID=* is a wildcard workaround to including a specific subaddress, however it does often make life much easier for the targeted device if the ID= is defined (when the target is not wildcarded) as it avoids having to search all subaddresses for matches. For example in my C-Bus gateway with over 2200 endpoints it makes a huge time/cpu difference as the ID can directly identify the endpoint.  If your target address is wildcarded then of course your ID= must be too and that's a real workout for the device as multiple endpoint matches can happen.

BodgeIT
Offline
London, United Kingdom
Joined: 10 Jun 2010
Hi Kevin,Running mine on a

Hi Kevin,

Running mine on a PiB+ along with the Mosquitto broker, no issues so far but not too many flows as yet.

One that I'm excited about is scraping the live bus times for the Bus that runs outside my house.  Means just pressing a button on xAPFlash will update me with live bus status, and when when I need to leave to catch the next one.

I only started looking at Node-Red because I think there's a way to link through to Growl with Node which we had both discussed on another thread.  There's so much the two systems can do together.

Definitely xAP node would be way to go, I think, but might be a little out of my reach.  There was a xAP Broadcaster which could be node-ified.  I picked it up from this blog: http://blog.aceshigh.net/2014/04/the-futures-bright-the-futures-red-node... . It was hosted on Github https://github.com/bigkevmcd/node-xap  but that seems to have been deleted and nothing else replaces it...bit of a shame.  I managed to install it before it got pulled but couldn't get it to work, so I could recover it I think and maybe expand it a little.

 

I've started a seperate thread to discuss xAP addressing issues.

kevin
Offline
Huddersfield, United Kingdom
Joined: 17 May 2010
NODE-RED xAP node

Martyns xAP function didn't work for me either but I didn't spend much time working out why. ISTR it was a call into some nodeJS function and I think I had it installed wrong. Regardless a  dedicated  xAP BSC node would be better but probably beyond me too. I'm doing useful xAP things with just the TCP node and iServer so I'm making progress.

I've looked a bit more at the  xAP MQTT bits that Brett posted on GitHub so I'm more familiar with how you're working currently.

The bus thing sounds novel and useful.

Keep us posted....  K

Hardware Info