ESP8266 very cheap WIFI ioT module
I've been away from the forums for a while with a few home improvments and building jobs taking place and missing my hobby time!!
I hope everyone is well and continuing with their HAHs and surrounding projects, i would hate to see a great project like this fade away with loss of interest
fairly often i search for new Arduino modules focusing around networking as i believe this is the way to go for all things IoT and HA rather than the current 433 / 868mhz RF radios we tend to play with, i know Brett did some work on the standalone Ethernet relay module sometime back http://www.dbzoo.com/livebox/ethrelay
This little beauty came up which appears to be a fairly new entry to the makers market and is exactly what i've been looking for and at such a low cost
I wonder if we could Xap enable it, some bright spark has already coded up a lua interpreter firmware - https://github.com/nodemcu/nodemcu-firmware here so i gather its reasonable possible
Thanks Andrew
Well I got the ESP chip flashed with the LUA running firmware and managed to make it connect to my iServer and display an xAP packet from a specific source. Baby steps. Code for this below:
function xapReceive(sck, msg)
for i in string.gmatch(msg,"<%a->(.-)</%a->") do
print(i)
end
end
function login(sck, c)
if c == "<ACL></ACL>" then
sck:send("<cmd>log<code>passwd</cmd>")
-- eat the <ACL>Joggler</ACL>
sk:on("receive", function(sck,msg)
sck:send("<cmd><flt+>dbzoo.arduino.weather</cmd>")
sk:on("receive", xapReceive )
end)
end
end
sk = net.createConnection(net.TCP, 0)
sk:on("receive", login)
sk:connect(9996, "192.168.1.8")
Very interested in the potential of this little fella! I think this can really make things like the etherRelay workable in most places as having a Cat5 cable where you want the relay was always an issue for me.
order placed!
Ha. Yeah lesson learned, always run a cable when possible. that said, there are occasions where wireless is a must.
I have had enormous success with jeenodes, doing jobs like heating and water relays, tank temp monitoring, motorbike battery voltage reporting, veg patch soil moisture measuring and irrigation switching(to keep swmbo happy), ir sending, the list goes on! Always rock solid.
Jeenodes are really good at this so the esp modules have a lot to match up against but I can see the benefits, looking forward to playing with them.
I have just bought version 1 for now, very cheap (<£3 delivered), header pins and built in antenna. Only got a few gpio pins but it's enough to test and think about what I can use it for.
decoding a xap message and taking action would be great, standalone web control such as the etherRelay also has some legs I think.
lots of stuff on the net already to read about. Interesting project. Nice find!
Garry.
Update of where I'm at. I managed to get about 5hr on the weekend to work on this and I made great progress. The LUA xAP library that runs on the ESP8266 is almost finished. I'm just testing a few things out before I post it. Next is the BSC framework.
There are memory limitation on this chip (20k) so I had to break things apart and you will need to compile up the source (on chip) to reduce the LUA memory footprint.
I even rewrote the xAP parser so that coroutines aren't used; to squeeze more memory.
The sample I'm working towards is an xAP node that exposes all the the GPIO pins as xAPBSC endpoints. Once I get that working (which should be this week) I'll post it up.
I also want to put together some simplier samples that show various things that are also possible. Once I'm done coding I'll backtrack and document what you need to do to use this. I've tried to keep to the same LUA patterns that we are use to coding for the HAH and Corona so the API should be familar.
Got to love LUA. It was good choice and very portable.
Brett
FWIW: This is the unit I'm using
LUA is lean but 20k of RAM just isn't enough to stand up a xAP stack and have it work. An alternative approach is going to be needed. A little like the RoomNode. You have the BSC endpoint being managed by the HAH with a lightweight protocol being used to communicate with the endpoint.
Perhaps mqtt could be that transport layer given or maybe just make the WIFI node look like a RoomNode ?!
I'll commit the ESP lua code that I've written so far (2mrw).
Where I was at before grinding to a halt over memory issues
If you want to play with this after uploading each file you need to compile then on the ESP chip.
node.compile('Frame.lua')
For Frame, Filter and xap etc..
The samples don't need to be compiled up just those files that are "required".
you can dofile("sample1.lua") to execute those.
Going the mqtt route I'd have to put a mqtt server on the livebox and some LUA extensions to interactive with it.
These play nicely together http://mosquitto.org/download/ and http://geekscape.github.io/mqtt_lua/
Attachment | Size |
---|---|
Filter.lua | 2.55 KB |
Frame.lua | 1.56 KB |
xap.lua | 2.11 KB |
sample1.lua | 172 bytes |
sample2.lua | 560 bytes |
sample3.lua | 616 bytes |
Mapping xAP to MQTT
Messages in MQTT are published on topics. Topics are treated as a heirarchy. This is equivalent to the address heirachy built up using the xAP source names.
Consider the following xAP message
xap-header { v=12 hop=1 uid=FF00D801 class=xAPBSC.cmd target=dbzoo.beaglebone.jeenode:office.relay source=dbzoo.acme.test } output.state { state=on }
We can map this into an MQTT tpoic space based on its target address
dbzoo/beaglebone/jeenode/office/relay
MQTT supports wildcarding + and #, these functionally are equivalent to * and > in the xAP world.
Filter mapping examples
xAP: dbzoo.beaglebone.jeenode:> MQTT: dbzoo/beaglebone/jeenode/# xAP: dbzoo.beaglebone.jeenode:*.light MQTT: dbzoo/beaglebone/jeenode/+/light
There is a nice overlap here.
What I want is client to be able to register their TOPICS (xAP Source address) with a broker and have some other xAP/MQTT gateway look at all topics registered and if its see an xAP targeted for a topic it will automatically forward this to MQTT.
For example this packet would come in.
xap-header { v=12 hop=1 uid=FF00D801 class=xAPBSC.event source=dbzoo.beaglebone.jeenode:office.relay } output.state { state=on }
Gateway would examine the source address and then check if there is a MQTT topic for it. The gateway would listen on topic /# all topics and look for source= lines to build up a list of publishers that need servicing.
If an incoming xAP target matches a source the gateway has registered. It would strip the header down and forward the rest to the /topic - Using large the XAPBSC.cmd from above.
xap-header { source=dbzoo.acme.test class=xAPBSC.cmd } output.state { state=on }
On the ESP we need our FRAME processing logic as per before and we can deal with different class and our response to them.
xAP still provide the PROCOTOL of what is going on, MQTT is simply the transport layer instead of using broadcast UDP or the iServer.
This means we can do away with the XAP.lua and Filter.lua processing files making our memory requirements much smaller as we are using built in C library to push/pull messages in and out of the system.
Demonstration
Rather than explain it in lots of words a demo.
On one terminal I type this:
$ lua sub1.lua
On another I do this
$ lua pub1.lua
On the original terminal I now see this output
$ lua sub1.lua
STATE: on
What I'm doing is using MQTT as the transport but I'm still pushing xAP packets through the broker. What remains is the Frame.lua class and the rest can be taken away this should make it light enough to process on the ESP chip still using xAP as the protocol for saying what has to happen.
I need to work some more down this route and see what happens....
Attachment | Size |
---|---|
sub1.lua | 610 bytes |
pub1.lua | 390 bytes |
Thanks for your efforts as always Brett. The process you describe sounds ideal to me. As you say, very similar in concept to the HAHnodes.
Cant wait to have a play when mine arrives.
No idea what I'm gonna use it for but sure I'll find something.
Garry
Work in progress. This is my xAP to MQTT gateway - I've not hook it up to the ESP chip yet but from testing it looks solid and a go'er. Its a plugboard applet.
To get it working you only need the two extra LUA libraries from https://github.com/geekscape/mqtt_lua/tree/master/lua
Set up a MQTT broker somewhere (I'll get this into the livebox distro later) and tweak the GW to point at it.
For those using beaglebones or RaspPI installing mosquitto (mqtt) is a snap - this is what I'm using.
... on to coding up the ESP chip to transmit and receive xAP packets via MQTT, which will magically interface via my GW to the rest of the xAP traffic on my network. That's a job for 2mrw.
xAP/Mqtt gateway: http://livebox-hah.googlecode.com/svn/branches/portable/userapps/hah/xap...
It's arrived...woohoo! But won't get time to play till weekend.....boo!
good work on the MQTT XAP stuff Brett. At this rate it'll be up and running before I get chance to play :)
for info, what firmware did you flash?
thanks,
Garry
ps it's much much smaller than I had imagined! Amazing.
I flashed version 0.9.5 of NodeMCU - this is the latest released, the flasher will download that for you. This page is what I consulted to get me going http://benlo.com/esp8266/esp8266QuickStart.html
I'm making some headway. I get about 2hrs a night to work on this so progress is slow (for me).
I've run aground for some reason my esp chip won't do an mqtt connect to my server ... I'm dead in the water. I'm starting to think about coding in C and removing all these layers of cruft.
Embedded stuff scream C to me - LUA was fun but when things don't work you've got no where to go!
For your enjoyment this is the client I was trying to get to work - it should happily operating with my gateway (not as published on this thread - this has a bug).
The GW is published as a sample applet in the "portable" distribution along with some other xap framework tweaks to make it happy.
Attachment | Size |
---|---|
espBscClient.lua | 1.13 KB |
Not that I thought It would be any different but I have same issue.
mosquitto on pi and mqttspy on PC can connect to it fine.
es8266 flashed with nodemcu and uploaded with numerous mqtt client examples. No connection :(
this little module is excellent though, be nic to use it like a jeenode somehow?!
Garry
Success.
i connect perfectly with the following firmware version : nodemcu_20150123
Downloaded from
https://github.com/nodemcu/nodemcu-firmware/tree/master/pre_build
obviously some regression in recent builds :(
Garry.
Gary Thanks for the confirmation. This was next on my list of things to do. Glad you beat me to it.
With the code that I've posted we should be good to get these talking xAP now - as long as its heap space hold out (my next worry) as node.compile() was only introduced in nodemcu_20150212.bin
Did you try my gateway and the espBscClient.lua sample? This should work as it all held together when I did this off of the ESP chip with the standalone MQTT libraries.
The sample needs a little more work to make it useful, like when the status change toggle a GPIO pin but thats not a hard change. Onwards.... (again).
Brett
Hi Brett
I burnt so much time on this that as soon as I got a mqtt connect from one of the demos, I did a victory dance and went to bed. Therefore not tried your script yet.
One issue based on your node.compile comment. Don't we need this to compile Frame.lua for your script to run?
anyhow, will get some time to play again later hopefully.
Cheers
Garry
So glad we've made this addition, alot of exiting possibilities, here's one:
http://nathan.chantrell.net/20141230/wifi-mqtt-display-with-the-esp8266/
Ok I have tried your code Brett. I had to remove some curly brackets around the topic in c:subscribe which allowed it to run but then run into this issue
http://www.esp8266.com/viewtopic.php?f=19&t=1278&start=30
protection error while sending. looks like trying to send messages too close together is an issue.
Garry
Great minds think alike Bodge! Ordered one of these myself yesterday
http://www.ebay.co.uk/itm/400846739132?ru=http%3A%2F%2Fwww.ebay.co.uk%2F...
To get the gateway running, where do I put these libraries mentioned here:
To get it working you only need the two extra LUA libraries from https://github.com/geekscape/mqtt_lua/tree/master/lua
Edit:
I'm think I can only do this if on pi? The folder for lua libs is ro, is that correct?
I'm thinking some sort of control unit like this:
http://hazymat.co.uk/2015/01/the-spec-for-the-ultimate-home-control-panel/
EDIT: Sorry to go off topic a little!
Stick lua files in /usr/share/lua/5.1
garry
EDIT: for pi users
hmm, not on Pi yet...
Andrew,
Is this the beginning of replacing xAP. NO.
In anycase MQTT is not a protocol its a TRANSPORT layer. xAP is a protocol.
These are different things. xAP can ride on and be delivered by MQTT.
Currently we use UDP Broadcast packets to transmit the xAP protocol. This comes with some restrictions in that you can't extend beyond your existing LAN. We can also use the iServer for TCP based connection - this is what the joggler uses.
MQTT another way of moving data around.
The reason I'm looking into MQTT is because its built in and connecting the ESP chip via the iServer is using too much resource which causes the chip to reset. Had that not happened I would not even be looking into MQTT (well not immediately) :)
--
The MQTT libraries are in the portable distribution but are not delivered on the livebox. If you really want to use them on the livebox put them into the /etc/plugboard directory and just "require" as normal. They don't have to be in the /usr/share/lua/5.1 path.
The xapmqttgw applet sample won't work on the livebox unmodified as I introduce a new API change in the xap library. -> frame:setValue(section,key,value)
You can work around that being missing with -> frame[section].key = value
The internals for Frame in the portable distribution are different from those on the livebox distribution. On portable you would say frame.frame[section].key = value
To avoid this problem I create a new API so the internals are not exposed.
Really I should re-align the two distributions as they have drifted in some regards.
In anycase you are still going to need a mqtt server somewhere which is not present in the livebox distro.
It might be a week before I come back to looking at this. I'm very time poor at the moment.
Brett
Back to basics. A very simply program that sends xAP heartbeats via MQTT to be picked up by the gateway applet and published. Sample output from a run:
dofile(heartbeatTest.lua) Tuesday, 3 March 2015 12:20:10
Connecting to MQTT broker. Please wait...
> Connected
Heartbeat Sent
Heartbeat Sent
Heartbeat Sent
Heartbeat Sent
Proving this stuff works one you get around its warts and it does seem to have a lots of those.
Attachment | Size |
---|---|
heartbeatTest.lua | 519 bytes |
Brett, like you say there appears to be some warts. One big one is the crash caused if multiple publish events are called at similar times.
There are a couple of clunky workarounds I've found but think until this is fixed in the firmware it will always be just that. a workaround. Error is logged on Github so let see.
Anyhow, I have knocked up an ugly mashup of various bit of code to get a working (sort of) version of your xapclient code.
This seems to send heartbeats regularly and is receiving payloads:
Dofile("xAPClient.lua")
Connecting to MQTT broker. Please wait...
heap: 6872
> Connected
subscribe success
payload received. heap: 6184
heartbeat sent. heap: 6072
heartbeat sent. heap: 5840
heartbeat sent. heap: 5776
payload received. heap: 5968
heartbeat sent. heap: 5880
heartbeat sent. heap: 5776
payload received. heap: 5968
payload received. heap: 5816
payload received. heap: 5816
payload received. heap: 5816
The code has an issue with the Frame call but just crashes with no message so I cannot debug (hence I've just commented out for now)
anyhow, ive attached it for info, nice to get some progress.
Don't worry about your time on this Brett, you put more than enough into this project and its much appreciated.
Got a feeling this wont be a quick implementation anyhow, your thoughts earlier on writing in C may still be the way to go?.
Attachment | Size |
---|---|
xAPClient.lua | 2.76 KB |
Gary, good job on making some progress. LUA is nice but I think there are many things workings against us.
1. The heap is too small to write anything of substance. Especially as you have to workaround firmware bugs using yet more LUA.
2. node.compile() which solves 1. is in later releases but those have broken MQTT implementations.
3. The MQTT when it does work is problematic with its publishing implementation which requires more LUA code consuming yet more heap bring you back to point 1 and 2.
As you say its "close" and as its matures I'm sure these will be solved.
For now a pure C implementation where get 40k of HEAP all to yourself is going to be the way to go, which also negates the need for MQTT as then I can do UDP broadcast and make it "just work".
MQTT was nice thou and the gateway/bridge I put together worked rather well. Of course I would need some sort of heartbeat reaper so the gateway could unsubscribe to XAP message for which there is no longer a subscriber, but as a proof-of-concept it did the job.
Brett
I used the Windows based ESP8266 devkit from http://www.esp8266.com/viewtopic.php?f=9&t=820&sid=783b2f3021078ab657221...
and I managed to compile up the blinky demo and flash it down.
Compiling up the blinky demo was as easy as:
8510w@8510w-PC /cygdrive/c/Espressif/examples/blinky
$ make
CC user/user_main.c
AR build/app_app.a
LD build/app.out
Run objcopy, please wait...
objcopy done
Run gen_appbin.exe
No boot needed.
Generate eagle.flash.bin and eagle.irom0text.bin successully in folder firmware.
eagle.flash.bin-------->0x00000
eagle.irom0text.bin---->0x40000
Done
$
The flashing should then have been this easy:
$ make flash
c:/Espressif/utils/esptool.exe -p COM2 -b 256000 write_flash 0x00000 firmware/eagle.flash.bin 0x40000 firmware/eagle.irom0text.bin
Entering bootloader...
Connecting...
Traceback (most recent call last):
File "esptool.py", line 559, in <module>
File "esptool.py", line 154, in connect
Exception: Failed to connect
Makefile:293: recipe for target 'flash' failed
make: *** [flash] Error 255
But as you can see it failed.
I hade to use the NodeMCU flasher.
Then my board blinked away the LED on GPIO2.
Time to get down and dirty with C on this sucker.
Flashing does work using the makefile.
a) type "make flash"
b) power cycle the unit.
Then magic happens!
$ make flash
c:/Espressif/utils/esptool.exe -p COM2 -b 256000 write_flash 0x00000 firmware/eagle.flash.bin 0x40000 firmware/eagle.irom0text.bin
Entering bootloader...
Connecting...
Erasing flash...
head: 8 ;total: 8
erase size : 16384
Writing at 0x00007000... (100 %)
Erasing flash...
head: 16 ;total: 36
erase size : 81920
Writing at 0x00063000... (100 %)
Leaving...
MQTT xAP Heartbeat in pure C.
Demo run - debug output goes to COM2
WiFi connecting...
ip:192.168.1.112,mask:255.255.255.0,gw:192.168.1.1
WiFi connected
MQTT_InitConnection
MQTT_InitClient
TCP: Connect to ip 192.168.1.8:1883
XAP heartbeart
MQTT: queuing publish, length: 120, queue size(0/2048)
MQTT: Connected to broker 192.168.1.8:1883
MQTT: Sending, type: 1, id: 0000
TCP: Sent
TCP: data received 4 bytes
Onwards !!
Attachment | Size |
---|---|
heartbeat.zip | 28.41 KB |
Good work Brett. A proper publish queue too!
Looks promising this time. Just don't know when I'll next get some time to play. It's always the way isn't it.
Garry
This example is more fleshed out. It allows you to control GPIO 2 on the ESP8266 as a xAP BSC endpoint. Once flashed and running I could send this:
xap-header
{
uid=FF00AB00
source=dbzoo.eap.test
target=dbzoo.eap.client
hop=1
class=xAPBSC.cmd
v=12
}
output.state.1
{
id=*
state=toggle
}
and the GPIO2 led on the ESP board would toggle on and off.
The MQTT implementation in C has no problem with sending multiple packets back to back. There is no need for the semaphore control logic that was needed in MQTT LUA interface.
Here you can see it using MQTT queue buffer space to hold BOTH messages!
XAP heartbeart
MQTT: queuing publish, length: 120, queue size(0/2048)
MQTT: queuing publish, length: 133, queue size(123/2048)
MQTT: Sending, type: 3, id: 0000
TCP: Sent
Also there there is plenty of heap with 40k of RAM now available to me - no crash in site, just rock solid operation.
If you want to customize it see bscGpio\include\user_config.h
You'll need the cross compiler environment up and running.
Attachment | Size |
---|---|
bscGpio.zip | 30.67 KB |
Works a treat! Cheers Brett.
I used the eclipse IDE to flash and must say its a pain in the ass. You seem to talk about command line flashing? What are you using to do the make and make flash?
Thanks again
Garry
Gary,
I'm running cygwin as my shell and I just type "make".
I did setup the Eclipse IDE but I found it easier with my favourite editor and the command line.
In Cygwin for packages to install make sure you have "Devel->make" selected that should drag in other dependencies and get you going.
Glad you got it running, you would have had to edit a few things so your obviously finding your away around my rough code. I'm not being too precious about what I write just slapping shit together to prove this is a viable path at the moment.
Brett
Cheers, I'll give Cygwin a go.
wasn't many changes required. SSID stuff and mqtt server ip. Had to move include files around but think was eclipse related
anyhow, Been stable for hours now. I've even faked disconnects etc and all good.
Looks promising.
Garry
This demo shows how to expose multiple GPIO pins as xAPBSC output endpoints.
Screenshot showing the chip exposing the endpoints to xFXViewer below.
The MQTT publishing buffer queue is working hard now as we push lots of messages out the door very close to one another. Still lots of room thou.
XAP heartbeart
MQTT: queuing publish, length: 120, queue size(0/2048)
Publishing topic dbzoo/eap/client/gpio/15/xap
MQTT: queuing publish, length: 149, queue size(123/2048)
Publishing topic dbzoo/eap/client/gpio/14/xap
MQTT: queuing publish, length: 149, queue size(276/2048)
Publishing topic dbzoo/eap/client/gpio/13/xap
MQTT: queuing publish, length: 149, queue size(429/2048)
Publishing topic dbzoo/eap/client/gpio/12/xap
MQTT: queuing publish, length: 149, queue size(582/2048)
Publishing topic dbzoo/eap/client/gpio/5/xap
MQTT: queuing publish, length: 147, queue size(735/2048)
Publishing topic dbzoo/eap/client/gpio/4/xap
MQTT: queuing publish, length: 147, queue size(886/2048)
Publishing topic dbzoo/eap/client/gpio/2/xap
MQTT: queuing publish, length: 147, queue size(1037/2048)
Publishing topic dbzoo/eap/client/gpio/0/xap
MQTT: queuing publish, length: 147, queue size(1188/2048)
MQTT: Connected to broker 192.168.1.8:1883
On my ESP board GPIO pin 12,13,15 - control legs of an RGB LED.
12 - Green
13 - Blue
15 - Red
Toggling those on/off I can create a different colour LED, awe pretty :)
Sample packet to control each GPIO pin.
xap-header
{
uid=FF00AC00
source=dbzoo.eap.demo
target=dbzoo.eap.client:gpio.13
hop=1
class=xAPBSC.cmd
v=12
}
output.state.1
{
id=*
state=toggle
}
This demo is probably the most useful as it gives you control of 8 pins all via xAP.
The MQTT/xAP lua plugboard applet that I wrote is holding up well with no changes necessary from what I checked in to get this all going.
Brett
Update: Included an "ip=x.x.x.x" key/value pair into the heartbeat so you can tell what IP address the ESP chip has acquired.
Attachment | Size |
---|---|
bscMultiGpio.zip | 31.87 KB |
HI all,
I have now received my little! oled display and have modded Bretts gpio demo to display the state on screen rather than control a gpio port.
anyhow works a treat. Im thinking rather than using MQTT it would be best for this unit to poll the webserver applet and display relevent display text responses.
Ill have a crack at this.
Garry
Attachment | Size |
---|---|
OLED state display.zip | 635.1 KB |
image.jpg | 37.53 KB |
Cool I'll take a look when time permits I was thinking of using the xAP-OSD.Display schema so its a generic display head for whatever you want to send it.
Brett
The osd schema is a very good shout!
the display can then be modded by a lua script rather than reflashing
change on plan then!
cheers
garry
Is this relevant: http://rayshobby.net/introducing-esptoy-1-2-with-lua-firmware/ ?
Hi Andrew,
Good to see you back.
Yes, this module has lots of potential. I've tried a couple and they work well. The current drawn is a bit much for battery operation. That's where the RFM12Bs win.
The ethRelay that you mention was fun to design & the firmware that Brett rolled for this makes it a true zero config 'plug in and go' module for xAP users. Sadly, I don't reckon that a huge number of folks use xAP these days - most of the PCBs are in a box in my basement.
Anyway, it seemed worthwhile to send an ESP8266 to the other side of the world. Brett received the package today.
Cheers,
Derek.