Up and coming changes

142 replies [Last post]

26 December, 2013 - 22:39

brett

Offline

Providence, United States

Joined: 9 Jan 2010

I've been working on some changes over the xmas break for the next HAH firmware release.
One of the items is the refactoring of the /etc/xap-livebox.ini file into separate files which will be contained in the new directory /etc/xap.d/

I've completed my first pass at modifying all the code, and there was a lot, and my livebox is up and running using this new location. Everything looks ok but I want to check that I caught everything before I release a beta. The upgrade will be complete automated and there will be nothing for you to do except verify that the migration process worked and then do some remedial delete's which I wanted to leave as manual processes.

I'm not sure how this will address the corruptions have been occuring but is has to help somewhat having many files instead of the one single file.

The other change, which is more subtle, is that the "hostname" of the unit will be used to identify the deviceID.

Recall the source/target strings are formed like this: VendorID.DeviceID.Instance

By default the DeviceID is hardcoded to "livebox" this makes moving the software to other platforms, or indeed standing up another livebox, difficult as you get namespace collisions. To resolve this I've done a couple of things.

The DeviceID defaults to being the "hostname", which by default for your livebox is "livebox" unless you have changed it. It is also possible to override this by using a configuration setting which is also exposed on the web gui.

/etc/xap.d/system.ini
[xap]
instance='deviceid'

This has always been in the system but its interpretation has been slightly adjusted. Previously this "instance" setting would do this to your target/source.

dbzoo.livebox.2.Controller - where the number 2 is an injected value from the instance= setting. This is not strictly xAP compliant so I have removed this interpretation.

If you have multiple liveboxes and have used this encoding all is not lost as after the upgrade instead of having this

instance=2

You would need to change it (below) to keep this 'old' interpretation and reduce the impact of the upgrade.

instance=livebox.2

Until such time as you can correct your addressing and set a new hostname for your livebox.

All these changes are quite under the covers and from a usage point of view there will not be any impact, however what it does do is set the system up for being able to be easily migrated and compiled to different platforms as such the PI or Linux without having "livebox" encoded anywhere. Its all part of a larger push I want to do in 2014.

Brett

27 December, 2013 - 10:25

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Beta 311.1 posted - this

Beta 311.1 posted - this contains quite a number of changes.

I've updated the wiki to already reflect the new API: plugboard and jeenode wiki pages.

After you update to this beta you can delete /etc/xap-livebox.ini and /etc/ini.conf as these are not required - do make sure that /etc/xap.d was populated with files before you do this. You might want to just rename those out of the way into say /root for safe keeping.

If the automatic upgrade fails, that is there are no files in /etc/xap.d you can run from the command line.
# ini-migrate
and it will tell you what the issue is. All is not lost if you get to this point as there is provision to use an external mapping configuration file if there are sections that it can't figure out how to convert. I'll then include those into the code base before this becomes the next production release.

A quick summary of some of the LUA framework changes that might affect you.

Some changes by example
xap.init("dbzoo.livebox.hithere","FFDEAD00")
This now becomes
xap.init{instance="hithere", uid="FFDEAD00"}

You don't need to, and its best you don't, specify the VENDOR or DEVICE components as part of the initialization. These will be figure out automatically based on some defaults.

Creating BSC endpoints
bsc.Endpoint{base="dbzoo.livebox.my:relay.3", ...}
This now becomes
bsc.Endpoint{instance="my:relay.3", ...}

Again to maintain vendor, device portability you don't specify these parts. The wiki write-up for the BSC API has more examples.

All the examples in /etc_ro_fs/plugboad have been amended to fit into the new way of doing business.

NOTE: Full backward compatibility for previous API calls has been maintained so nothing should break on upgrading. *should* but you never know.

This time when I say BETA I really mean it due to the amount of code that has been reworked.

Brett

27 December, 2013 - 13:55

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Brett,Thanks for your work on

Brett,

Thanks for your work on this, especially over Xmas.

It looks like a substantial update that will enable the project "move on" in the future. Whilst I personally am in no rush to ditch the livebox and onboard rf/input/1wire, the jeenodes (which have been a new addition to my system this year) allowing remote relay control of my heating without wires and ir tx/rx along with the more recent bluetooth stuff has highlighted to me the benefit of "non central" hardware. As always you seem to be one step ahead allowing the rest of us to simply use the things you code for us!

The time both you and Derek put into this project is much appreciated. I hope you both had a good christmas and wishing you all the best for the new year. Let's see what 2014 brings for the HAH :)

Garry

27 December, 2013 - 16:03

allanayr

Offline

Ayr, United Kingdom

Joined: 25 Sep 2011

Hear Hear!

30 December, 2013 - 00:35

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

Always moving forwards Brett

Your work is very much appreciated. I look forward to trying this beta and I know your plans for 2014 will be good for us all.

30 December, 2013 - 01:20

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Thanks Mark. I've also been

Thanks Mark. I've also been working on porting the HAH specific parts to the ARM architecture targetting the BeagleBone (armv7), the RaspPI is (armv6). I figure if I can do one I can do the other without too many drama's.
The plan is to bundle this up as a opkg/ipkg that you can install on a standard distro.

I've comitted some changes that allow all the xAP and LUA(+ addons) to be built and now I'm just waiting on my BeagleBone from China so I can test out my build. I pick one up one cheap $30 US from seedstudio as they where having some sort of special/runout sale of these. Now they have run out ! They do have the Blacks for $45 thou.

I also have a RaspPI but I always had issues with the ethernet port, ie it would not work, and I burnt way to much time fecking around with it, and ended up putting it back in its box where its sat for a year. I hate wasting time. Anyway the idea of having to attach a monitor and keyboard seems pointless for embedded hardware that sit in a cupboard I might give it another go later...

Brett

30 December, 2013 - 20:47

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

BeagleBone stock

The BeagleBone looks to be out of stock pretty much everywhere. Looks like most suppliers have them on backorder for February so you have some time to play :)

I have a few alerts set up for when they restock.

3 January, 2014 - 18:30

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

The BeagleBone has

The BeagleBone has landed.

Time to play :)

Open for alpha testing too...

3 January, 2014 - 22:51

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

Beta update

Brett

I have tried the beta. Check your email. All up and running here again.

4 January, 2014 - 06:20

brett

Offline

Providence, United States

Joined: 9 Jan 2010

I'm jealous you got your bone before I did

Mark I'm jealous you got your bone before I did. I'll push up the ARM binaries for the HAH system and post back here with the location when that' done.

The 311.2 beta dropped the all important ini-migrate program due to me missing the dependency to build it in the Makefile.
I've fixed this and posted a new 311.2 beta (yes I didn't bump the version).

NOTE: the directory /etc/xap.d must be empty before you attempt to upgrade again.
CORRECTION: the /etc/xap.d directory must not be present.

Brett

4 January, 2014 - 08:58

#10

brett

Offline

Providence, United States

Joined: 9 Jan 2010

ARM binaries

For what its worth here are the HAH binaries compiled for ARM. Once I get my bone I will package these up as a proper opkg file to make installation easier etc. etc..

http://www.dbzoo.com/hah-arm/bb.tar.bz2

They are dynamically linked and I've included the libraries which you will need to check pick to get them running. You'll at least need libxap2.so as for the others you can use "ldd" on the executables to see if they are required.

As you can see the xap-* binaries for the most part only need the libxap2.so, the other linkables should already in your bone distro. (I'm using a cross compiled ldd if you're doing this natively on your bone just ldd will suffice and -root ../.. isn't required either.)

brett@bb-dev bin]$ /opt/bb/toolchain/bin/arm-linux-ldd --root ../.. xap-hub
        libxap2.so => /lib/libxap2.so (0xdeadbeef)
        libc.so.6 => /lib/libc.so.6 (0x8badf00d)
        ld-linux.so.3 => /lib/ld-linux.so.3 (0x8badf00d)

For LUA I linked in the optional libraries, readline, history, ncurses to make using it from the command line a little easier so these libraries are required.

[brett@bb-dev bin]$ /opt/bb/toolchain/bin/arm-linux-ldd --root ../.. lua
        libm.so.6 => /lib/libm.so.6 (0x8badf00d)
        ld-linux.so.3 => /lib/ld-linux.so.3 (0x8badf00d)
        libc.so.6 => /lib/libc.so.6 (0x8badf00d)
        libdl.so.2 => /lib/libdl.so.2 (0x8badf00d)
        libreadline.so.6 => /lib/libreadline.so.6 (0xdeadbeef)
        libhistory.so.6 => /lib/libhistory.so.6 (0xdeadbeef)
        libncurses.so.5 => /lib/libncurses.so.5 (0xdeadbeef)

Anway lets see how this goes - btw these all use the new /etc/xap.d location for their configuration files.

Brett

5 January, 2014 - 03:00

#11

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

I'll try the beta again...As

I'll try the beta again...

As for the BeagleBone...just give me a little while to get to grips with it then I'll see how the distro goes

cheers

Mark

8 January, 2014 - 12:02

#12

kevin

Offline

Huddersfield, United Kingdom

Joined: 17 May 2010

Raspberry Pi HAH

From an installed base point of view - and for getting maximum exposure for HAH the Raspberry Pi is obviously the important platform so if anyone has any progress reports on that then do post. I'm hoping Bretts Pi might leap into life or that he might have access to another board.

8 January, 2014 - 17:20

#13

aivo

Offline

Tallinn, Estonia

Joined: 2 Mar 2011

Some Pi experience here

Hi, I have some experience with HAH on Pi for about a year, see if this is type of info you are looking for .. :)

Got into it because was unable to keep Livebox running stable with all wanted applets on it, occasionally found it hanging despite efforts.

I have only complied xaplib, hub, serial + ini and plugboard as this was my main interest. Lua/pl are via Raspbian.
First binaries were from revision 436 source, later "upgraded" to 466 and lately to 519 just of interest, do not recall stability or other issues.

For now then running xap-hub, xap-serial and applets/plugboard on "hahpi". To hahpi is also connected Jeelink/basenode, CurrentCost++ still "normally" to Livebox/HAH.

Regards,
Aivo

8 January, 2014 - 22:10

#14

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Aivo,That's good to here.

Aivo,

That's good to here. The changes that I made circa r519 try to separate the mega xap-livebox.ini into separate files should tidy this porting excerise up too. There is another change in r526 which addresses a bug in xaplib in this regard.

Part of the motivation for this work was to remove the word "livebox" as a default in the configuration path, or the files themselves. This makes the system more host agnostic and friendly for porting. Separate files in /etc/xap.d/ is a better in this regard than one /etc/xap-livebox.ini file.

Yes I should look more into my PI but it did my head in last time.

Brett

11 January, 2014 - 20:33

#15

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Rasp pi info

Thanks to aivo's news on the Pi, I have had a play with this myself over the last day or so.

I have never done any compiling before so it's real trial by fire but so far, as Aivo has already, I have the main HAH components running nicely on the Pi. (hub, serial, twitter, plugboard)

For interest, I installed subversion and did the source checkout directly to the pi.

I then compiled each component separately using their individual make files, modding them to use the pi complier (gcc) not a cross compiler. Few hiccups along the way but pleased with progress so far.

once I know a little more of what I'm doing I'll write up and maybe try to modify the overall makefile to make this simpler. I see that Brett has already got one for the beagle bone that would be a great starting point. Although it's still beyond me at the moment.

post more info/instructions soon!

Garry

(did I say in another post a few weeks back I was in no rush to move from the livebox? Funny what the temptation to try something new can do! :)

12 January, 2014 - 02:58

#16

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

It's hard work resisting

LOL Garry

I know just what you mean. I've got a beaglebone and I have tried to get the code on it but stumbling along so far.

I'm almost tempted to send my BB to Brett to hurry things along :)

No rush Brett, honestly, I really do need a while to play with this board first :)

12 January, 2014 - 05:03

#17

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Let me help you compile everything

Gary

Let me help you compile everything.
I've made some changes to the Makefile.bone file which I've been using to cross compile for ARM so it will build natively using a couple of modifications. Revert all the changes you have made and try this:

# vi Makefile.bone

Comment out these 3 lines for native building.
HOST=arm-linux
XHOST=--host=$(HOST)
CROSS_COMPILE=$(TOOLCHAIN)/bin/$(HOST)-

# make -f Makefile.bone

I verified with these changes I can build on x86_64 Linux without any problems.
When its done everything will be layed out for you in -> targets/bb.src

Mark I'm still eagerly waiting on my bits and pieces.

Brett

13 January, 2014 - 10:21

#18

GaryP

Offline

Joined: 9 Jan 2014

Raspberry Pi version

A port of HAH to RPi would be very useful.

I am just starting on the home monitoring journey and have got my feet wet using emonCMS from openenergymonitor.org.

It looks like oem and HAH might be getting closer in your goals as oem are looking at linking their monitoring software into home control which HAH already seems to be doing. The HUH work on grabbing data from lots of different products would have a lot to offer to this.

One thing to watch out for on RPi and other SD card based boards is failures. Oem are now advising a hybrid SD card boot and HDD main drive as SD cards seems to be flaking out after 3 months or so of writing.

The emonCMS route might also offer HAH some cloud connectivity as well as they are currently doing free hosting of your emonCMS data as well as you being able to have a local install of this.

Looking forward to seeing where the port gets to

Gary P

13 January, 2014 - 18:48

#19

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Thanks Brett, I did try to

Thanks Brett, I did try to have a hash at the makefile for the bone myself but got errors on iserver and curl.

I have backed out my alterations and gave your modified makefile a go (after commenting out the 3 lines) but still get the same errors.

Curl error is:

/usr/bin/install -c -m 644 './curl-config.1' '/usr/trunk/targets/bb.src/share/m an/man1/curl-config.1'

make[8]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7/docs'

make[7]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7/docs'

make[6]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7/docs'

make[5]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make[4]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make[3]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make[2]: *** [install-recursive] Error 1

make[2]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make[1]: *** [install-strip] Error 2

make[1]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make: *** [libcurl] Error 2

I have managed to install curl ok via the apt-get install commend though.

Iserver error is:

make[1]: Entering directory `/usr/trunk/userapps/hah/iServer'

lex -t tokenize.l > tokenize.c

/bin/sh: 1: lex: not found

make[1]: *** [tokenize.c] Error 127

make[1]: Leaving directory `/usr/trunk/userapps/hah/iServer'

make: *** [iServer] Error 2

I am yet to get around this one.

On the plus side, I now have xively, currentcost, twitter, hub, serial (with jeenodes) and plugboard up and running.

I plan to put the HAH hardware onto the GPIO port tonight and attempt the xap-livebox install.

Fingers crossed!!!

Garry

13 January, 2014 - 22:12

#20

brett

Offline

Providence, United States

Joined: 9 Jan 2010

I know that curl error

I know that curl error you are getting as I saw that too. Its because openssl (compiled earlier) is not using -fPIC which means its not relocatable so when this static library is linked with libcurl as a shared library it throws a wobble.

I did fix that when I made my changes and I too ran into the same problem.
Did you "svn revert --depth=infinity" from the top level to get ALL my changes?

Did you also "make clean" as you need to recompile EVERYTHING from scratch any library compiled without -fPIC will cause you problems. I think you did not do this step.

lex is also known as flex. Try installing this package its a std dev tool. "yum install flex" or whatever equiv for PI is.

Brett

15 January, 2014 - 19:55

#21

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Thanks Brett,as you thought,

Thanks Brett,

as you thought, I did not make clean!. However, I have now and unfortunately the curl error persists.

I might get a virgin SD card imaged and try again in case it's something I've done with my playing around.

The flex did fix iserver issue though, cheers. I have now officially powered off my livebox and the Pi is in control!. It better behave itself, the wife is very sceptical?!

Garry.

15 January, 2014 - 21:58

#22

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Gary,I suspect CURL is not

Gary,

I suspect CURL is not the problem you need to scroll up from the make output and look for another error.
The "install-recursive" error is simply saying that no libcurl.so is found for installation, which means it didn't build.

Check the GCC output for openssl do you see -fPIC ?
See what other errors exist above this one you posted as its more a symptom error of something else failing.

libcurl is a dependency for building xap-googlecal (broken) and xap-twitter. So all that is missing for you is twitter.

What you could do is simply remove xap-googlecal and xap-twitter from this list in the Makefile.bone file

HAH_APPS = xaplib2 xap-hub xap-livebox xap-snoop xap-xively xap-sms iServer \
xap-currentcost xap-googlecal xap-twitter xap-serial xap-plugboard \
xap-flash xap-urfrx

That should give you a "clean" build avoiding libcurl - at then you are not messing around with individual builds.

Good that flex sorted out the iServer. I'd still like to get a build without messing around with individual compiles.

Brett

17 January, 2014 - 17:16

#23

BoxingOrange

Offline

United Kingdom

Joined: 11 Jun 2010

Libcurl

running "apt-get libcurl4-openssl-dev" installs a curl library that allows xap-twitter to be compiled, but googlecal still seems to get an error :-

rm -f -rf /home/pi/livebox-hah-read-only/targets/bb.src/share

make -C /home/pi/livebox-hah-read-only/userapps/opensource/libgcal install-strip

make[1]: Entering directory `/home/pi/livebox-hah-read-only/userapps/opensource/libgcal-0.9.6'

make[1]: *** No rule to make target `install-strip'. Stop.

make[1]: Leaving directory `/home/pi/livebox-hah-read-only/userapps/opensource/libgcal-0.9.6'

make: *** [libgcal] Error 2

Once compiled an appropriate directory structure seems to need creating to get the "xap-?" to run.

./xap-livebox: error while loading shared libraries: libxap2.so: cannot open shared object file: No such file or directory

pi@raspberrypi ~/livebox-hah-read-only/targets/bb.src/usr/bin $ ./xap-sms

./xap-sms: error while loading shared libraries: libxap2.so: cannot open shared object file: No such file or directory

Once I've got these little issues ironed out I've got a full script of how to setup a RasPi, compile and run the HAH on it. Awesome work Brett, perhaps Google will take a bit of notice then ;)

17 January, 2014 - 21:30

#24

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Hi,try moving the .so file to

Hi,

try moving the .so file to /usr/lib and the .h files to /usr/include.

this worked for me and it's been running great for a few days now.

still struggling to get a complete compile though as Brett would like, I am gonna try again later and push the errors into a file for the master to look at ;)

Garry

23 January, 2014 - 22:51

#25

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

Toolchain

Brett do you have available the toolchain for the armV7 build?

/opt/bb/toolchain/bin/arm-linux-gcc

23 January, 2014 - 23:47

#26

brett

Offline

Providence, United States

Joined: 9 Jan 2010

BB rough notes

Mark,

I posted up some of the rough notes I wrote last night whilst I was playing around with my BB.

http://www.dbzoo.com/livebox/beagleboneporting

I didn't write anything up when I built my toolchain so I'll have to go back and rebuild it and make some notes.
Its a little large to upload (130Mb) so it would be best if you build your own.

I believe I just followed this: http://code.google.com/p/my-beagleboard/wiki/Toolchain

I've attached the config.txt (aka .config) which lives in my /opt/cross_arm/bin directory used to build the toolchain these are my exact settings.

Brett

Attachment	Size
config.txt	10.77 KB

23 January, 2014 - 23:49

#27

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

cheers Brett

I'll take a look

24 January, 2014 - 10:30

#28

brett

Offline

Providence, United States

Joined: 9 Jan 2010

beaglebone .ipkg installation of the HAH system

Mark,

This is the HAH system compiled up for the beaglebone as an .ipkg installation.
These are armv7 binaries and they will not work on the Raspberry pi which is armv6.
Once I get my bone running happy I'll do a armv6 cross compile.

Create you own configuration file:

root@beaglebone:/etc/opkg# cat /etc/opkg/hah.conf
src/gz hah http://homeautomationhub.com/feeds/ipkg/eglibc/armv7a/hah

(Opps typo on that hah.conf line - I've swapped the ipkg/eglibc fields and adjusted the path on the web server too)

Refresh your package caches and then install the HAH system, its that easy.

root@beaglebone:~# opkg update
root@beaglebone:~# opkg install hah

By default it only starts: xap-hub and kloned.

The kloned is listening on port 8080 instead of 80 which is where the beagle bone http is sitting and I didn't want to disturb it. On the web gui you'll see it can find its build number as its looking /etc_ro_fs/build which of course is not there - I'll probably fallback to looking /etc/xap.d/build once I see what else is broken.

The filesystem is layed out exactly like the livebox so things will be where you expect them.

I should add I'm running the Angstrom linux distribution on my bone.

# head -1 /etc/angstrom-version
Angstrom v2012.05 (Core edition)

Brett

24 January, 2014 - 18:22

#29

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Looks like solid progress on

Looks like solid progress on the beaglebone!, good news, can only be good for the future of the HAH project.

As for the pi, I have attempted to compile as one again using the makefile.bone with the three comments for local compiling. I redirected stderr and stdout to a log for investigation.

Brett, you were right, there are earlier errors, I cannot fathom what they are trying to say though :( any ideas?

as for running the HAH system (compiled seperately) on the pi though, its been a week now and all good except for xively. This needs to be restarted a number of times before it becomes stable. once running though it will stay running for days.

when it goes down it complains of "glibc detected" realloc() invalid next size. obviously relating to the memory reallocation in he xively code but I cannot find any indication as to what part of the code its unhappy about.

Is this likely to be due to my compiling? any thoughts?

like I say its intermittent and when xively is working, it works great for days?

Garry

Attachment	Size
log.txt	309.33 KB

25 January, 2014 - 20:04

#30

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

Angstrom here too

I'm running Angstrom here too as I have no wish to replace that at the moment.

Angstrom v2012.12 (Core edition)

Running Ubuntu on the Mac in a VM for the cross compiler, so I NEED to get that running.

The package install ran well. I now have the web gui up on the BB and on my Mac :)

Attachment	Size
Screen Shot 2014-01-25 at 01.50.41.png	108.03 KB

26 January, 2014 - 01:54

#31

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

I've had an issue where the

I've had an issue where the ip address, subnet mask etc reversed in the gui. They were also reversed in the system.ini file when I viewed that in the terminal. However the ini files downloaded using the backup option were the right way.

Shortly after I noticed that the BB stopped.

I also noticed a file named 533745ni in the etc/xap.d folder

Now back up and running after reflashing the eMMC from a backup image.

Just got to redo the wifi drivers and reload HAH

27 January, 2014 - 21:51

#32

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Mark on the BB pretty much

Mark on the BB pretty much everything in system.ini is ignored. You cant adjust your IP from the web gui. Its a straight port from the livebox so there is LOTS of stuff that just won't work. The port is a proof of concept only. Having said that you should not get strange files being created like you did. Sounds like a memory corruption somewhere. I think when moving to another platform I'm going to drop kloned and rewrite it not in C.

Brett

27 January, 2014 - 21:54

#33

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Garry,I had a brief look but

Garry,

I had a brief look but nothing obvious comes to mind. Let me compile native on my BB and see if I can reproduce. I suspect I will be able to. You can drop the building of libcurl and libopenssl as the libraries will already be present. I will adjust the makefile for a local build to do this I might need to rethink how this makefile is put together to allow (livebox vs other platforms) including cross compile and native compile.

Brett

27 January, 2014 - 22:59

#34

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

HAH on BB update

Brett

The HAH on BB seems solid enough now. I took the BB back to a clean install and reloaded all from scratch. It's possible that the erroneous files got there during my meddling.

I've now got the WH node connected and getting xAP messages from that. I'm wondering what route to go down for the RF transmission. Is your plan to release the ethernet RF nodes?

I'm going to hook up the HAH baseboard to the BB and see what comes of that.

I was thinking for now to use a jeenode board connected to the serial monitor port on the BB and fit a 433Mhz tx to it, as I have no need for the relays (LCD and I2C would be nice though)

Cheers

Mark

27 January, 2014 - 23:33

#35

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Cheers for looking Brett. I'm

Cheers for looking Brett. I'm all up and running now so no rush for me. I am happy to test further makefiles, etc to get the porting neater for others to follow.

Mark,

I managed to get my hah board running nicely off the pi gpio ports, it's been great, I just needed a gpio breakout board to wire the hah board into.

It's worth doing if you have the hardware. Like you say, rf, 1wire and lcd are useful. I do also use inputs for alarm monitoring.

Think in the future though rf nodes will be the easier option

28 January, 2014 - 00:44

#36

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Mark: you wrote "I've now got

Mark: you wrote "I've now got the WH node connected and getting xAP messages from that. I'm wondering what route to go down for the RF transmission. Is your plan to release the ethernet RF nodes?"

To get the NODE RF transmission into xAP I use this http://www.dbzoo.com/livebox/hah_hahnode/nanode_gateway
It work really well, and this is what I use full time as I have no BaseNode connected to my livebox.
To get RF transmission out: http://livebox-hah.googlecode.com/svn/trunk/userapps/arduino/EthURF/
Which is a standalone URF transmitter that can be ethernet tethered. I wrote up the code but there is no wiki link on how this work. I think from the code is pretty self explanatory.

Garry: I am curious about xap-xively just dumping. Perhaps run the process manually and use valgrind to see what it reports. You might have to compile up xap-xively with -g to give valgrind something to churn on. I do recall using this previous with various program but I made have introduced a regression recently as I've not valgrind the code for a while now.

Brett

28 January, 2014 - 09:41

#37

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Beaglebone iServer compiling

I hit TWO nasty bugs trying to get this to compile up.

1. There is no "lex" link, so you need to create one. I will work around that in the Makefile but for now.
# ln -s /usr/bin/flex /usr/bin/lex

2. m4 is not installed by default so you need to install it as flex needs this.
# opkg install m4

3. There is a compiled in path in the flex binary that is WRONG !!!

I could not get flex to do its thing so I ran strace on the command (after installing strace that is).

 strace -f flex -o tokenize.c tokenize.l

What the f*** is that path doing in there?

[pid 16777] execve("/home/koen/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/m4", ["/home/koen/setup-scripts/build/t"..., "-P"], [/* 14 vars */]) = -1 ENOENT (No such file or directory)
[pid 16777] write(2, "flex: fatal internal error, exec"..., 40flex: fatal internal error, exec failed
) = 40

Something is seriously screwed up with that flex .ipkg. Workaround:

root@beaglebone:# mkdir -p /home/koen/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/
root@beaglebone:# ln -s /usr/bin/m4 /home/koen/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/m4

That clears the bug. Now flex does it thing. Onward and upward for native BB compiling.
Thought I'd share that little pisser with you.

Brett

28 January, 2014 - 12:06

#38

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Didn't have these issues on

Didn't have these issues on the pi compile. Had to install flex but no issues with links or m4.

28 January, 2014 - 19:49

#39

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Valgrind is now running

Valgrind is now running xively but as you might expect, I cannot get xively to break. How annoying!.

will leave it running overnight and if no joy will start doing a few reboots. Guaranteed it will break as soon as I take valgrind off.

keep you posted.

2 February, 2014 - 10:51

#40

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Typical!

I've had valgrind running xively as part of the init script for days now and no issues (plus numerous reboots!) It's almost as though running valgrind has stabilised it?! Is this even possible?

i will remove and see if it breaks again :(

problems I can deal with but pesky intermittent ones with no indication of the issue test my patience!

2 February, 2014 - 12:45

#41

mark_baldwin

Offline

Blackburn, United Kingdom

Joined: 19 May 2012

Sorry Garry but that made me

Sorry Garry but that made me chuckle :)

2 February, 2014 - 21:41

#42

brett

Offline

Providence, United States

Joined: 9 Jan 2010

From memory it only reports

From memory it only reports issues once you stop the process. So you have to stop it to see what it found.
That fact that nothing is crashing means that valgrind is preventing the problems so it should have something to say about it.
If you are stuggling I'll do the same and see what I can find. Nigglings bugs are the worse.

I found one very minor leak that happens in xively on startup but its only leaking 382 bytes so its not a showstopper.
I've checked in a fix for this none the less.

==26094==
==26094== HEAP SUMMARY:
==26094==     in use at exit: 10,372 bytes in 213 blocks
==26094==   total heap usage: 511 allocs, 298 frees, 143,320 bytes allocated
==26094==
==26094== 117 bytes in 9 blocks are definitely lost in loss record 44 of 55
==26094==    at 0x4A05E1C: malloc (vg_replace_malloc.c:195)
==26094==    by 0x38C9878C81: strdup (in /lib64/libc-2.5.so)
==26094==    by 0x401C77: getDynINI (main.c:143)
==26094==    by 0x401FA4: parseINI (main.c:190)
==26094==    by 0x402075: main (main.c:204)
==26094==
==26094== 265 bytes in 9 blocks are definitely lost in loss record 48 of 55
==26094==    at 0x4A05E1C: malloc (vg_replace_malloc.c:195)
==26094==    by 0x38C9878C81: strdup (in /lib64/libc-2.5.so)
==26094==    by 0x401C77: getDynINI (main.c:143)
==26094==    by 0x401F79: parseINI (main.c:189)
==26094==    by 0x402075: main (main.c:204)
==26094==
==26094== LEAK SUMMARY:
==26094==    definitely lost: 382 bytes in 18 blocks
==26094==    indirectly lost: 0 bytes in 0 blocks
==26094==      possibly lost: 0 bytes in 0 blocks
==26094==    still reachable: 9,990 bytes in 195 blocks
==26094==         suppressed: 0 bytes in 0 blocks
==26094== Reachable blocks (those to which a pointer was found) are not shown.
==26094== To see them, rerun with: --leak-check=full --show-reachable=yes
==26094==
==26094== For counts of detected and suppressed errors, rerun with: -v
==26094== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 4)

Brett

2 February, 2014 - 23:11

#43

martynw

Offline

Joined: 18 Feb 2012

I've been running various

I've been running various bits of your xap code for some time now, mainly on kirkwood devices (sheevaplug, pogoplug) and see the same xap-xively crashing as Gary above.

I compiled the code on an x64 debian box and ran it there to rule out anything platform specific but got the same issue. So I ran it under valgrind (with xap-xively compiled with the -g flag).

Does the attached shed any light on why it's crashing?

2 February, 2014 - 23:10

#44

martynw

Offline

Joined: 18 Feb 2012

Actually it crashes pretty

Actually it crashes pretty consistently for me, so if you want me to try any valgrind options etc let me know!

(attached another valgrind crash log)

2 February, 2014 - 22:53

#45

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Martin,Thanks those are

Martin,

Thanks those are useful... I might also run this on my ARM beaglebone and see if I can reproduce the crash myself.
From those valgrind logs I should be able to figure out what is going on - it points me in the right direction.

Thx.

2 February, 2014 - 23:12

#46

martynw

Offline

Joined: 18 Feb 2012

Ok cool, let me know if you

Ok cool, let me know if you need anything else - I removed the log files from my 2 posts as I realised I hadn't sanitised the API keys :-(

2 February, 2014 - 23:23

#47

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Thanks. I'm looking at the

Thanks. I'm looking at the code and now I know what to look for I see a glaring defect in my logic. I don't know what I was thinking at the time this is just so wrong... !
I'll work on fixing this. I have the logs in a browser tab so feel free to delete them.
Your key is safe with me.

UPDATE: I checked in a change to dscache.c that should fix the issue. If you want to compile this up and report back that'd be great. This should nail the issue.

Brett

3 February, 2014 - 08:36

#48

martynw

Offline

Joined: 18 Feb 2012

Working well so far, been

Working well so far, been running for about an hour now with no crashing - previously it only ran for a few minutes or so.

I'd say it's fixed :-)

Thanks,

3 February, 2014 - 10:23

#49

brett

Offline

Providence, United States

Joined: 9 Jan 2010

Great xively is working!

Great xively is working! I've pushed 311.10 for livebox users to get this fix.

Now I understand why I can see what will excaserbate the bug. As only 1k of memory was setup for the XML if you have a large number of xively feeds this would run out, the intention was to automatically expand this as requried, however the logic was flawed. For those with a small number of feeds that fitted withing 1k of XML generation you would never hit this bug but once you start to push this boundary memory overflow and heap corruption occured resulting in random crashes depending on what got clobbered. I'm suprised that bug has been dormant for so long actually.

As the fix correctly allocate the right amount of memory instead of defaulting to 1k of ram it only grabs 200 bytes and then expands as required. So all in all this fix is more memory friendly if you only have a low feed count. You get a few hundred bytes of memory back :) It all counts.

Anyway another one down. Thx to all for reporting and providing logs, feedback etc.

Brett

3 February, 2014 - 10:43

#50

garrydwilms

Offline

United Kingdom

Joined: 31 Mar 2011

Ha! Here's me waiting

Ha! Here's me waiting patiently for valgrind to tell me what's up!

When stopped, the logs did indeed appear, but looks like you've already nailed it!

Thanks all for the efforts, I do have a large number of feeds so prob why it affected me, only when I went to the pi though?!

Anyhow, cheers to all involved, I'll update tonight!

Garry

Site Navigator

Shop

Who's online

Up and coming changes