Twitter Service Stopped working

21 replies [Last post]
danward79
Offline
Joined: 19 Jan 2011

Hi,

This post is more for info, as I have been unable to recreate this.

I was playing around with the twitter feature from my phone. Using ubertwitter to post commands to the relays. I did not get any response. I had got responses in the morning but not in the afternoon. 

By response I mean deleting of the command tweet and a tweet from the box, which says "Relay x is off".

When I eventually got home tonight, using the xfx viewer there was no twitter present, in the item list and no heart beat.

I have spent the last while posting various messages and no luck.

If it does happen again, how can I get at information that would be of use to you? Are there any logs etc?

Cheers

 

Dan

brett
Offline
Providence, United States
Joined: 9 Jan 2010
Twitter died

If the twitter process isn't showing up in the xfxviewr then it sounds like it has crashed.  Unfortunately due to space limitiations on the HAH all core dumps are prevented and even if they where enabled all the binaries have their symbolic debug tables stripped (due to limited flash space).   So there is not much you can provide except keeping you eye on it and seeing if its reproducable.   I haven't had any other crash reports.

Brett

danward79
Offline
Joined: 19 Jan 2011
Ok thanks Brett. If it

Ok thanks Brett. If it happens again I might be able to set up something via the Plugboard scripting.

danward79
Offline
Joined: 19 Jan 2011
It Happened Again

Hi Brett,

It happened again yesterday while I was at work. During this time the service was not used at all.

So I scheduled a auto reboot at 0100 hrs everyday, hoping that would be a work around. It rebooted this morning at 0100 hrs and when I got up at 0545 and checked it, it was not responding to twitter.

I rebooted the box at 0605 and it has been ok till now 1100 hrs.

No one else has posted suggesting it is an issue. I wonder if they are using twitter much and if they have any reboots etc scheduled.

In terms of capturing this, like you say it is difficult.

I was wondering about setting yo a timer script which is reset by the heartbeat, logging this on google calendar when it is not detected for more that the time out.

Is there anyway I could setup a script to record a text file to a usb stick?

Do you have any ideas?

Cheers

 

Dan

danward79
Offline
Joined: 19 Jan 2011
Debug log

Hi, I setup Putty to log to a text file, with xap-twitter -d 7 -i br0 Right at the end of the file, you can see where the service crashes and terminates. This took all afternoon to catch. It did crash three times today in total. Link is here. https://docs.google.com/leaf?id=0B3lq5J_TSkjFOGE4Yjk3ZjctZmNjZC00MTUzLTl... Hope this is of use? Cheers Dan

brett
Offline
Providence, United States
Joined: 9 Jan 2010
That log was very useful

The logs where uber useful I have a fix for this problem (I think).  The easiest way to test this is for me just to push a new build and see what happens rather then waiting for twitter to time out its API again, which could take a while.  I'll leave my system running twitter with debug on as well.

Unfortunately I've been working on some other changes in the mainline trunk that I have yet to regression test using the older M8 firmware.  So if I push this it (might) break everybody's HAH system.  I'm using the yet to be released 328 chip and different AVR firmware you see.  *sigh* the delimer.

Tell you what I'll push this as a BETA and you can test this out and make sure my other changes are ok too.

/etc_ro_fs/update-dev hah-beta.dbzoo.com

Brett

danward79
Offline
Joined: 19 Jan 2011
Hi,Excellent. I'll happily

Hi,

Excellent. I'll happily test a beta version for you, my system is not doing anything useful at the moment. So there is zero risk.

I have updated and will keep my eyes open. Thanks again.

 

Dan

danward79
Offline
Joined: 19 Jan 2011
Hi Brett,Been running this

Hi Brett,

Been running this all day so far I have not seen a crash. I have removed the scheduled reboot for tonight to give it a chance.

Looking Good!

Thanks

Dan

brett
Offline
Providence, United States
Joined: 9 Jan 2010
good stuff

Well let it run over the weekend before declaring this a success if you don't notice any other defects then I suspect the other changes I did are compatible with your AVR firmware.  This is good, they should have been.  Once its in the clear I'll push a new build for all.

Brett

danward79
Offline
Joined: 19 Jan 2011
It died!

Hi,

I hate to say it but it died again today. I have not managed to log it yet thou.

I'll try and log it tonight / tomorrow.

Dan

brett
Offline
Providence, United States
Joined: 9 Jan 2010
Mine died too

I'm going to have to run the twitter in GDB overnight and try and catch the error.  It might take me a little to figure out now. *sigh*

danward79
Offline
Joined: 19 Jan 2011
On a positive not at least

On a positive not at least you are seeing it too... ;-)

Let me know if you want me to capture the logs too.

 

Thanks

Dan

derek
Offline
Glasgow, United Kingdom
Joined: 26 Oct 2009
3days 5hrs ...

... since my last reboot. xap-twitter still running. No tweets handled, but it's there.

Derek. 

danward79
Offline
Joined: 19 Jan 2011
Hi Brett,Did you manage to

Hi Brett,

Did you manage to find the problem?

Dan

brett
Offline
Providence, United States
Joined: 9 Jan 2010
not yet

I ran it in GDB and the program is not crashing per se its actually doing a clean exit.  Which is worse a crash is easier to find.

I think there is something funky happening in the Curl library I even ran an strace but it didn't tell me anything I didn't already know.... so now I'm going to have to mock up an apache server that generates a 503 return code and run this snippet locally stepping through the code.  As you can tell this is going to take time - rather damn annoying actually.

Brett

brett
Offline
Providence, United States
Joined: 9 Jan 2010
new Beta to try out

I'm pretty sure I've nail this problem update to beta 279.3 let it run for a few days and let me know how you get on.

PS: This build has the new LUA engine so don't just upgrade from the mainline build unless you are prepared to rewrite any LUA scripts you have.


danward79
Offline
Joined: 19 Jan 2011
Hi Brett,I've just uploaded

Hi Brett,

I've just uploaded it to the box, all my scripts are converted to the new lua stuff anyhow.

I'll let you know how I get on.

 

Dan

danward79
Offline
Joined: 19 Jan 2011
Hi Brett,It has lasted till

Hi Brett,

It has lasted till today from the 17th. So a vast improvement, unfortunately just not a total fix.

Dan

brett
Offline
Providence, United States
Joined: 9 Jan 2010
Maybe this is something different

I'll run my twitter in debug mode and see if something turns up - can you run yours in debug mode too?

One of us might catch something useful.  I know i fixed the 503 return code issue so not sure what this might be.

danward79
Offline
Joined: 19 Jan 2011
Hi, Just to let you know I am

Hi, Just to let you know I am still trying to capture a crash.

I did have it crash after two days of logging but the data was lost by swmbo.... I have since not been able to get a crash! The log fine is over 140Mb.

Have you spotted anymore your end?

Dan

brett
Offline
Providence, United States
Joined: 9 Jan 2010
I had a server outage

I ran my capture for a while but I did not see any issue.   I had a power outage and my server rebooted since then I haven't bothered to set it back up again.  Having said that I've never seen twitter down since my fix.

danward79
Offline
Joined: 19 Jan 2011
I must admit I am on the

I must admit I am on the verge of giving up on the logging for now. I might right a script to restart it if it crashes.

Hardware Info