May 292012
 
June 1st, 2012

Since the weekend I’ve been trying to work out why the code to upload data to my Cosm feed was falling over after random periods of time. Sometimes it lasted for a few hours before dying, but others it lasted just minutes. There was no readily visible pattern, so it was time to get Googling and also accumulate a bit more data so I could get a better handle on where the problem was occurring. All I could see from the Cosm feed graph was that all of a sudden no further updates were being received and the graphs flatlined. I had nothing to tell me if it was because I was no longer getting any data from the CurrentCost itself, or I was getting data but was failing to post it to Cosm via the EtherCard library.

Cosm data upload flatlines

To validate the data from the CurrentCost, I put in some additional debugging to check that values were still being received and sure enough, every six seconds or so it spat out a line which matched that on the CurrentCost’s display – so they were still being updated, but not being posted.

Not knowing what may be up inside a library that I’d never really used before, I added some more debugging to the console and additional metrics to the Cosm feed. The console now showed me the HTTP request going out and (through the recently added ether.tcpReply() function) the HTTP response coming back. I added current available memory (using the MemoryFree library from the Arduino Playground) and stash.size() & stash.freeCount() (the Stash is implemented in the EtherCard library and uses the RAM inside the ENC28J60 Ethernet controller as a scratchpad) to the Cosm feed, restarted and waited for the data to come rolling in. Sure enough, after twenty minutes or so, it died in a pathetic whimpering kind of way …

There was no apparent drop in available memory on the Arduino, nor was their a change in the Stash size (though I hadn’t expected there to be) but what was apparent was the way the freeCount() on the Stash dropped off and never recovered, until there were no free buffers. At this point, the HTTP debugs show that the outbound request becomes corrupted – which is why the Cosm feed stops updating. The requests still get sent, it’s just that they’re full of crap, so Cosm ignores them.

Death of the Stash

So what’s the solution?“, I hear you ask. Well, now that I had an idea of where the problem might be I tracked down a number of discussions of similar sounding problems, most of which seemed to be being experienced by users in the Nanode community (as the Nanode uses the same ethernet chip and thus the same library). Figuring that if the freeCount() kept dropping and never recovered, I could try re-initialising the Stash before it got into a terminal state. Taking inspiration from EtherCard::begin() I added the following just before I tried to use the Stash as a quick’n’dirty hack to see if I could fend off the crash;

if (stash.freeCount() <= 3) {
  Stash::initMap(56);
}

Success! Here we can see the stash.freeCount() dropping over time (third line in the graph) and then getting re-initialised … and all the while we’re still posting power and temperature readings with no flatlining, which was the original point of all of this, if you remember.

EtherCard library Stash::freeCount() drops over time then recovers

As always seems to be the way with these things, once I’d found a solution (of sorts) I was then able to do a more targeted Google and found a discussion on the nanode-users group where the same solution was mooted. Looking back over the debug log, it would also seem that the freeCount() dropped when a corresponding reply was not found for a request, as noted there by SomeRandomBloke.

I’ll tidy up the code with the new features in and post that up later on.

UPDATE (01/06/12): The problem with the Stash freeCount() dropping appears to be related to the frequency of updates; if I throttle it so that it only sends every second update to Cosm (i.e. once every 12 seconds instead of every 6) then the freeCount() never drops. Whilst re-initialising the stash is a bit of a hack, it’ll do just fine till Jean Claude is able track down the underlying cause of the problem.

  5 Responses to “There’s something wrong with my Stash”

  1. Fantastic piece of comment!
    I’m encountering the same issue, however I’m experiencing a decrease of 2 after every Pachube post.

  2. Thanks a million for this. Been scratching my head for a while over this.
    The cludge is working fine an I can now post continuously.
    I’ve posted freecount on my stream 76650.

  3. Hi,
    This is my code :

    #include

    // change these settings to match your own setup
    #define FEED “106042”
    #define APIKEY “VTtwyNkkJ5gaSP8ndhJ_DlD7NRKSAKxjQTFndzJFakJ0bz0g”

    // ethernet interface mac address, must be unique on the LAN
    byte mymac[] = { 0x74,0x69,0x69,0x2D,0x30,0x31 };

    char website[] PROGMEM = “api.cosm.com”;

    byte Ethernet::buffer[700];
    uint32_t timer;
    Stash stash;

    void setup () {
    Serial.begin(57600);
    Serial.println(“\n[webClient]”);

    if (ether.begin(sizeof Ethernet::buffer, mymac, 10) == 0)
    Serial.println( “Failed to access Ethernet controller”);
    if (!ether.dhcpSetup())
    Serial.println(“DHCP failed”);

    ether.printIp(“IP: “, ether.myip);
    ether.printIp(“GW: “, ether.gwip);
    ether.printIp(“DNS: “, ether.dnsip);

    if (!ether.dnsLookup(website))
    Serial.println(“DNS failed”);

    ether.printIp(“SRV: “, ether.hisip);
    }

    void loop () {
    ether.packetLoop(ether.packetReceive());

    if (millis() > timer) {
    timer = millis() + 10000;

    // generate two fake values as payload – by using a separate stash,
    // we can determine the size of the generated message ahead of time
    byte sd = stash.create();
    stash.print(“0,”);
    stash.println((word) millis() / 123);
    stash.print(“1,”);
    stash.println((word) micros() / 456);
    stash.save();

    // generate the header with payload – note that the stash size is used,
    // and that a “stash descriptor” is passed in as argument using “$H”
    Stash::prepare(PSTR(“PUT http://$F/v2/feeds/$F.csv HTTP/1.0″ “\r\n”
    “Host: $F” “\r\n”
    “X-CosmApiKey: $F” “\r\n”
    “Content-Length: $D” “\r\n”
    “\r\n”
    “$H”),
    website, PSTR(FEED), website, PSTR(APIKEY), stash.size(), sd);

    // send the packet – this also releases all stash buffers once done
    ether.tcpSend();
    }

    where should i add the above lines of code??

  4. Hey Thank you for your blog about this, since I have had similar issues that my feed always froze after a short while. I was sure it wasn’t due to some tcp/ip stuff, I have a webserver that is running rock-solid with same jcw enc28j60 lib, but needed just your blog to get me into the right direction, to understand the real issue :-) Thank you.

    My feed have now been stable for about 6 hours, it has never before been this long :-)

    My hack inspired by you is to clear the Stash memory, just before I create the Stash.
    stash.cleanup();

    //Create the stash to save temp data
    byte sd = stash.create();

Go on, leave a reply...

%d bloggers like this: