Wednesday, June 18, 2014

I Want To See Violet

One of my earliest memories is standing in front of My Wonderful Mom as a young lad proclaiming how much I loved the purple and white pants I was wearing.  They were so cool.  They were the best.  There was only one problem: they weren't purple.  They were blue.

It probably came as no great surprise to my parents that I was colorblind (the politically correct name for which these days is "color vision deficient", but I have been and always will be colorblind).  My uncle on my mom's side is red-green colorblind (and how he made a successful living as a farmer is beyond me).  Color blindness is genetic and passed along from your mom.  I have her to thank for my colorblindness, my puny musculature, and 50% of the other things I can blame on my DNA.  Thanks Mom.

But I was young and able to bounce back.  I was accepting of the fact that many of the professions I might otherwise aspire to probably would not work out in the end: interior designer ("What have you done to my beautiful house!?!?"), bomb disposal expert ("Cut the green wire."), and last but not least, professional telephone cable splicer.  I was the butt of these jokes and many others through life.
I Have Nightmares Like This
My chosen profession of Engineering is bad enough.  Geology labs back in University were horrible.  I remember having to identify different types of rocks in an exam, and one of the keys in doing so is going by color.  Worse was the infamous resistor color code. Carbon film resistors are marked with a series of colored bands to indicate their value and tolerance.  To make matters worse, my difficulty in resolving colors is compounded when the colored bit I'm looking at is small.  To this day, I have never bothered to memorize this fundamental code: there's no point in me doing so. 
Pointless
When people think of color-blindness, they think of the whole rods and cones thing, but there is more to it than that.  The best explanation I've come across is this one:
Colour vision is based on an opponent process [1]. There are three kinds of colour receptors called L, M and S (for Long, Medium and Short) and they are receptive to colours centred around the red, green, and blue parts of the visible light spectrum respectively.

Here is a good graph showing how sensitive each receptor is to different wavelengths of light: 
But we don't perceive colour directly from these receptors. Instead, our visual systems combine the output of these receptors to form three channels: black vs. white (L+M+S), red vs. green (L-M) and blue vs. yellow (S-(M+L)). This is why there is no such colour as a reddish-green or a bluish-yellow; our visual systems are not able to perceive these (except under exceptional laboratory conditions). As you can see in the graph, the L, M and S receptors aren't just receptive to pure red, green and blue but to a range of wavelengths centred near these colours, and they all overlap to some degree. But the L (green) and M (red) receptors overlap quite a bit.
In red-green colour blindness, either the L or M are shifted so they're even closer. This means that the red vs. green (L-M) channel is no longer able to distinguish between the two, so this channel is always close to zero and the person no longer sees either red or green just something muddy in between. Blue-yellow colour blindness is similar, but in this case it's the S channel that's shifted towards the L and M. In this case it overlaps with both and this means there's no longer any colours that activate the S without also activating L and M (and vice versa), making it impossible to distinguish blue from yellow.

I stumbled on this when skimming through YCombinator on a lazy Sunday afternoon.  The link above pointed to an article on BoingBoing called "Color for the Colorblind".  I read it and was fascinated.  The article describes sunglasses from a company called EnChroma that promises to correct colorblindness while you are wearing them.  Wow.

First you take a test on their website to determine what kind of colorblindness you have and how bad it is.  The test results told me I was a "strong duotan", and I had no idea initially what that meant.  A little research told me that they were telling me what I already knew: I was red-green colorblind, and I had it bad.
Damned If I Can See Anything but a Bunch of Dots
Then I got to this video on the Press section of their site.  At around 45 seconds into it, I started to cry.
...there is always an "aha" moment.  When they'll turn and look at something and say... "Is that flower... violet?" And they've never seen violet before...
Neither had I.

I pulled myself together, went downstairs, and asked My Lovely Wife to bring up first this video called No Such Thing as Color.  It starts with a guy looking at different houses and trying to guess what color they are.  Thing is, I can't tell if he is getting them right or wrong.  I'm guessing he gets a bunch of them wrong, or he wouldn't be colorblind.

 Skip The Last Half - It Gets Pretty Lame

Then I asked My Lovely Wife to watch the EnChroma video.  At around 45 seconds in, I started crying again.  I think she thought I was laughing until she turned around and saw the tears in my eyes.  I desparately wanted to see violet too.

I put in my order to EnChroma later that night for a pair of EnChroma Cx Receptors.  These are designed to fit over top of my prescription glasses, without which I see nothing but a blur.  The three to five business day wait for shipping turned into three weeks.  Seems a lot of other people had read that same BoingBoing article and wanted to see violet as well.  Oh well.  I had been colorblind all my life.  I could wait a few more weeks.

As I waited for my order to arrive, I didn't tell anyone besides My Lovely Wife, My Wonderful Mom, and My Cool Sister.  I had to tell somebody, but I didn't want to spread the news around too much; I wasn't sure if these things were going to work in the first place.  I did drop the odd hint though, like changing my Facebook profile picture.
I kept a close eye on my UPS tracking number as my precious shipment wound it's way up from California to my home in the Great White North.  When the happy day arrived, I went to the courier to pick them up and got a little more than I bargained for, to the tune of...
  • $20.77 in customs fees
  • $21.81 in taxes
  • $64.60 in UPS brokerage fees (!).  No wonder there have been several class action lawsuits filed against UPS for shipments into Canada.  This is exhorbitant.
Now none of these are Enchroma's fault.  Fees and taxes are inescapable, but I do wish they would offer to ship by US Postal Service to minimize shipping costs to non-US residents.

Anyhoo, they say the glasses work best on a bright sunny day.  I got home late on Friday so decided to wait until Saturday to try them out.  The weather on Saturday was crap so I grudgingly pushed the first try to Sunday.  The crappy weather continued, and it wasn't until we got a small break in the crappiness on Tuesday (with the forecast for extreme crappiness in the days following) that I decided to finally give them a shot.

I wanted to try them first in our backyard.  It has a big flowerbed in the back with several different plants in bloom.  Our backyard also faces a river with farmers fields beyond that, making for quite a sight, colorblind or not.  I also had my wife but on her blouse that she tells me is purple.  I wanted to see violet, after all.

I took a long look around before putting on the EnChromas.  I wanted to get a good sense of "before".  My Lovely Wife got the camera out, started recording, and I put the glasses on for the first time.

The partly-cloudy sky transformed itself into something from an alien world.  I didn't really know if what I was looking at now was accurate or just... different.  Whatever I was looking at had far more "pop" than what I had noticed before.  I just stood there and stared for a while, repeating that it "looked different".  I noticed my mouth was hanging open.

I turned around and our pink stucco house exploded into my view with a vibrancy I could not have imagined.  What I thought had been dull and faded was far from it.  Then I looked down and the chickweed in our lawn blazed in front of my eyes like it had been electrified.  I was taken aback.  I tried to describe it to My Lovely Wife as the recording went on but my voice started to crack and I felt the emotion washing over me like that day from weeks ago.  I am probably the first person ever to be overcome by emotion from a weedy lawn.

I looked over at My Lovely Wife and could see that she was getting choked up as well, her violet blouse shining before my eyes.

I had gotten what I was after.  I signaled to her to stop the recording so we could see how it turned out.  I wanted to share the experience I'd just had with others but, alas, there were technical difficulties and these words will just have to do.

I kept the sunglasses on and took a walk around the yard.  The brown stain on our deck was no longer brown, it was brown, with a richness I had not seen before.  I then noticed how browns stood out so much more from greens than they did before.  A small dead pine tree some distance away that would have been almost invisible to me before in the grassy background was easy to spot.  And was our green truck ever green!  A world that had been hidden from me was starting to reveal itself.

I looked down at myself and the burgundy shirt I was wearing lit up before my eyes.  "WOW!", I shouted.  "LOOK AT MY SHIRT!!!"  My Lovely Wife had always liked the shirt more than I did.  To me, it seemed a bit dull.  I was wrong.  I kept staring back at it thinking I had made some kind of mistake.

It was time to go back into the house.  I kept the glasses on but didn't expect much wearing them inside.  As I thought, I didn't see any real improvement, and the folks at EnChroma caution you about this: they work best out in the open with bright natural light.  They came up empty in early evening on a cloudy day.

I am looking forward to the coming days and weeks.  Apparently, they become more effective as your brain adapts to a new way of seeing the world when the sunglasses are on.  The crappy weather might slow that process down, unfortunately.  On the other hand, I'm really hoping for a nice rainbow sometime.  People with this eyewear rave about rainbows (and fortunately for me, the price on these things has come down since that article as well).

I got into this wanting to see violet, but so far it is the greens that are blowing me away.  The new growth on the tips of spruce trees stand out like the lights on a Christmas tree.  I find myself distracted by the rain soaked, green grass growing brightly in the ditches on the drive in to work.  And I will always remember the first time I saw that chickweed and really saw green.  It is only too bad that these sunglasses aren't a cure for colorblindness.  They only do their magic while being worn.  Once I take them off, I return to my muted, drab world.  But to me, something is a hell of a lot better than nothing.

Now my new Facebook profile picture should make sense to everyone.

Sunday, June 15, 2014

Diablo Bread, The Bread of the Devil

I first blogged about bread around three years ago.  I look back at that post discussing my tips and tricks for Tartine bread and it makes me cringe.  What I thought was a good loaf of bread back then is far below what I'm able to come up with these days.  What has made me a better baker?  The same way you get to Carnegie Hall.  Practice.  I make a loaf of bread pretty much every weekend.
Tartine Bread Try #43
And it isn't just Tartine.  I find I am making that recipe ever month or so, while the other weekly bakes will sometimes be a No Knead bread of some kind or another based on Jim Lahey's technique outlined here.  A post back in May 2012 gave a link to another excerpt from "My Bread" that has more recipes you should check out as well.

Now with familiarity comes the courage to strike out on one's own.  I've been on quite a Sriracha kick lately and I got to thinking, what about a Sriracha No Knead Bread?  Put that term into Google and what do you get?

No results found for "Sriracha No Knead Bread".

Not acceptable.  Time to blaze a trail.

I call this bread Diablo Bread, The Bread of the Devil.  The Sriracha in the recipe is amped up by copious amounts of crushed red pepper flakes.  The orange-red color of the baked result is a warning that this is not a bread to be messed with.

Before starting, watch this video from the man himself to get an idea what you'll be doing.

No-Knead Bread From the Master

My recipe is along these lines, but my method is somewhat different from the second rise onward.  I also use a round Lodge Combo Cooker rather than the oval shaped Dutch Oven Jim uses.  You'll have to tweak my procedure a bit if you are using the latter.  But I'll tell ya, the Lodge is pretty great for baking bread.  And it is an awesome abuse of Amazon's free shipping policy.
Lodge LCC3 Combo Cooker
So without further adieu...
Diablo Bread, The Bread of the Devil
Inspired by Jim Lahey's No Knead Breads 
Ingredients
400 grams unbleached all purpose flour
7 grams table salt
½ teaspoon quick rise instant yeast
1 ½ tablespoons crushed red pepper flakes (Yes.  That is a lot.  Not a typo)
260 grams cool water (55 - 65 °F)
60 grams (4 tablespoons) Sriracha sauce (Don't wimp out!  Not a drop less!)
Wheat bran for dusting
Tools
Cooking spray or oil
Parchment paper
Tin pie plate
Dough scraper
Plastic wrap
Lodge LCC3 Combo Cooker or cast iron Dutch Oven
Serrated knife
Oven mitts
An ice cube
 Procedure
  1.  Mix the flour. salt, yeast, and pepper flakes together in a medium sized bowl.  Add in the water and Sriracha sauce.  Mix using a wooden spoon or your hand until all of the flour is incorporated and the dough is sticky.  This should only take 30 seconds or so.  Add more water if the dough seems too dry.
  2. Transfer the dough to another bowl lightly oiled or sprayed with cooking spray.  This will help prevent the dough from sticking when you dump it out after the first rise, and this in turn helps to prevent the dough from deflating.  The dough will increase in volume probably by a factor of four at least, so be sure to use a big enough bowl.  Use a bowl that is red on the inside to amp up the viciousness that is this dough.
  3. Cover the bowl with plastic wrap so the surface of the dough doesn't dry out during the bulk ferment (a fancy-schmancy baker's term for first rise).  Let it rise at a temperature around 70°F in a spot out of the sun for 18 hours.  Or you could be like me (everybody should be like me) and put the bowl in a water bath maintained by your home-built Sous Vide rig at a rock-solid 21.2°C.  You'll have to shorten the rise or use a little less yeast if rising at temperatures that are a lot warmer, but try not to get into this pickle in the first place.  The long rise time contributes to the flavor of the loaf.
  4. While the dough is rising, cut a circular piece of parchment paper to fit the Combo Cooker using the pan as a template.  Cut it wide enough to stick up above the edges of the cooker.  You'll be putting this paper in the pie plate for the dough's second and final rise.
  5. After the 18 hour bulk ferment (fancy-schmancy!), generously flour your work surface.  The circular piece of parchment paper should be sitting on top of your pie plate.
  6. Gingerly tip the bowl containing the risen dough over the floured surface, taking care not to deflate the dough.  You don't have to have your dog looking out the window when you do this, but it helps.
  7. I usually end up with a round blob of dough that I want to shape into a ball.  To do so, I first use a dough scraper (actually a plastic drywall knife, 'cause that's how I roll) or my hands to gently fold two opposite ends of the blob toward the center to make kind of a rectangle.  Then I use my hands and gently grab one long end of the rectangle and roll it up towards the other to get my ball of dough.  Place the ball of dough seam side down in the center of the parchment paper that is sitting on the pie plate.




  8. Dust the top of the dough with some wheat bran.  Cover the whole thing with a clean tea towel or plastic wrap to keep the top of the dough from drying out.  I use a shower cap stolen from a hotel - the loose elastic does a great job of getting around the pie plate without pulling the plastic too tight.
  9. Place the pie plate containing the dough in a nice warm spot for the second and final rise (around 75 - 80°F).  I put it in the oven with the light on if the house is cool.  I let this rise go for two hours.  It should look something like this when it is done.  Note the little dimples on top.  I was giving this dough the poke test: the idea is that you poke the bread and the dough should hold the indentation when it has properly proofed.  Don't go longer than you need to here, or the dough will be over-proofed and rise poorly in the oven.
  10. Half an hour before the end of the final rise, put the Combo Cooker into your oven with the rack set in the middle.  Set the oven to 475°F.  You want to be using the Combo Cooker with the shallow frypan part as the base and the deep cooker part as the lid.  It is beyond obvious that if you are doing the final rise in the oven with the light bulb on, that you should take the dough out first.
  11. After the half hour preheat at 475°F, take the cover off your risen dough and place it next to your oven.  Have your ice cube and serrated knife standing by.  You are going to work quickly here.  Quickly and extremely carefully.
  12. Put on your oven mitts and take the smoking hot Combo Cooker out of your oven and on to the top of your stove.  Take off your oven mitts, pick up your pie plate and gently set it about halfway into the pan of the Combo Cooker.  Slide the parchment paper with the dough sitting on top out of the pie plate and onto the pan, using your fingers on the parchment paper to maneuver it into place.
  13. Once the dough is centered on the pan, use a serrated knife to cut a slash or two onto the top of the dough.  The dough will rise in the oven and expand where you make these cuts.
  14. Working really fast, put an ice cube between the pan and the parchment paper under the dough.  The idea is to add extra steam inside of your cooking vessel and prevent a crust from prematurely forming on the bread that would inhibit its rise.
  15. Get you oven mitts back on, flip the deep part of the Combo Cooker upside down, and use it to cover the pan.  You'll hear the ice cube sizzling away even with the cover on.  Put the Combo Cooker into your hot oven.  Keep the oven temperature at 475°F.  Let the bread cook covered for 30 minutes.
  16. After 30 minutes, take the cover off the bread and gasp at its beauty.  You'll want it to go back into the oven until the top is a nice dark brown.  This usually takes only an extra five to ten minutes.  Keep a close eye on it so it doesn't burn.  I usually set the lid of the Combo Cooker under the pan to serve as a heat shield for the bottom of the bread to prevent it from getting too dark.  If you have a thermometer, you'll want to see around 205°F in the center of the loaf when it is done.
  17. Once the loaf is a nice dark brown, take it out of the pan and set it on a wire rack to cool for at least an hour.  You'll be rewarded by a lovely crackling sound as the bread cools if everything went well.
Notes
  • If you don't have a kitchen scale to measure out these quantities, go buy one.  I'll wait.  Every  baker should have a scale.
  • The usual No-Knead recipe calls up ¼ teaspoon of yeast instead of the ½ teaspoon I use here.  It seems the spices kick the hell out of the yeast so extra is required.
  • A little less salt is used in this recipe vs. the usual because of the salt in the Sriracha.  The liquid volumes were adjusted accordingly as well.
This bread goes great with pretty much everything.  It is a match made in heaven with chili.  Or pop it into the toaster oven with some sliced cheddar on top.  Me?  I often eat it plain without even butter or jam.  I just close my eyes and savor every bite.  If you are thinking there is something that this bread won't go with, you are wrong.

This bread is a favorite of mine, and not just because I invented it.  The Sriracha and crushed red pepper flakes are great in combination but are not overwhelming - the bread naturally moderates the heat and you are left with some pretty fantastic flavor that can stand up well with anything you dare pair it up against.
Go Make Some.  Now.

Monday, February 17, 2014

Build Your Own Davis Weather Station Console!!!

This Davis weather station hacking stuff is addictive like crack cocaine is addictive.  I always seem to be taking another hit from the pipe.  I've figured out how to get into the Davis console's serial port (after which the two of us had a complete and frank discussion COMPLETELY IN UPPERCASE).  I then built my own Data Logger so I wouldn't have to buy the ridiculously expensive Davis dongle.  Then, after I figured out the wireless transmissions from the outdoor sensor suite, I made my own ISS receiver with a Pretty Pink Pager.  But of course, I can quit any time I want.  Me and Amy Winehouse.
"They tried to make me go to rehab I said No, No, No"  -  Bad Decision

Famous dead drug-addled musicians aside (and how long a list is that?  Don't they know that only Keith Richards is Immortal?), the path I was to go down became clear after Davis pulled a ridiculously stupid stunt in the infamous Firmware 3.0 release they started shipping in new consoles a while back.  The new firmware read 64 bytes from a fab-written, read-only section on the data logger's memory chip.  It took this data, did some algorithmic juggling, and then did a check against another 64 byte write-once block on the data logger put there by Davis.  If the console didn't like what it saw, it declared that the logger was invalid and refused to function further.

This was an obvious and misguided attempt to shut down a few hackers from building and selling third-party data loggers for a fraction of the price Davis charged.  Problem #1 was that people with an older, authentic Davis logger discovered that their pricey little dongle stopped working after a firmware "upgrade".  Problem #2 was it also disabled the serial interface connection I had discovered.  And that pissed me off.

Unfortunately for Davis, the console does not run a 2 GHz Core i7 processor capable of state of the art encryption and decryption algorithms.  It has a little Atmel Atmega processor running at 1 MHz or something like that.  It's abilities are somewhat more... restricted.  Like something that could be cracked by a 256 byte lookup table and a trivial bit of code. Hats off to Watson on WXForum for figuring this out. His most excellent hack is posted below for posterity.  Break out your Arduino and start cutting and pasting if you want to calculate a valid security code for a DIY logger.

u8 const GreenDot_Table[256] =
{
0x00, 0x04, 0x08, 0x0C, 0x10, 0x14, 0x18, 0x1C, 0x21, 0x25, 0x29, 0x2D, 0x31, 0x35, 0x39, 0x3D,
0x46, 0x42, 0x4E, 0x4A, 0x56, 0x52, 0x5E, 0x5A, 0x67, 0x63, 0x6F, 0x6B, 0x77, 0x73, 0x7F, 0x7B, 
0x8C, 0x88, 0x84, 0x80, 0x9C, 0x98, 0x94, 0x90, 0xAD, 0xA9, 0xA5, 0xA1, 0xBD, 0xB9, 0xB5, 0xB1, 
0xCA, 0xCE, 0xC2, 0xC6, 0xDA, 0xDE, 0xD2, 0xD6, 0xEB, 0xEF, 0xE3, 0xE7, 0xFB, 0xFF, 0xF3, 0xF7,
0x18, 0x1C, 0x10, 0x14, 0x08, 0x0C, 0x00, 0x04, 0x39, 0x3D, 0x31, 0x35, 0x29, 0x2D, 0x21, 0x25,
0x5E, 0x5A, 0x56, 0x52, 0x4E, 0x4A, 0x46, 0x42, 0x7F, 0x7B, 0x77, 0x73, 0x6F, 0x6B, 0x67, 0x63,
0x94, 0x90, 0x9C, 0x98, 0x84, 0x80, 0x8C, 0x88, 0xB5, 0xB1, 0xBD, 0xB9, 0xA5, 0xA1, 0xAD, 0xA9,
0xD2, 0xD6, 0xDA, 0xDE, 0xC2, 0xC6, 0xCA, 0xCE, 0xF3, 0xF7, 0xFB, 0xFF, 0xE3, 0xE7, 0xEB, 0xEF,
0x31, 0x35, 0x39, 0x3D, 0x21, 0x25, 0x29, 0x2D, 0x10, 0x14, 0x18, 0x1C, 0x00, 0x04, 0x08, 0x0C,
0x77, 0x73, 0x7F, 0x7B, 0x67, 0x63, 0x6F, 0x6B, 0x56, 0x52, 0x5E, 0x5A, 0x46, 0x42, 0x4E, 0x4A,
0xBD, 0xB9, 0xB5, 0xB1, 0xAD, 0xA9, 0xA5, 0xA1, 0x9C, 0x98, 0x94, 0x90, 0x8C, 0x88, 0x84, 0x80,
0xFB, 0xFF, 0xF3, 0xF7, 0xEB, 0xEF, 0xE3, 0xE7, 0xDA, 0xDE, 0xD2, 0xD6, 0xCA, 0xCE, 0xC2, 0xC6,
0x29, 0x2D, 0x21, 0x25, 0x39, 0x3D, 0x31, 0x35, 0x08, 0x0C, 0x00, 0x04, 0x18, 0x1C, 0x10, 0x14,
0x6F, 0x6B, 0x67, 0x63, 0x7F, 0x7B, 0x77, 0x73, 0x4E, 0x4A, 0x46, 0x42, 0x5E, 0x5A, 0x56, 0x52,
0xA5, 0xA1, 0xAD, 0xA9, 0xB5, 0xB1, 0xBD, 0xB9, 0x84, 0x80, 0x8C, 0x88, 0x94, 0x90, 0x9C, 0x98,
0xE3, 0xE7, 0xEB, 0xEF, 0xF3, 0xF7, 0xFB, 0xFF, 0xC2, 0xC6, 0xCA, 0xCE, 0xD2, 0xD6, 0xDA, 0xDE
};

u8 const Adesto_Factory_Programmed[64] = 
{
  // put here AT45DB011D Security Register from byte 64 to 127
};

u8 OneTime_User_Programmable[64];

OneTime_User_Programmable[0] = 0x00; // or whatever you want
OneTime_User_Programmable[1] = 0x00; // or whatever you want
OneTime_User_Programmable[2] = 0x00; // or whatever you want

u8 n, value;

for(n=3;n<64 br="" n="">{
 value = (u8)(Adesto_Factory_Programmed[n] + n);
 OneTime_User_Programmable[n] = GreenDot_Table[value];
}

For more on this whole sad saga, you'll want to read this masterwork by torkelmj, especially if you enjoy reading stuff dripping with spite and vitriol.  Like I do.  Or skip that and just buy a security-coded third party logger from WXForum awesomedude Belfryboy.  Or order up his PCB on oshpark and build your own.  There are options now, and options are good.

Now one might think that everything is just peachy now that DIY loggers are possible once again.  Not me.  I was pretty grumpy that Davis had tried to screw their customers over even if they did release subsequent firmware updates that tried to undo some of the damage.  Once burned, twice shy, so they say.  I started thinking of a Plan B.  I want to build my own console.
Gratuitous Picture of My Dog
My IM-ME thing was a neat hack, but that is about as far as it goes.  IM-ME's aren't actually rolling off the production lines in millions of units per month these days, and their ability to hook up to other little do-dads would not be easily done.  Cheap transceiver boards based off of the popular CC1101 chip that work in the 915 MHz band are hard as hell to come by.  Ask me how I know.  The Seeedstudio RFBee comes tantalizingly close, but others who have tried to go this route run out of RAM and Flash on the limited processor found on this unit before getting too far.

I needed another way and that led me to the Moteino with its awesome little RFM69B transceiver module.  I got it picking up Davis ISS transmissions earlier this year.  Since then I have been coding away in my spare time and have been pulling together the various bits of hardware that I need to get something going.  Something like this.
Beautiful, Isn't It?

Some cool stuff has sprung up from this very breadboard.  My serial connection to the console grew up there on the left side.  The DIY logger design is in the middle.  The stuff on the right half starting from the left is the humidity sensor, the pressure sensor module, the Moteino itself, and finally my USB to Serial Adapter for the project I'm writing about today.

Now I know what you are saying to yourself: "The title of the blog is "Build Your Own Davis Weather Station Console!!!" but that doesn't look like a console at all!  Where is the keyboard?  Where is the display?" Well, I guess I'm not going to get anything past you today.  There isn't a keyboard or a display... unless you want to add one.  The hardware and software here is all open for you to add on what you'd like.  Me, I'm thinking of hooking this bad boy to a serial port on my Beaglebone running Openhab and having all my weather data available from anything in my house that can bring up a web browser.  Very doable, and no display or keyboard necessary for that kind of application.

But why call it a "Console"?  Because I couldn't think of a better name.  And I did indeed try.  It isn't a "Receiver" because it does more than receive given that it has its own sensors hanging off of it.  "Indoor Unit" sounds stupid.  "Thingamahooie" was tempting, I must admit, but didn't quite convey the application well enough.  And hey, the guy that builds the hardware, develops the code, and writes the blog gets to call what he comes up with whatever the hell he wants.  I'm going with "Console", and you'll just have to deal with it.

But I, as I so often do, digress.  Back to business.  On the hardware side of things, here is what I'm working with in more detail:
  • Moteino with an RFM69 module on it.  Pick the 915 MHz RFM69W (standard transmit power) or RFM69HW (high transmit power) at your option.  I have the serial version but it probably would have made sense to pick up the USB variant so I wouldn't need to tie up my USB-Serial converter all the time.  You probably want to add on the male headers option (why even think about it for the one dollar price?) and the flash memory option for four dollars.  I'm not using that yet, but it might come in handy for data logging at some point.  And if you re-dedicate the Moteino to something else later, this memory can be used for wireless remote programming.  Cool.
  • a DHT22 Temperature and Humidity Sensor for less than five bucks.
  • a BMP085 Temperature and Pressure Sensor for less than six bucks.  But it looks like that module is now much cheaper thanks to the compatible BMP180 sensors now out there for under three bucks.  The two are compatible, and I'd probably go with the newer BMP180 myself just because it is smaller.
  • a (yet to arrive and be connected) DS3231 Real Time Clock and Memory Module for less than three bucks (I felt bad for getting something at this price with free shipping so I bought two).  Note that this module also has a 24C32 EEPROM on it that is bound to come in handy.  Don't be dumb and order a DS1307 clock module instead: that one doesn't work well with the 3.3V Moteino and it is horribly inaccurate.  What good is a clock chip that can't keep time?
Arduino is all about little software libraries, and these are the ones I'm working with right now.
  • the LowPowerLab RFM69 library.  Felix has merged the changes I've needed so no need to use my fork of his code any longer.  I don't know if I'll be using this in the long term or come up with my own that is better suited for this application.  Another library that I'm including right now but not really using yet is his SPIFlash library.
  • the JeeLabs DHT22 code.  I have extracted what I need from the JeeLib code and included it into my own library.  JCW's stuff is a leaner and meaner version of some of the other ones floating around on the net.
  • the Adafruit BMP085 library.  I used this as is.  I did look at using the JeeLabs code but it was a little too tied up with JCW's PortsLib for me to bother untangling at the time.
  • the Arduino Serial Command library to parse command input.  This is a really great little library that lets you register callbacks for each separate command and also does command line argument parsing.  I needed to make a few tweaks as noted below, but nothing major.  My changes have been pushed but not merged at this time, so use my fork here.
  • my own DavisRFM69 library, of course.  I've got to be doing something to make all this stuff work together.
Enough chitchat.  Let's cut to the chase, shall we?  Take a look at this thing actually doing something.
Success - I Never Tire of It
This is SandaySoft's Cumulus software talking to my homebrew weather station console.  The outdoor readings are those from my ISS in the front yard.  The indoor readings are from the sensors I have hanging off the Moteino.  On the left you can see the Serial Port Monitor Software recording the RS-232 messages going back and forth.  Cumulus has no idea that it isn't talking to a real Davis console.

Thanks to the Davis Serial Protocol Manual, getting this working wasn't too terribly difficult.   I knew Cumulus used the Davis SDK to talk to the console so the Cumulus author might not be aware of every single command being sent to the station and back.  I also knew that the LOOP command was used to get the data out of the station so I implemented that and some other useful commands on the Moteino.  Then I hooked my real console up and watched some packets go by and found a few more I needed to implement (namely BARDATA, HILOWS, and RXCHECK).  I had a few hiccups of course.  For example, Davis says that a command should always be terminated with '\n' (0x0a) like this...
 except when they don't, like this...
And that is the fun with emulation: you have to emulate not only the functionality, but the bugs as well.  This one caused me to tweak the serial port library to accept multiple command terminators.  Another oddity is that the console itself goes to sleep all the time and is woken by simply sending a lone '\n' that the console is supposed to acknowledge.  The serial library I'm using hadn't actually accounted for that situation.  Now it does.

Want to play along?  Here is what you need to do.   From within your Arduino's libraries directory...
  • Get my version of the SerialCommand library that supports empty commands and a second line terminator.  And of course, you'll want to get a copy of my DavisRFM69 library.  The pared down DHT22 library is tucked away inside the latter.
git clone https://github.com/dekay/Arduino-SerialCommand.git SerialCommand
git clone https://github.com/dekay/DavisRFM69.git
git clone https://github.com/adafruit/Adafruit-BMP085-Library.git Adafruit_BMP085
git clone https://github.com/LowPowerLab/RFM69.git
git clone https://github.com/LowPowerLab/SPIFlash.git
Hardware hookup is like this.

DHT22 Pin 1 -> Moteino 3.3V Out
DHT22 Pin 2 -> Moteino D4 with 10K Pullup resistor to Moteino 3.3V Out
DHT22 Pin 3 -> Not connected
DHT22 Pin 4 -> Moteino GND

BMP085 Module Pin VCC -> Moteino 3.3V Out
BMP085 Module Pin SDA -> Moteino SDA
BMP085 Module Pin SCL -> Moteino SCL
BMP085 Module Pin GND -> Moteino GND
Once you've got all of this hooked up, you'll find a "VP2.ino" sketch in the DavisRFM69 library's Examples directory.  Open that up in the Arduino IDE and select Control-U to compile and download it to the Moteino (be sure to tell the Arduino IDE that you are using an Arduino Uno).  Open up the serial monitor, set the baud rate to 19200 and "Newline" as the line ending.  You should see... absolutely nothing.  Nothing that is, until you type in some (uppercase) commands like "LOOP 1", "TEST", or "STRMON".  The Moteino will respond exactly the same way as a real console would.  Hook it up to Cumulus (with the "Use Data Logger" option unchecked) and revel in a job well done.

I'm off to a decent start here but there is still lots to do.  Here are some things on my list.

  • integrate the real time clock functionality
  • put in the frequencies for consoles in Europe and other regions
  • add some more useful commands like PUTRAIN so you can tell this thing how much rain you've had this year that it can keep a running count of afterward
  • add in some kind of EEPROM functionality so it retains settings like total rainfall between power downs.  Conveniently, there is an EEPROM on my yet to arrive three dollar clock module just begging to be used.
  • make the reception more resilient.  There are some odd things going on here: I get a spurious interrupt on startup (setup issue?) and when I move the board around (need a pulldown on the interrupt line?)  Reception is also poor during the day.  Might be that I just need to adjust the clear channel RSSI value a bit to accommodate more background noise.
  • integrate the real time clock functionality (daily / weekly / monthly / year highs and lows)
  • etc
Anyone that has done much coding on the Arduino gets to know memory limits.  The Moteino has double the RAM and EEPROM than the standard Arduino UNO, so I've got a whopping 2K of RAM and 32K of Flash to play with.  So far I have used roughly half of each, so I've got some headroom.  I haven't done much to try and streamline any of the code besides a few const arrays here and strings stuffed in the F() macro there, so I am sure there is room for improvement if things get really tight.

With this project, I think I've hit the goals I set out to achieve: come up with something functional, inexpensive, and with a low barrier to entry so that other people can dig in.  Have a new feature you'd like to see?  Send me a pull request on Github.  Want to develop a plug in circuit board (kind of like this) with all the dodads attached to lower the barrier to entry even further while making a few bucks on the side?  Be my guest.

I'll wrap this up by saying that the Davis guys make good stuff.  The ISS is very well put together and I love that the thing is solar powered with battery backup.  I've had mine for years and it has been operating trouble free on Battery #1.  The display on the indoor console is nice, big, and easy to read and it also just sips on battery.  I'm guessing that 99.9% of Davis' customers are going to continue buying standard Davis stuff off the shelf despite whatever I come up with here.  Anything I play around with  is ony going to be picked up by that other 0.1% of enthusiasts.  And it is those enthusiasts that are not only going to continue buying Davis products but will recommend them to their friends as long as they are kept happy.  If Davis pulls another stupid Firmware 3.0 thing, they'll do more damage to themselves than I ever did.  I hope they continue documenting their communication protocol and maintaining accessible expansion ports in their current and future products.  That would be a win-win for everybody.

With that in mind, it seems as good a time as any to unveil the new profile picture.
Be Nice


Monday, January 20, 2014

For Our Dogster Friends

Update February 11th: Monster Speedup and Experimental Photobucket Rules

One thing that has become quickly apparent is that HTTrack can be ridiculously slow especially on bigger pages.   I have found a way to speed it up.  Way up. It appears that it throttles the download speed without telling you. There is a spot in the user interface that lets you set the maximum speed, but that does nothing unless you know a trick

Here is a shortcut to getting the speedup. Try putting the following in your Scan Rules
--disable-security-limits
--max-rate 1500000
--assume php=text/html
The first one is the secret to taking the brakes off. The second one is the maximum rate in bits per second. We have a two megabit download rate at our house, so I've set the maximum to three quarters of that, or 1.5 megabits per second. If you set this number too big, it just means it will hog your entire network connection. A rough rule of thumb might be that if you are getting internet through your phone company, try 1500000 as we have. If to the cable company, they are usually faster so try 5000000 (five million).

The "assume" bit is recommended in the HTTrack manual to speed up how it parses the files if you can make certain assumptions, and this assumption seems to work great.  Now it is only taking minutes to download smaller pages. The difference is night and day.  These scan rules are now in the main body of the blog post.

Not in the main body of the blog post is some new filters I have been using that seems to work well for downloading from Photobucket.
+*.photobucket.com/*.jpg
+*.photobucket.com/*.gif
+*.photobucket.com/*.png
-blog.photobucket.com/*
-secure.photobucket.com/*
-support.photobucket.com/*
-tutorials.photobucket.com/*
The lower in the list, the higher the priority, so the order above of these filters is important. The rules start by saying you can only load .jpg, .gif, and .png files from Photobucket. That stops it from downloading all the html pages that then takes you down the rabbit hole and downloading the entire site. After that, it specifically blacklists (with the minus signs in front) the blog, secure, support, and tutorial sections of the site. This is just to be on the safe side.  This has worked well for all the sites we have tried, but keep an eye on it if you decide to try it.  I have not put this in the main body of the blog post but give it a shot.

Finally, another problem we've been having is that some pictures weren't showing up after a successful download. I suspect this is because of occasional heavy traffic clobbering the site. Try the following if you are having trouble:
  • From the same window the Scan Rules are entered in, select the "Flow Control" tab
  • In that tab, enter 5 into the Retries field. 
  • Everything else is as per usual. 
Good luck!

Update February 9th: Disable Photobucket Scan Rules

It sounds like some people are running issues where HTTrack wants to download half of Photobucket.  It looks like you will need to figure out more specific scan rules than the ones I have in there now.  I have removed references to Photobucket from the post.

Update February 8th: Fixes and the Addition of Catster!

I have updated the scan rules to fix a few things and have created a set of scan rules for Catster as well.  The new rules are in the main body of the post.  A shout-out to the Dogster (and Catster!) folks that have helped out so much.
  • The gift archive now works.  A new scan filter "+www.dogster.com/subscr/gift_archive.php?pet_id=590841" (or whatever your pet's number is) does the trick.
  • Thumbnails in photo comments now work.  Dogster, for some bizarre reason, has those thumbnails in a URL that looks like "+files.dogster.com./blahblahblah".  Did you catch the weird period between "com" and "/" in that last sentence?  Neither did I at first.  Tweaking the Scan Rule from "+files.dogster.com/*" to "+files.dogster.com*" did the trick.
  • Added a scan rule"+badge.dogster.com/*" to fix missing badges.
  • Added a few new scan rules that should help images from flickr, photobucket, and one other site I'd never heard of come through: "+*mytextgraphics.com" "+*.photobucket.com/*",  "+farm*.staticflickr.com/*", and "+farm*.static.flickr.com/*".  You can skip these if you don't have images off the main site, but leaving them in doesn't hurt anything.
  • Added a set of scan rules for Catster.  Did you know that the images on Catster are stored on servers named Dogster?  Neither did I at first.  For this reason, the Catster list will have a "+files.dogster.com*" entry in it on purpose.  It will also have a "+files.catster.com*" on it that I doubt is needed, but better safe than sorry.
  • Added a scan rule that pulls in a specific file used to format the display of the photo pages "+www.dogster.com/photos/photo.css", though I didn't really see a difference when it was added.
  • Reordered the list of scan rules to be a little more sane
Attempts by the Dogster folks and myself to display images in the Pup Pals pages have failed miserably.  I suspect that it has something to do with that being a page generated on the fly with PHP and this is a known issue with HTTrack, but the gift history is also PHP and it works.  Weird.

It doesn't hurt to test the archive out by disconnecting your Internet connection, clearing your browsers cache (Google is your friend here), and then bringing up the web page created by HTTrack for your pet to see if anything is unexpectedly missing.  Note that the page might be slow to load this way because it is trying to reach out to some sites like Facebook, Google, and Twitter that I have intentionally not mirrored.  Your browser will wait for a bit, give up, and then load the archive anyway.  These links don't affect the display of your stuff and they'll be there long after Dogster Doomsday.  Once you connect the Internet back up, it will reach these sites normally anyway

If you are an old hand at this archive process, the only substantial changes to this post are in the "Specify The Options" part.  The rest is the same, so you can save yourself some reading.
Finally, some of you might be wondering why this archive takes so damn long sometimes.  I was mystified by this for a while too until I ran a test on a cat that had 36 pages (!) of diary entries.  I checked out the folder where the diaries are stored and there were a ton more entries than that.

WHY?

Well, there might be 36 pages of diary entries, but there are also five or six diary entries per page.  If you click on a "Comments" link, a whole new page opens up for just that diary entry.  Same thing for photos.  So if you have ten pages of diary entries and ten pages of photos, you don't end up downloading twenty pages.  You download twenty pages where entries are grouped together PLUS over a hundred pages dedicated to each individual diary entry and photo.  Yeah, that is a lot, but that is the only way to see the comments.  It also doesn't help that the download speeds from Dogster and Catster seems to be less than speedy in the first place.

"Beepers" on the Dogster forum posted a suggestion on how to speed up an update if you already have an existing download.
  • On the first option page, you can choose "Existing project name".  Click the arrow to choose the one you want to update.
  • On the next page/screen, in the "Action" block, click the arrow again and choose *Update existing download.
  • Then click on "Set options" and the "Scan Rules" tab as usual.  All your rules for the previous save will be there.  Just add the new rules at the bottom and proceed as usual.
Give it a shot!
          

Update February 5th:

The Bad News

Folks on the Dogster forums groups discovered that the archives would not work when they had no Internet connection.  That implied that the download would stop working once Dogster shut down.  I verified that this is a problem and am very sorry to say that all the downloads that everyone has done to this point will be useless once Dogster shuts down. The web browser was doing some stuff behind the scenes that I wasn't aware of.

The Good News

I have a fix that I would like people to try. I have tested this on a couple of pages now with my browser cache cleared and our internet disconnected and it seems to work perfectly. It still has the same limitations as before (can't click on gifts to get details, etc) but all the important stuff will be there. You will still be able to navigate around the page just like you did before.

The Fix

A new set of Scan Rules is now in the main body of the post.  The rest of the steps should be the same.  Try these new scan rules out and post feedback in the comments.  If you are computer savvy, you can double check by clearing your browser cache, disconnecting your Internet connection, and loading the archived page.  If everything looks good, you are ready for Doomsday when Dogster shuts down.

Please test this out and give feedback in the comments.  Once again, I'm sorry for my error. If I knew what I was doing here, it would help!

P.S.

Some people have been worried about seeing stuff like this in their browser address bar.

file:///C:/My%20Web%20Sites/cocopuffondogster/www.dogster.com/dogs/500107.html

The worry is that www.dogster.com is showing up there still.  That isn't actually a problem as long as the start of the address starts with "file:///C:/blahblahblah".  In this case the "www.dogster.com" part in the middle is actually now a directory on your hard drive.  You can use Windows Explorer or whatever to actually navigate to that directory and below.  There is all kinds of neat stuff tucked away in there!

          

Update Jan 25th:

See near the bottom of the page for instructions on how to save your videos!  A couple limitations of this technique are also discussed.  And after all of this, look who was made Dog of The Day for January 25th, 2014!!!


          

One of the saddest times of my life was when our dear dog Abby passed. She was my best friend and a big part of our lives. That was about a year and a half ago now and it still brings a lump to my throat just thinking about that awful time.

In the days that followed, I was struck by the sympathy and compassion expressed by My Lovely Wife’s friends in the Dogster community. Dogster is a website that dog lovers from all over the world congregate to share photos, stories, and advice on their best friends. While I don't doubt that some of these folks might be a few Milk Bones short of a full bag, My Lovely Wife has developed a true friendship with many people on that site, none of whom she has ever met in person. Our dog now gets more Christmas cards than we do.

Unfortunately, word went out a short time ago that Dogster will be closing down. All of the pet pages with the photos, diaries, and associated comments will disappear. People are understandably upset because they don't know how to save all of those memories. Some are printing the pages off, some are resorting to screenshots.  And some are just at a loss.

But I think I know a better way. And I'll show you how to do it yourself. Step by step.  When you are done, you'll have all of your stuff for your dog.  It won't copy forum discussions and stuff like that.  But it will copy all of your diary entries, all of your photos, and all of the comments on that stuff.  And it will store all that away on your local hard drive.

Ready to go?  Good.  Start with...

Read This Whole Thing Through

Like a recipe, I highly recommend reading this through once or twice to get some idea of what you'll be doing.  It does assume you know your way around your computer a little bit, but you don't need to be a guru or anything like that.  If you are comfortable installing a program on your computer and navigating around the various files on your PC, you should be just fine.

Download HTTRack

HTTrack bills itself as a website copier.  Click on this link, download the appropriate file for your PC, and install it on your computer.  There are a bunch of different versions for the various flavors Windows and whether you are running the 32-bit or 64-bit version, so choose wisely.  You'll be out of luck if you are on a Mac (sorry) unless you know how to compile your own programs.  There looks to be versions for various Linux distributions though for the 0.001% of Dogster users on that operating system.

During the installation process, the program will ask you if you want to put an icon on the Desktop and in the Start Menu.  Let it do so.

Start WinHTTrack

Click the icon the installer put on your Desktop.  The program should start up, displaying one of the ugliest user interfaces in existence.  That's OK.  We aren't using it for its looks.  When you get the screen below, hit the "Next >" button to continue.


Start a New Project

We start out by defining a new project as follows:
  • New project name: This is what the program will name the directory where it stores the files you will download.  I'll be using Abby's Dogster page as an example, so I called my project "AbbyOnDogster".  Use whatever name you like here.
  • Project category: Just leave this blank
  • Base path: If you were going to download multiple Dogster pages, then all of them would go into this directory.  You can use the default "C:\My Web Sites" (it works fine) or specify any other folder you like.

Once you're entered that information, hit the "Next >" button to continue.

Specify the Site To Mirror

So far so good, but now it starts getting a little trickier.  Fill out the next window as follows.
  • Action: Stick with the default of "Download web site(s)"
  • Web Addresses: (URL): The best way to fill this in is to navigate to the Dogster page you want to clone, select the address area, copy it (Control-C), and paste it (Control-V).  Or just type it in from scratch.  The window below shows the address for Abby's page.  It will look something like http://www.dogster.com/dogs/590841.  Click that link to see what I mean.
  • The tricky part is coming up.  Do not hit the "Next >" button yet.  Instead, select the "Set options..." button above it.

Specify the Options

(This Section updated February 5th and then on February 8th to fix the Scan Rules as mentioned at the top of this post).

When you hit the "Set options...." button, there will be a bunch of tabs on top.  Click on the tab that says "Scan Rules".  Don't worry about any of the other tabs.  See the area I have circled in blue?  Click your mouse in that area and delete every single character in there.  You want it to be perfectly empty.  Do not hit the "Exclude link(s)..." button.  Do not hit the "Include link(s)..." button.


Now, copy the four thirteen all of the lines of text below and paste it into the now empty area that has the blue circle above.  Cut and paste is your friend here.
-*
+www.dogster.com/dogs/590841
+www.dogster.com/dogs/590841/*
+www.dogster.com/subscr/gift_archive.php?pet_id=590841
+www.dogster.com/static/*
+www.dogster.com/files*
+www.dogster.com/images/*
+badge.dogster.com/*
+files.dogster.com*
+www.dogster.com/photos/photo.css
+*.cdnsters.com/*
+tetra.saymedia.com/*
+*amazonaws.com/*
+use.typekit.com/*
+*mytextgraphics.com*
+farm*.staticflickr.com/*
+farm*.static.flickr.com/*
--disable-security-limits
--max-rate 1500000
--assume php=text/html
Note that there is now a * after "+files.dogster.com".  This is a change from my first and second cracks at this.  Don't miss that one.

What if you have a cat?  Other than having to vacuum hairballs off the carpet on a regular basis, you want to use this set of scan rules instead.  Abby was not a cat, but work with me here.  Again, cut and paste is your friend.
-*
+www.catster.com/cats/590841
+www.catster.com/cats/590841/*
+www.catster.com/subscr/gift_archive.php?pet_id=590841
+www.catster.com/static/*
+www.catster.com/files*
+www.dogster.com/files*
+www.catster.com/images/*
+badge.catster.com/*
+files.catster.com*
+files.dogster.com*
+www.catster.com/photos/photo.css
+*.cdnsters.com/*
+tetra.saymedia.com/*
+*amazonaws.com/*
+use.typekit.com/*
+*mytextgraphics.com*
+farm*.staticflickr.com/*
+farm*.static.flickr.com/*
--disable-security-limits
--max-rate 1500000
--assume php=text/html
As explained up top, there are lines for both "+files.catster.com*" and "+files.dogster.com*" because Catster's images are actually stored on Dogster's servers.

This might look like Greek to you, but every single character here is important.  Make sure that these four thirteen lines get copied EXACTLY into the area circled in blue above.  That first line is indeed a minus sign followed by an asterisk.  All the other lines start with a plus.  All of this is important.

Or almost all of it.  The "+*mytextgraphics.com/*" won't be necessary for most people and could be safely left out.  One of our friends had a link to this website for an animated image of her pets name so this had to be included to get that to show up.  Others had images from Flickr (+farm*.staticflickr.com/*) and Photobucket (+*.photobucket.com/*).  Again, you can leave these out if your page doesn't link to these sites, but leaving them in there doesn't hurt anything either.  You would have to do something similar for any other off-Dogster sites that you link to for special images like that.  Just be sure to start the name with a plus and follow the format I've used.

You should end up with something that looks exactly like this when you are done.  Unfortunately the Scan Rules list is kind of long now so stuff is getting cut off on the bottom of the list.  But you get my drift.

Now if we were to proceed on from this point, it would copy Abby's page on Dogster because her Dogster ID number is 590841.  What you need to do now is replace the number 590841 with your own dog's ID number in all three places: the second, third, and fourth lines of the list.  Your dog's ID number is the one that shows up in the address bar of your web browser whenever you go to your dog's pet page.  It is usually six or seven digits long.  Do not add or subtract asterisks, slashes, or plus signs, or anything else unless you really know what you are doing.  Just change the number.  Get this wrong and you could end up downloading half of YouTube.  Ask me how I know...

Once you have changed 590841 to your own dog's ID number, hit the "OK" button at the bottom of the window and pat yourself on the back.  The hard part is done.

After hitting the OK button in the window above, you'll be back to the window that offered you the "Set Options" button in the first place.  You've already done that, so you can hit the "Next" button to Continue.


Breeze Past Connection Parameters

The next screen you'll see is shown below.  This is just getting fancy now.  You shouldn't have to worry about anything on this window, so hit the "Finish" button at the bottom.  And cross your fingers, because once you hit "Finish" the download will begin.


Watch the Download

Here is where it gets exciting.  The program is now going to go out and fetch your Dogster page, with all the associated diary entries, photos, and comments.  It is going to store them on your hard drive and change all the links around so that everything works on your local computer.  What it won't do is  create a copy of all the forum discussions or download all of your friends pages.  It gets your stuff for your dog.  And that is the most important thing.


How long this part takes depends on how much stuff you put on Dogster in the first place and the speed of your Internet connection.  Our connection isn't that great but there is a fair bit of stuff on Abby's page, and it still took around five fifteen minutes (the bigger set of Scan Rules from my first crack at this causes a fair bit more stuff to be downloaded).  As it runs, you'll hopefully see some familiar text going by.  The window above has some text that says "...bys_adventures", short for "Abby's Adventures".  This kind of thing told me I was on the right track.

Success!

When your pet's page has been copied, you should see a screen like this. Click "Browse Mirrored Website" and hopefully what you'll be looking at is a copy of your pet's Dogster page.

 

The actual page will be found where you specified it in that very first screen.  In this example, it would be in a place called "C:\My Web Sites\AbbyOnDogster".  That file will have a directory in it called "index.html".  Just click on that anytime you like and you should see a replica of your dog's pet page.  For our example, the address bar should look something like this.


See how it says "file:///C:/My Web Sites...".  That means that this file is on my local hard drive.  Contrast this with what it says when I access it over the web:

If everything went well, you'll have a copy of your dog's pet page tucked away nice and safe on your hard drive.  Back it up to a USB stick, move the directory around, or do whatever you like.  Have more than one dog on Dogster?  Just repeat the process again as many times as you like.  The ID number is the key.  To be on the safe side,it might be best to use a different project name each time.

Please note that I've put together this method and this writeup in a hurry.  There might be some cases where the process doesn't work.  If you run into trouble, leave me a message in the comments section and I'll try to help you out.  I'll also try to update this page based on people's feedback to clarify any spots where I haven't been clear.

Now I know that not everybody is going to be computer-savvy enough to do this.  But some will be.  All I ask is that those who can figure it out pay it forward and help out those who can't.  Back up their page, zip up the file, and email it to them.  They'll be incredibly grateful, and you'll feel good for helping them out.

Update - January 25th, 2014

First of all, I've been made aware of a couple limitations of this technique:
  • the link to the "Rosette, Star, and Special Gift History" doesn't work
  • clicking on those items from the main page doesn't give the pop-up message from the sender
  • the link to "See all my pup pals" doesn't work
  • the link to the video page doesn't work and the videos aren't downloaded
Now, I am no web guru but it looks like all these things have something in common: the links are not "static" but are dynamically generated when the link is clicked.  That is something that the web mirroring software used in this technique can't handle.  Other web mirroring software is similarly handicapped.

I was asked in one of the comments about videos and, while I can't get that to work as part of this process, I have found a workaround that will let you save each of your videos to your computer as separate files in case you no longer have the original version.

Basically, we are going to set up a "flash downloader".  This is generally an add-on to your web browser that lets you save the videos that you normally don't have an option to save.  The videos on Dogster are in something called "flash video" format, so you'll need a "player" for that too once the file has been downloaded.  I get into that as well.

Other methods besides the one I have tested below are bound to work and you are free to give this a shot with a different browser or technique, but I probably won't be able to help you much if you veer off this beaten path.  And this method should work for everybody on any operating system.

Want to get your videos back?  Here is how.

Step 1: Download Firefox If You Don't Already Have It

Why Firefox?  Because that is what I use, it worked for me, it is awesome, and it is free.  If you already have Firefox, you are awesome and can skip this step.  If you don't, download it from this link and become awesome.  Go ahead.  I'll wait.

Step 2: Open This Post in Firefox

If you are already using Firefox, skip this step as well.  If not, copy the address link for this post from your current browser to Firefox

Step 3: Get the Download Flash and Video Add-On

Like other browsers, Firefox's functionality can be extended with things called Add-Ons.
Install the Download Flash and Video Add-On from this link and clicking the "Add to Firefox" button that shows up in big letters on something like the snapshot below.

Firefox might give you a big scary warning like the one below.  No worries here.  There will be a number that counts down on the greyed out "Install" button on a window...

After a few seconds, the button will change to say "Install now" and it will no longer be greyed out.  Click on "Install Now".  The add-on will install and Firefox will tell you that it will be enabled when you restart Firefox.  Select "Restart Now".  Firefox should restart and bring you back to this page.

Step 4: Show Add-On Bar

Firefox Add-Ons are shown by default on an Add-On bar that is at the bottom of your browser window.  If you don't see a little box with a down arrow in the bottom right hand corner of your browser, hit the "Control" and "/" keys at the same time to make it appear, like so (I've enlarged it to make it easier to see).

Step 5: Get Your Video

So far, so good.  Next thing you want to do is open up your Dogster page and click on the link to open the page showing all of your videos using the "See my Video Book" link on the left side of your pet page.  What you are going to do is...
  • Start the video
  • Pause the video once it starts playing  (it seems to be important that you do this).
When you do this, you'll see the little icon in the bottom right change color.  Click on the icon and you should see something like the window below.  Ignore the "Flash Files to Download" on top.  Click on the video title under "Videos to Download" and the file should download to your computer!

I expect that the name of the downloaded file will look like some random collection of letters and numbers.  You'll certainly want to rename it to something that is understandable.

Step 6: Play the Video

I won't go into too much detail here.  Some computers might already have the software installed to play these videos.  Try opening / double clicking the file to see if it plays for you.  If it does, you are all set and can proceed with downloading all of your other videos.  If not, you'll want to install a piece of software to do so.  I recommend VideoLAN, as it is free, fantastic, and can play just about any kind of video file you throw at it.  It also runs on just about any computer out there.  Get it from this link.

          

Good luck retrieving the memories of your dog and those of your friends.  Mine are incredibly important to me.