ntang (ntang) wrote,

  • Mood:

Nicholas and the Terrible, Horrible, No Good, Very Bad Morning

So, as mentioned last night, I've been up since 1 am, give or take. Give or take what? I don't know, have some walnuts*.

At around 5 am I hopped in the shower, got ready, and took off. I got into the city around 6:30 or so, got on the subway, and rode down to Globix. So far, so good.

Got to Globix at around 6:45, and E (the new SA) was waiting for me, she took a cab from Brooklyn and beat me. Oh, did I mention it was cold and raining?

We went into the cage and got to work. Everything went smoothly, more or less - we shut down the 2nd netapp, took off the disk shelf, moved a couple of disks from that shelf to one of the 2 remaining shelves, and then shut down the 1st netapp and installed the new shelf on it. So far so good. We plug it in, turn it on, it comes up ok, but then we notice a failed disk. It automatically fails onto one of the spares, and starts repairing the volume. We yank a spare out of netapp2, plug it in netapp1, and add 3 disks (leaving 1 spare in the shelf). Then we wait for it to finish growing the volume.

While we wait, several machines commit suicide for our amusement, including one of our big expensive sun boxes which happened to be one of our primary db's. No big deal. :P Oh, and while rebooting one box, it randomly ejected the cd tray. I think it was sticking its tongue out at me.

There was a bit of a mess on the netapp; it turned out it was set to autonegotiate and the switch to 100-full, and my boss found and fixed that while we worked on bringing the db server back.

I guess it doesn't sound all that bad, but we were there for almost 4 hours working on stuff, waiting for volumes to rebuild and recover, having multiple machines die... it was ugly.

Oh, and on the way back due to a police investigation at 42nd the trains were all fscked up so we had to walk over to another train to get up here. Not a big deal but worth noting.

* - Nonsensicality comes easily when sleep has not.

  • Where I am nowadays

    I haven't updated this in a million years... in case you're wondering why, it's because I've mostly moved on to other places. You can find my…

  • DSL

    I've been a loyal Megapath customer for years. (Something like 8 or 10, crazy, in that range...) They've had great service (and a great service -…

  • MySQL failover

    So we're running some MySQL at work, which is a little unusual for us, but is probably long overdue. (Specifically, it's for some Wordpress…

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded