Wherein the author re-establishes the audience of this journal
- dishwasher detergent ++
- diapers (1 bag) +
- wipes (2-3 containers) ++
- dryer sheets (2 boxes?) ++
- laundry detergent +
- baby food +
- cheese +
- ham (or other deli meats) +
- (damnit I forget the other thing(s)) --
One of the DBA's noticed some problems with one of the dev DB's. So my boss asked me to take a look at it.
It's running solstice disk suite. 2 scsi disks appear to be dead. The metadb info is all corrupted. The disks were all striped/concat'ed, no mirroring or raiding. The gui tool is totally confused and reports that half of the slices are empty and that the others are huge... according to it we've got over 1TB of disk on there (we have ~240G) and it's almost all on unused slices. It also says every single resource is in a critical state. The command line is a bit more sane, and it reports accurate looking figures, but it also seems to say there are no problems. I don't think any backups were being done on this box.
The only saving grace? There's a spare 9G HD that isn't on either of the external controllers, and it's not being used by disksuite. So, we can probably export the oracle data to that disk and then I can replace the failed disks, and then (fun fun) rebuild it all from scratch and/or try to recover it.
SIGH. Today's going to be jam-packed with excitement. :P
We had a few webservers down... one had been down since 8 pm last night and no one did anything about it. We did actually get emails, and the on-call admin did get pages... but...
So, understandably, the boss is pissed. We all should've been more up on things, especially the on-call admin. Gotta start _reading_ those Netsaint alerts I guess. ;)