SDSrecover

From My Admin Page
Jump to: navigation, search

Repairing Corruption on a DiskSuite File System

(Copied from here)

The procedure described here is provided by Sun in order to undo DiskSuite, restore affected file systems from backups, and redo DiskSuite. We adapt this procedure to run fsck safely on bad file systems.

Why is this needed? If a file system cannot be unmounted (e.g. root file system), but you need to fsck it, then you need to boot from CD-ROM. But the OS on the CD-ROM has no DiskSuite drivers, so it can’t understand DiskSuite metadevices. You should never fsck the underlying physical slices of a DiskSuite metadevice (e.g. mirrors round-robin, and fscking their underlying slices can put them out of synch). Thus, you must undo DiskSuite first before fscking such file systems, and re-do DiskSuite afterwards.

In our cases, the root file system was okay, but others were corrupted. To fix the corruption, the outline is:

  • Boot from CD-ROM.
  • Undo DiskSuite following Steps 5 through 7 of the following procedure from Sun.
  • Fsck the underlying slices on whichever of the mirrored disks you’re going to consider to be your “good” one
  • Repeat fsck until all slices fsck without errors. If fsck finds a problem and repairs it, fsck again until no errors are found!
  • Continue with the procedure from Sun at Step 8 to re-do DiskSuite.


NOTES:

  • The following document is unaltered except for formatting.
  • At Step (8), boot single user first, and set a root password. Recall that our standard practice is to have no root password in /etc/passwd or /etc/shadow, but you won’t be able to become root using suw after logging in as yourself when you’re booting single-user and leaving other file systems (e.g. /u) unmounted. So you must set a root password and then log in as root.
  • Step (10) talks about restoring contents of md.tab. This file contains only comments, even on live systems. And the example they show is not actually md.tab; it matches with what is found in md.cf.
  • Rather than using md.cf and running metainit to automatically recreate all the DiskSuite metadevices as per Step 11, I recreated all the DiskSuite metadevices (including metadb replicas) “by hand” using documentation we typically stash in the local_<hostname>.uwaterloo.ca package about the machine’s DiskSuite configuration. So I can’t confirm whether or not the automatic approach works as documented.
* Resolution *  	*Top*
http://sunsolve.central.sun.com/search/document.do?assetkey=1-25-14650-1

NOTE: In order to restore the machine exactly the way it was, it is necessary
to have a record of the locations of the state databases, and also the
configuration of the metadevices.   These are *not* held in user-readable form
on the root filesystem, so it is useful to get the information BEFORE a crash
with the commands:

  # metadb -i
  This command shows the condition and location of DiskSuite's state
  database replicas. It is useful to have this information in case their
  recreation is necessary.

  # metastat -p
  You will redirect this output to a file for safekeeping. This
  information is most often sent to the /etc/opt/SUNWmd/md.tab file for
  disaster recovery purposes (see step 6 below). Check the md.tab file prior
  to overwriting it, though. If a valid one exists, redirect the metastat -p
  to another filename and run a 'diff' on it to check for inconsistencies.
  The metastat -p, however, will reflect the most current, known, configuration.

Of course, you will need to create a new root disk and restore the last
full backup to that new disk.   It is expected that you will partition
the disk in much the same manner as you had it before.   Remember that
if there was a small partition on your original disk for a state
replica that you do the same thing for this disk.

Once the disk has been created, restore your root filesystems from
backups.   However, before attempting to boot the system after the
restore, we must make some necessary changes to let Solaris know that
the root disk is no longer mirrored.   To do this, the /etc/system and
the /etc/vfstab files must be modified.                                                                                         

1) Boot cdrom, and mount the root filesystem to /a.

     ok boot cdrom -s
           # mount /dev/dsk/c#t#d#s# /a

2) Restore the root filesystem from backup tape into /a and initialize
   the boot block using

       # installboot /usr/platform//lib/fs/ufs/bootblk /dev/rdsk/c#t#d#s#

3) Mount and restore any other critical filesystems such as /usr /var /opt etc.

4) If the replacement machine does not have it disks connected via the same
   paths as the original then you may have to follow the procedure documented in
   SRDB 15010 to rebuild the /devices and /dev structures.

5) vi /a/etc/system (before using vi you may need to set the terminal
   type e.g. TERM=sun; export TERM)

     Remove ALL lines between the "MDD root info" lines as well as those
     between the "MDD database info" lines.   In the following example
     file, all these lines would be removed from the file:

       -------------------------------------------------------------
       * Begin MDD root info (do not edit)
       forceload: misc/md_trans

       forceload: misc/md_raid
       forceload: misc/md_hotspares
       forceload: misc/md_stripe
       forceload: misc/md_mirror
       forceload: drv/sd
       forceload: drv/esp
       forceload: drv/espdma
       forceload: drv/sbus
       forceload: drv/iommu
       rootdev:/pseudo/md@0:0,3,blk
       * End MDD root info (do not edit)* Begin MDD database info (do not edit)
       set md:mddb_bootlist1="sd:14:16 sd:15:16"
       * End MDD database info (do not edit)
       -------------------------------------------------------------

6) vi /a/etc/vfstab

     Change all metadevices for the root filesystems (root, usr, var, and
     opt) back to regular slices (/dev/dsk/c#t#d#s#).   Comment out all
     the other metadevices for the time being.   For example:

   ** EXAMPLE FILE BEFORE:
   ---------------------------------------------------------------------------
   #device         device           mount         FS    fsck mount     mount
   #to mount       to fsck          point         type  pass at boot options
   #
   /proc           -                /proc         proc  -    no          -
   fd              -                /dev/fd       fd    -    no          -
   swap            -                /tmp          tmpfs -    yes         -
   /dev/md/dsk/d1  -                -             swap  -    no          -
   /dev/md/dsk/d0  /dev/md/rdsk/d0  /             ufs   1    no          -
   /dev/md/dsk/d2  /dev/md/rdsk/d2  /usr          ufs   1    no          -
   /dev/md/dsk/d3  /dev/md/rdsk/d3  /var          ufs   2    yes         -
   /dev/md/dsk/d4  /dev/md/rdsk/d4  /opt          ufs   2    yes         -
   /dev/md/dsk/d5  /dev/md/rdsk/d5  /export/home  ufs   3    yes         quota
   /dev/md/dsk/d6  /dev/md/rdsk/d6  /export/home1 ufs   3    yes         quota

   ---------------------------------------------------------------------------

   ** EXAMPLE FILE AFTER:
   ---------------------------------------------------------------------------
   #device           device             mount         FS    fsck mount    mount
   #to mount         to fsck            point         type  pass at boot options
   #
   /proc             -                  /proc         proc  -    no      -
   fd                -                  /dev/fd       fd    -    no      -
   swap              -                  /tmp          tmpfs -    yes     -
   /dev/dsk/c0t0d0s1 -                  -             swap  -    no      -
   /dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 /             ufs   1    no      -
   /dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /usr          ufs   1    no      -
   /dev/dsk/c0t0d0s7 /dev/rdsk/c0t0d0s7 /var          ufs   2    yes     -
   /dev/dsk/c0t0d0s4 /dev/rdsk/c0t0d0s4 /opt          ufs   2    yes     -
   #/dev/md/dsk/d5   /dev/md/rdsk/d5    /export/home  ufs   3    yes     quota
   #/dev/md/dsk/d6   /dev/md/rdsk/d6    /export/home1 ufs   3    yes     quota
   ---------------------------------------------------------------------------

7) Remove all lines (except for the 2 comment lines at the top) from
     the "mddb.cf" file.   This file exists either in the /etc/opt/SUNWmd
     or /etc/lvm directory, depending on the version of DiskSuite you are
     running.

 8) Boot the system from the newly restored boot disk.   When the system
     comes up, only root (and /usr, /var, and /opt, if they exist) will be
     mounted using the slices on the new root disk.                                                                       

9) Re-add the state databases with the 'metadb' command.   Use the
     output of the 'metadb' command you saved to get the locations.

               metadb -a -f
               metadb -a -f
               metadb -a -f 

10) Recreate the "md.tab" file from the 'metastat' output you saved (or
     from memory), except make sure that all mirrors to be one-way
     mirrors and all RAID5 devices contain the "-k" option.   For the boot
     disk, ensure that the one-way mirrors refer to the side which has
     been restored.   Make sure the order is correct so that mirrors aren't
     created until the submirrors already exist. See below the necessary
     changes that must be made to ensure one-way mirrors are used.

     The "md.tab" file exists either in the /etc/opt/SUNWmd or /etc/lvm
     directory, depending on the version of DiskSuite you are running.

     For example:

           vi md.tab

           ** EXAMPLE FILE BEFORE:
           -------------------------------------------------
           d0 -m d10 d20                       <-- see the two-way mirror!
           d10 1 1 c0t0d0s0
           d20 1 1 c1t0d0s0
           d1 -m d11 d21
           d11 1 1 c0t0d0s1
           d21 1 1 c1t0d0s1
           d2 -m d12 d22
           d12 1 1 c0t0d0s6
           d22 1 1 c1t0d0s6
           d3 -m d13 d23
           d13 1 1 c0t0d0s7
           d23 1 1 c1t0d0s7
           d4 -m d14 d24
           d14 1 1 c0t0d0s4
           d24 1 1 c1t0d0s4
           d5 -m d15 d25
           d15 4 c0t1d0s2 c0t2d0s2 c1t1d0s2 c1t2d0s2
           d25 4 c0t3d0s2 c1t3d0s2 c1t4d0s2 c1t5d0s2
           d6 -r c1t6d0s2 c1t8d0s2 c1t9d0s2
           -------------------------------------------------

           ** EXAMPLE FILE AFTER:
           -------------------------------------------------
           d10 1 1 c0t0d0s0
           d20 1 1 c1t0d0s0
           d0 -m d10                               <-- see the one-way mirror!
           d11 1 1 c0t0d0s1
           d21 1 1 c1t0d0s1
           d1 -m d11
           d12 1 1 c0t0d0s6
           d22 1 1 c1t0d0s6
           d2 -m d12
           d13 1 1 c0t0d0s7
           d23 1 1 c1t0d0s7
           d3 -m d13
           d14 1 1 c0t0d0s4
           d24 1 1 c1t0d0s4
           d4 -m d14
           d15 4 c0t1d0s2 c0t2d0s2 c1t1d0s2 c1t2d0s2
           d25 4 c0t3d0s2 c1t3d0s2 c1t4d0s2 c1t5d0s2
           d5 -m d15
           d6 -r c1t6d0s2 c1t8d0s2 c1t9d0s2 -k
           -------------------------------------------------

11) Run the 'metainit' command to create all the metadevices that are
     listed in the "md.tab" file.

               # metainit -f -a

12) Run the 'metaroot' command to set the metadevice as a root device.

               # metaroot d0

13) Add in the other metadevices into the /etc/vfstab file for the
     other root filesystems as well as the other metadevices that you
     created in step 7.

14) Reboot.

15) You will now attach all the second mirrors to all the mirrored
    metadevices. For example:
	# metattach d0 d20