Links

Before I’m ready to trust ZFS I need to make sure I can replace a disk when it dies. With the setup described here, as a first experiment I removed the primary disk.

So, power down and remove the primary disk (ad4). Note that if you’re doing this on the Proliant system I mentioned, then you really should replace the drive mount (it is needed for cooling). Luckily I have a spare system so I just borrowed one.

Reboot. Comes up fine on the secondary disk without further intervention.

$ zpool status
  pool: scratch
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        scratch         ONLINE       0     0     0
          gpt/scratch8  ONLINE       0     0     0

errors: No known data errors

  pool: system
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: none requested
config:

        NAME             STATE     READ WRITE CKSUM
        system           DEGRADED     0     0     0
          mirror         DEGRADED     0     0     0
            gpt/system8  ONLINE       0     0     0
            gpt/system4  UNAVAIL      0     0     0  cannot open

errors: No known data errors

Note that the system pool is now degraded. How would we have known if we hadn’t checked? Well, turns out we missed something from the previous setup.

We should have put

daily_status_zfs_enable="YES"
daily_status_gmirror_enable="YES"


in /etc/periodic.conf. Then in the daily mail we’d see:

Checking status of zfs pools:
  pool: system
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: none requested
config:

	NAME             STATE     READ WRITE CKSUM
	system           DEGRADED     0     0     0
	  mirror         DEGRADED     0     0     0
	    gpt/system8  ONLINE       0     0     0
	    gpt/system4  UNAVAIL      0     0     0  cannot open

errors: No known data errors

Checking status of gmirror(8) devices:
       Name    Status  Components
mirror/swap  DEGRADED  gpt/swap8

So remember, boys and girls, read your daily mails!

So far, so good. One disk failed, the system came back up without intervention, and would have alerted us in daily mails had we configured it correctly (of course it now is). So what happens if we put the disk back in? Since we’ve modified the other disk in the meantime, we’d hope that would get reconciled. Let’s see…

Power down and replace the missing disk, reboot.

Now we see

$ zpool status
  pool: scratch
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        scratch         ONLINE       0     0     0
          gpt/scratch8  ONLINE       0     0     0

errors: No known data errors

  pool: system
 state: ONLINE
 scrub: resilver completed after 0h0m with 0 errors on Sat Mar 26 10:48:56 2011
config:

        NAME             STATE     READ WRITE CKSUM
        system           ONLINE       0     0     0
          mirror         ONLINE       0     0     0
            gpt/system8  ONLINE       0     0     0
            gpt/system4  ONLINE       0     0     0  345K resilvered

errors: No known data errors

$ gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gpt/swap4
                       gpt/swap8

and there we are, back to where we started. But suppose the disk had really failed, then what? See the next exciting installment!

Share This

This entry was posted on Sunday, March 27th, 2011 at 16:12 and is filed under Open Source. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.