Ambrosia

From Hyades
Jump to: navigation, search

Ambrosia (hostname: ambrosia.ucsc.edu) is a file server for the Hyades cluster, acquired in November 2013. Its hardware specifications are as follows:

IP Addresses
Subnet IP Address Netmask Gateway
Public 10GbE 128.114.126.229 255.255.255.224 128.114.126.238
Private GbE 10.6.7.1 255.255.0.0  
Private 10GbE 10.7.7.1 255.255.0.0  
IPMI 10.9.7.1 255.255.0.0  

Installing FreeBSD on a mirrored ZFS root

We installed FreeBSD 9.2, in November 2013, on a root ZFS filesystem created on the two Intel SSDs. We closely followed the instructions in the FreeBSD wiki article Installing FreeBSD 9.0 (or later) Root on ZFS using GPT.

Note3.png
Notes:
  1. The instructions in Installing FreeBSD Root on ZFS (Mirror) using GPT are only applicable to FreeBSD 8.x; thus are outdated;
  2. There is no support for booting FreeBSD on a ZFS root under UEFI yet. So we'll boot under legacy BIOS; but we'll use GPT instead of MBR.

Downloading FreeBSD 9.2

Download FreeBSD-9.2-RELEASE-amd64-memstick.img (the memory stick image of FreeBSD 9.2 for x86_64) from ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.2/

Write the image to a USB stick. On my iMac (running OS X Maverick 10.9), I did the following:

$ diskutil list
$ diskutil umount /dev/disk2s2
$ sudo dd if=/dev/zero of=/dev/disk2 bs=64k count=10
$ sudo dd if=FreeBSD-9.2-RELEASE-amd64-memstick.img of=/dev/disk2 bs=64k

Creating a bootable ZFS filesystem

Boot FreeBSD from the USB stick.

Go through the initial setup as usual[1].

When the partitioning dialogue in bsdinstall comes up, choose the Shell option.

Create new GPT (GUID Partition Table) on the two Intel SSDs:

# camcontrol devlist
<INTEL SSDSA2CW120G3 4PC10362>     at scbus1 target 0 lun 0 (ada0,pass0)
<INTEL SSDSA2CW120G3 4PC10362>     at scbus2 target 0 lun 0 (ada1,pass1)
# gpart destroy -F ada0
# gpart destroy -F ada1
# gpart create -s gpt ada0
# gpart create -s gpt ada1

Add partitions for the boot loader and swap then install the protective MBR and gptzfsboot boot loader. All partitions are aligned to 4k for optimal performance with advanced format drives.

# gpart add -s 222 -a 4k -t freebsd-boot -l boot0 ada0
# gpart add -s 8g -a 4k -t freebsd-swap -l swap0 ada0
# gpart add -a 4k -t freebsd-zfs -l disk0 ada0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

# gpart add -s 222 -a 4k -t freebsd-boot -l boot1 ada1
# gpart add -s 8g -a 4k -t freebsd-swap -l swap1 ada1
# gpart add -a 4k -t freebsd-zfs -l disk1 ada1
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
Note3.png
Note:
The Intel 320 Series SSDs don't appear to be advanced format drives:
# camcontrol identify ada0 | grep "sector size"
sector size           logical 512, physical 512, offset 0
but using a 4k sector size probably doesn't hurt.

Create virtual devices with 4k sectors, to ensure ZFS uses the correct block size:

# gnop create -S 4096 /dev/gpt/disk0
# gnop create -S 4096 /dev/gpt/disk1

Load the necessary kernel modules:

# kldload zfs

Create the ZFS pool:

# zpool create -o altroot=/mnt -O canmount=off -m none zroot mirror /dev/gpt/disk0.nop /dev/gpt/disk1.nop

This will create a zpool called zroot which will not be mounted. This zpool is only used to derive other file systems from.

Installing FreeBSD to the ZFS filesystem

Create ZFS filesystem hierarchy:

# zfs set checksum=fletcher4 zroot
# zfs set atime=off zroot

# zfs create -o mountpoint=none zroot/ROOT
# zfs create -o mountpoint=/ zroot/ROOT/default

# zfs create -o mountpoint=/home -o setuid=off zroot/home
# zfs create -o mountpoint=/tmp -o compression=lz4 -o setuid=off zroot/tmp
# chmod 1777 /mnt/tmp

# zfs create -o mountpoint=/usr zroot/usr
# zfs create zroot/usr/local
# zfs create zroot/usr/obj
# zfs create -o compression=lz4 -o setuid=off   zroot/usr/ports
# zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
# zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
# zfs create -o compression=lz4 -o exec=off -o setuid=off zroot/usr/src

# zfs create -o mountpoint=/var zroot/var
# zfs create -o compression=lz4 -o exec=off -o setuid=off zroot/var/crash
# zfs create -o exec=off -o setuid=off zroot/var/db
# zfs create -o compression=lz4 -o exec=on -o setuid=off zroot/var/db/pkg
# zfs create -o exec=off -o setuid=off zroot/var/empty
# zfs create -o compression=lz4 -o exec=off -o setuid=off zroot/var/log
# zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
# zfs create -o exec=off -o setuid=off zroot/var/run
# zfs create -o compression=lz4 -o exec=on -o setuid=off zroot/var/tmp
# chmod 1777 /mnt/var/tmp
Note3.png
Notes:
  1. The fletcher4 checksum algorithm is more robust than the old default fletcher2 algorithm.
  2. Set atime off to avoid writing a metadata change every time a file is accessed, a serious performance penalty.
  3. Compression may be set to on, off, lzjb, gzip, gzip-N (where N is an integer from 1 (fastest) to 9 (best compression ratio); gzip is equivalent to gzip-6).
  4. On FreeBSD 8.4 and 9.2 or later, lz4 compression is also supported, providing the best trade-off (significantly faster compression and decompression with moderately higher compression ratios).
  5. During installation, the new root file system is mounted under /mnt.

Set the dataset to boot from:

# zpool set bootfs=zroot/ROOT/default zroot

Add the swap devices to fstab, so that they will automatically show up when the system starts:

# cat << EOF > /tmp/bsdinstall_etc/fstab
# Device        Mountpoint  FStype  Options  Dump  Pass#
/dev/gpt/swap0  none        swap    sw       0     0
/dev/gpt/swap1  none        swap    sw       0     0
EOF

Once all of the ZFS filesystems have been created, type exit in the shell and proceed with the installation as normal. When prompted to 'Choose the services you would like to be started at boot', I chose sshd, ntpd, & powerd.

Once the installation is complete, choose Exit from the main menu.

The next dialogue will offer the option to 'open a shell in the new system to make any final manual modifications'. Select Yes.

Configure ZFS:

 
# mount -t devfs devfs /dev
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf

Set read only on /var/empty, which is supposed to be empty at all times:

# zfs set readonly=on zroot/var/empty

To finish the installation, exit the shell, remove the USB stick and choose the Reboot option from the next dialogue.

Creating ZFS filesystems on the SAS HDDs

The next step was to create ZFS filesystems on the 36 4TB SAS hard drives. Those drives are controlled by a hardware RAID controller, namely a LSI MegaRAID SAS 9271-8i 6Gb/s SAS and SATA RAID Controller Card:

# mfiutil show adapter
mfi0 Adapter:
    Product Name: LSI MegaRAID SAS 9271-8i
   Serial Number: SV34127542
        Firmware: 23.18.0-0014
     RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
  Battery Backup: present
           NVRAM: 32K
  Onboard Memory: 1024M
  Minimum Stripe: 8k
  Maximum Stripe: 1M

However, ZFS prefers to manage the drives directly by itself, as it offers software RAID through its RAID-Z and mirroring organization schemes. We worked around it by creating a RAID0 volume for each drive.

List all hard drives controlled by the LSI MegaRAID card:

# mfiutil show drives

Create a RAID0 volume for each of 24 drives in the front of the chassis:

# mfiutil create jbod -v e1:s0 e1:s1 e1:s2 e1:s3 e1:s4 e1:s5 \
    e1:s6 e1:s7 e1:s8 e1:s9 e1:s10 e1:s11 \
    e1:s12 e1:s13 e1:s14 e1:s15 e1:s16 e1:s17 \
    e1:s18 e1:s19 e1:s20 e1:s21 e1:s22 e1:s23

which created 24 RAID0 volumes mfid0 – mfid23.

Create a RAID0 volume for each of 12 drives in the rear of the chassis:

# mfiutil create jbod -v e2:s0 e2:s1 e2:s2 e2:s3 e2:s4 e2:s5 \
    e2:s6 e2:s7 e2:s8 e2:s9 e2:s10 e2:s11 \

which created 12 RAID0 volumes mfid24 – mfid35.

Create a zpool with RAIDZ-2 on the 24 front drives:

# zpool create -m none ztrove raidz2 \
    mfid0 mfid1 mfid2 mfid3 mfid4 mfid5 \
    mfid6 mfid7 mfid8 mfid9 mfid10 mfid11 \
    mfid12 mfid13 mfid14 mfid15 mfid16 mfid17 \
    mfid18 mfid19 mfid20 mfid21 mfid22 mfid23

Create a zpool with RAIDZ-2 on the 12 rear drives:

# zpool create -m none zhome raidz2 \
    /dev/mfid24 /dev/mfid25 /dev/mfid26 /dev/mfid27 /dev/mfid28 /dev/mfid29 \
    /dev/mfid30 /dev/mfid31 /dev/mfid32 /dev/mfid33 /dev/mfid34 /dev/mfid35

Create ZFS filesystems:

# zfs set checksum=fletcher4 ztrove
# zfs set atime=off ztrove
# zfs create -o mountpoint=/export/trove -o setuid=off ztrove/trove
# chmod 1777 /export/trove

# zfs set checksum=fletcher4 zhome
# zfs set atime=off zhome
# zfs create -o mountpoint=/export/home -o setuid=off zhome/home
# chmod 1777 /export/home

Network Configuration

Add the following stanza to /etc/rc.conf:

hostname="ambrosia.ucsc.edu"
ifconfig_ix0=" inet 128.114.126.229 netmask 255.255.255.224 -tso"
ifconfig_ix1=" inet 10.7.7.1 netmask 255.255.0.0 mtu 9000 -tso"
ifconfig_igb0=" inet 10.6.7.1 netmask 255.255.0.0 -tso"
defaultrouter="128.114.126.238"

Note due to a device driver bug for the Intel 10GbE adapter in FreeBSD 9.2, I turned off TSO (TCP Segmentation Offload). Ambrosia has since been upgraded to FreeBSD 9.3 and the bug may have been fixed.

Apply the settings to the networking system:

# service netif restart
# service routing restart

or:

# /etc/rc.d/netif restart && /etc/rc.d/routing restart

NFS

Configuring NFS Server

To enable starting NFS server at boot time, add these options to /etc/rc.conf[2]:

rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 256"

Ambrosia will serve NFS in both UDP and TCP transports using 256 daemons (nfsd -u -t -n 256).

Start the NFS server:

# service nfsd start

Exporting the ZFS filesystems over NFS

The goal was to export the ZFS filesystems, /export/home & /export/trove, over NFS to both of the private subnets in the Hyades cluster:

  1. to private GbE subnet 10.6.0.0/16, with root_squash
  2. to private 10GbE subnet 10.7.0.0/16, with no_root_squash

One can use the zfs set sharenfs command to export ZFS filesystems over NFS. However, there is no clean way to export a ZFS filesystem to 2 subnets using the command. I use the following hack.

Export the ZFS filesystems to the private 10GbE subnet:

# zfs set sharenfs="-maproot=root -network=10.7.0.0/16" zhome/home
# zfs set sharenfs="-maproot=root -network=10.7.0.0/16" ztrove/trove

which produced the following /etc/zfs/exports:

# !!! DO NOT EDIT THIS FILE MANUALLY !!!

/export/home    -maproot=root -network=10.7.0.0/16 
/export/trove   -maproot=root -network=10.7.0.0/16

The shares are instantaneously exported to the private 10GbE subnet, with no_root_squash (-maproot=root). There is no need to reload/restart mountd.

Manually create /etc/exports:

/export/home	-maproot=nobody -network=10.6.0.0/16 
/export/trove	-maproot=nobody -network=10.6.0.0/16

The shares will be exported to the private GbE subnet, with root_squash (-maproot=nobody). However, to make the change to take effect immediately, we have to force mountd to reread /etc/exports:

# service mountd onereload

Let's test it:

# showmount -e
Exports list on localhost:
/export/trove                      10.6.0.0 10.7.0.0 
/export/home                       10.6.0.0 10.7.0.0

ZFS tunings

The combination of ZFS and NFS stresses the ZIL to the point that performance falls significantly below expected levels[3]. Let's disable ZIL on the exported datasets:

# zfs set sync=disabled zhome/home
# zfs set sync=disabled ztrove/trove

Linux NFS clients

There appears to be some interoperability issue between the FreeBSD NFS server and a Linux NFS client! If we mount the exported NFS share /export/home at /home on a client running RHEL/CentOS 6, without the nolock option, a (t)csh login session will hang for a long time, trying to obtain a lock from the server! On the other hand, bash seems to work fine.

On FreeBSD, the NFS file locking daemon, rpc.lockd, provides monitored and unmonitored file and record locking services in an NFS environment. It typically operates in conjunction with rpc.statd, to monitor the status of hosts requesting locks. Those 2 daemons are optional, and disabled by default. Note that a sane NFS server requires at least 3 services running: rpcbind, mountd and nfsd.

However, even after starting the lockd & statd services one the FreeBSD server, if we again mount the home share without the nolock option, a (t)csh login session will still hang for a long time, unable to obtain a lock!

The solution is to use the nolock option to mount the exported NFS shares on Linux. We use the noatime,nosuid,nolock,soft,retrans=16 options to mount /export/home & /export/trove on Linux:

# mkdir /trove
# mount -t nfs -o noatime,nosuid,nolock,soft,retrans=16 10.7.7.1:/export/trove /trove

Combined with the defaults, the above command results in the following mount options:

# grep trove /proc/mounts 
10.7.7.1:/export/trove /trove nfs rw,nosuid,noatime,vers=3,rsize=65536,wsize=65536,namlen=255,soft,nolock,proto=tcp,timeo=600,retrans=16,sec=sys,mountaddr=10.7.7.1,mountvers=3,mountport=758,mountproto=udp,local_lock=all,addr=10.7.7.1 0 0

Firewall

We use the stateful firewall IPFW on Ambrosia[4].

To configure the system to enable IPFW at boot time, add the following to /etc/rc.conf:

firewall_enable="YES"
firewall_script="/etc/ipfw.rules"

Add IPFW ruleset to /etc/ipfw.rules:

#!/bin/sh
# Flush out the list before we begin.
ipfw -q -f flush

# Set rules command prefix
cmd="ipfw -q add"
pif="ix0"     # public network interface

$cmd 00001 deny ip from any to any not antispoof in

# No restriction on private network interfaces
$cmd 00005 allow all from any to any via igb0
$cmd 00008 allow all from any to any via ix1

# No restriction on loopback interface
$cmd 00010 allow all from any to any via lo0

# Allow the packet through if it has previously been added to the
# the "dynamic" rules table by an allow keep-state statement.
$cmd 00025 check-state

$cmd 00100 allow tcp from me to any out via $pif setup keep-state
$cmd 00150 allow udp from me to any out via $pif keep-state
$cmd 00200 allow icmp from me to any out via $pif keep-state
$cmd 00299 deny log all from any to any out via $pif

# Deny all inbound traffic from non-routable reserved address spaces
$cmd 00300 deny all from 192.168.0.0/16 to any in via $pif
$cmd 00301 deny all from 172.16.0.0/12 to any in via $pif
$cmd 00302 deny all from 10.0.0.0/8 to any in via $pif
$cmd 00303 deny all from 127.0.0.0/8 to any in via $pif
$cmd 00304 deny all from 0.0.0.0/8 to any in via $pif
$cmd 00305 deny all from 169.254.0.0/16 to any in via $pif
$cmd 00306 deny all from 192.0.2.0/24 to any in via $pif
$cmd 00307 deny all from 204.152.64.0/23 to any in via $pif
$cmd 00308 deny all from 224.0.0.0/3 to any in via $pif

# Deny public pings
$cmd 00310 deny icmp from any to any in via $pif

# Deny rpcinfo
$cmd 00315 deny tcp from any to any 111 in via $pif

# Deny any late arriving packets
$cmd 00330 deny all from any to any frag in via $pif

# Deny ACK packets that did not match the dynamic rule table
$cmd 00335 deny tcp from any to any established in via $pif

# Allow in SSH
$cmd 00410 allow tcp from any to me 22 in via $pif setup limit src-addr 3

# Reject all incoming connections from the outside
$cmd 00499 deny all from any to any in via $pif

$cmd 00999 deny log all from any to any

Start the firewall:

# service ipfw start

Maintenance

Replacing a failed hard drive

We experienced a hard drive failure in June 2013. We saw the following error in dmesg:

mfi0: 501114 (455297952s/0x0002/info) - Unexpected sense: PD 12(e0x09/s5) Path 50000c0f0123e536, CDB: 8f 00 00 00 00 00 91 ca 32 56 00 00 10 00 00 00, Sense: 3/11/00

The zpool status command spat out error too:

# zpool status zhome
  pool: zhome
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	zhome       ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    mfid24  ONLINE       0     0     0
	    mfid25  ONLINE       0     0     0
	    mfid26  ONLINE       0     0     0
	    mfid27  ONLINE       0     0     0
	    mfid28  ONLINE       0     0     0
	    mfid29  ONLINE       0     0   296
	    mfid30  ONLINE       0     0     0
	    mfid31  ONLINE       0     0     0
	    mfid32  ONLINE       0     0     0
	    mfid33  ONLINE       0     0     0
	    mfid34  ONLINE       0     0     0
	    mfid35  ONLINE       0     0     0

errors: No known data errors

mfid29 was the RAID0 volume for physical drive PD12 (E2:S5). Let's fail it:

# mfiutil fail E2:S5

# mfiutil show drives | grep E2:S5
18 (   0.0) FAILED <WD WD4001FYYG-01SL3 VR07 serial=WD-WMC1F1253920> SCSI-6 E2:S5

# mfiutil show volumes | grep 29
    29 ( 3725G) RAID-0      64k OFFLINE Writes  

Now zpool status gave a different error:

# zpool status zhome
  pool: zhome
 state: DEGRADED
status: One or more devices has been removed by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
  scan: none requested
config:

	NAME            STATE     READ WRITE CKSUM
	zhome           DEGRADED     0     0     0
	  raidz2-0      DEGRADED     0     0     0
	    mfid24      ONLINE       0     0     0
	    mfid25      ONLINE       0     0     0
	    mfid26      ONLINE       0     0     0
	    mfid27      ONLINE       0     0     0
	    mfid28      ONLINE       0     0     0
	    1791299633  REMOVED      0     0     0  was /dev/mfid29
	    mfid30      ONLINE       0     0     0
	    mfid31      ONLINE       0     0     0
	    mfid32      ONLINE       0     0     0
	    mfid33      ONLINE       0     0     0
	    mfid34      ONLINE       0     0     0
	    mfid35      ONLINE       0     0     0

errors: No known data errors

I then attempted to hot swap the failed drive, but without success!

After removing the failed drive from and inserting a good drive into slot 5 in the rear of the chassis, I got:

# mfiutil show drives | grep E2:S5
46 (   0.0) UNCONFIGURED GOOD <WD WD4001FYYG-01SL3 VR07 serial=WD-WMC1F1254853> SCSI-6 E2:S5

I no longer saw mfid29 in the output of mfiutil show volumes. However, I failed at any further attempt at bringing the new drive online:

# mfiutil create jbod -v E2:S5
Adding drive 46 to array 35
Adding array 35 to volume 29
mfiutil: Command failed: Status: 0x54
mfiutil: Failed to add volume: Input/output error

# mfiutil add E2:S5 mfid29
mfiutil: Invalid volume mfid29: Invalid argument

# mfiutil delete mfid29
mfiutil: Invalid volume mfid29: Invalid argument

Perhaps the MegaCLI utility could have worked[5]. I, however, took the easier route of rebooting the server.

When the server was booting, I pressed Ctrl-H to start the LSI WebBIOS configuration utility[6]. I then discarded the preserved cache of the virtual drives in WebBIOS.

When FreeBSD was again booted up, I logged in and found that the virtual volume names were not persistent:

# mfiutil show volumes
mfi0 Volumes:
  Id     Size    Level   Stripe  State   Cache   Name
 mfid0 ( 3725G) RAID-0      64k OPTIMAL Writes  
 mfid1 ( 3725G) RAID-0      64k OPTIMAL Writes  
 ...
mfid28 ( 3725G) RAID-0      64k OPTIMAL Writes  
mfid29 ( 3725G) RAID-0      64k OPTIMAL Writes  
mfid30 ( 3725G) RAID-0      64k OPTIMAL Writes  
mfid31 ( 3725G) RAID-0      64k OPTIMAL Writes  
mfid32 ( 3725G) RAID-0      64k OPTIMAL Writes  
mfid33 ( 3725G) RAID-0      64k OPTIMAL Writes  
mfid34 ( 3725G) RAID-0      64k OPTIMAL Writes

Note the new mfid29 – mfid34 were the old mfid30 – mfid35!

The output of zpool status was different too:

# zpool status zhome
  pool: zhome
 state: DEGRADED
status: One or more devices has been removed by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
  scan: none requested
config:

	NAME            STATE     READ WRITE CKSUM
	zhome           DEGRADED     0     0     0
	  raidz2-0      DEGRADED     0     0     0
	    mfid24      ONLINE       0     0     0
	    mfid25      ONLINE       0     0     0
	    mfid26      ONLINE       0     0     0
	    mfid27      ONLINE       0     0     0
	    mfid28      ONLINE       0     0     0
	    1791299633  REMOVED      0     0     0  was /dev/mfid29
	    mfid29      ONLINE       0     0     0
	    mfid30      ONLINE       0     0     0
	    mfid31      ONLINE       0     0     0
	    mfid32      ONLINE       0     0     0
	    mfid33      ONLINE       0     0     0
	    mfid34      ONLINE       0     0     0

errors: No known data errors

Now I was able to create a RAID0 volume for the new drive:

# mfiutil show drives | grep E2:S5
46 (   0.0) UNCONFIGURED GOOD <WD WD4001FYYG-01SL3 VR07 serial=WD-WMC1F1254853> SCSI-6 E2:S5

# mfiutil create jbod -v E2:S5
Adding drive 46 to array 35
Adding array 35 to volume 29

# mfiutil show config
...
    array 35 of 1 drives:
        drive 46 (   0.0) ONLINE <WD WD4001FYYG-01SL3 VR07 serial=WD-WMC1F1254853> SCSI-6
...
    volume mfid35 (3725G) RAID-0 64k OPTIMAL spans:
        array 35

I then ran zpool replace to replace the failed disk in zhome (1791299633 was the GUID of the failed device[7]):

# zpool replace zhome 1791299633 mfid35

Note the command I actually used was zpool replace zhome mfi29 mfid35, which succeeded without a hitch!

It took about 4 hours for the resilver to complete:

# zpool status -v zhome
  pool: zhome
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Jun  6 11:30:11 2014
        25.7G scanned out of 4.49T at 347M/s, 3h45m to go
        2.11G resilvered, 0.56% done
config:

	NAME              STATE     READ WRITE CKSUM
	zhome             DEGRADED     0     0     0
	  raidz2-0        DEGRADED     0     0     0
	    mfid24        ONLINE       0     0     0
	    mfid25        ONLINE       0     0     0
	    mfid26        ONLINE       0     0     0
	    mfid27        ONLINE       0     0     0
	    mfid28        ONLINE       0     0     0
	    replacing-5   REMOVED      0     0     0
	      1791299633  REMOVED      0     0     0  was /dev/mfid29
	      mfid35      ONLINE       0     0     0  (resilvering)
	    mfid29        ONLINE       0     0     0
	    mfid30        ONLINE       0     0     0
	    mfid31        ONLINE       0     0     0
	    mfid32        ONLINE       0     0     0
	    mfid33        ONLINE       0     0     0
	    mfid34        ONLINE       0     0     0

errors: No known data errors

Upgrading to FreeBSD 9.3

We upgraded Ambrosia to FreeBSD 9.3 in July 2014.

Upgrade Ambrosia from FreeBSD 9.2 to FreeBSD 9.3[8]:

# freebsd-update -r 9.3-RELEASE upgrade
# freebsd-update install

Reboot the machine:

# shutdown -r now

Restart freebsd-update:

# freebsd-update install

freebsd-update

Applying Security Patches:

# freebsd-update fetch
# freebsd-update install
# shutdown -r now

See Also

References

  1. FreeBSD Handbook - Installing FreeBSD 9.X and Later
  2. FreeBSD Handbook - Network File System (NFS)
  3. ZFS Tuning Guide
  4. FreeBSD Handbook - IPFW
  5. mfi (Dell H700) + hot swapping doesn't appear to work with RC1
  6. MegaRAID SAS Software User Guide
  7. FreeBSD Handbook - zpool Administration
  8. FreeBSD Handbook - Updating and Upgrading FreeBSD