This how-to documents how to setup ZFS and install FreeBSD 9.1 into it. ZFS is a great modern file system (invented by SUN Microsystems) to use under FreeBSD. It is very fast and will give you the same features available in a modern SAN (mirrors, raid, snapshots, clones, and pooled storage just to name a few).
In this document I am building two ZFS pools, one across two drives (zroot) in a mirror configuration and the other (zlocal) that is a RAID setup across 3 drives. The system will be setup to boot from the mirror pool where most of the system resides and the RAID drives will be reserved for stuff under /usr/local. It aligns the ZFS file systems to 4K clusters for newer drives and also does not use ZFS legacy mountpoints. The plan is to have two SSD drives that contain the boot and system files for speed with the RAID drives using some large SATA disks for storage.
The first thing to do is to boot from a FreeBSD 9.1 DVD or memstick. Select the “Live CD” option once the system comes up. Login as “root”, there is no password.
Partitioning the disk.
# gpart create -s gpt ad0
# gpart create -s gpt ad1
# gpart create -s gpt ad2
# gpart create -s gpt ad3
# gpart create -s gpt ad4
Create the boot, swap, and zfs partitions on the two mirror drives.
# gpart add -b 34 -s 64k -t freebsd-boot ad0
# gpart add -s 4G -t freebsd-swap -l swap0 ad0
# gpart add -t freebsd-zfs -l disk0 ad0
# gpart add -b 34 -s 64k -t freebsd-boot ad1
# gpart add -s 4G -t freebsd-swap -l swap1 ad1
# gpart add -t freebsd-zfs -l disk1 ad1
Create zfs patrition on the three RAID drives.
# gpart add -t freebsd-zfs -l disk2 ad2
# gpart add -t freebsd-zfs -l disk3 ad3
# gpart add -t freebsd-zfs -l disk4 ad4
Install the Protected MBR and gptzfsboot loader on the mirror drives.
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad1
Load ZFS Kernel module
# kldload /boot/kernel/opensolaris.ko
# kldload /boot/kernel/zfs.ko
Align the disks to 4K clusters.
# gnop create –S 4096 /dev/gpt/disk0
# gnop create –S 4096 /dev/gpt/disk1
# gnop create –S 4096 /dev/gpt/disk2
# gnop create –S 4096 /dev/gpt/disk3
# gnop create –S 4096 /dev/gpt/disk4
Setup the ZFS mirror pool (zroot) on the two mirror drives.
# zpool create -m / –o altroot=/mnt –o cachefile=/var/tmp/zpool.cache zroot mirror /dev/gpt/disk0.nop /dev/gpt/disk1.nop
# zpool export zroot
# gnop destroy /dev/gpt/disk0.nop
# gnop destroy /dev/gpt/disk1.nop
# zpool import –o altroot=/mnt –o cachefile=/var/tmp/zpool.cache zroot
Create the /tmp file systems
# zfs create –o compression=on –o exec=on –o setuid=off zroot/tmp
# chmod 1777 /mnt/tmp
Create the /usr,/var and /home file systems
# zfs create zroot/usr
# zfs create zroot/usr/home
# cd /mnt; ln –s usr/home home
# zfs create zroot/var
Setup the ZFS RAID pool (zlocal) on the three extra drives.
# mkdir –p /mnt/usr/local
# zpool create –o altroot=/mnt/usr/local –o cachefile=/var/tmp/zpool.cache zlocal raidz1 /dev/gpt/disk2.nop /dev/gpt/disk3.nop /dev/gpt/disk4.nop
# zpool export zlocal
# gnop destroy /dev/gpt/disk2.nop
# gnop destroy /dev/gpt/disk3.nop
# gnop destroy /dev/gpt/disk4.nop
# zpool import –o altroot=/mnt/usr/local –o cachefile=/var/tmp/zpool.cache zlocal
Setup some final ZFS settings for the boot drives and the checksum algorithm to use.
# zpool set bootfs=zroot zroot
# zfs set checksum=fletcher4 zroot
# zfs set checksum=fletcher4 zlocal
Create the rest of the file systems
Create the ports file systems
# zfs create –o compression=lzjb –o setuid=off zroot/usr/ports
# zfs create –o compression=off -o setuid=off –o exec=off zroot/usr/ports/distfiles
# zfs create –o compression=off -o setuid=off –o exec=off zroot/usr/ports/packages
Create the system src tree file system.
# zfs create –o compression=lzjb –o setuid=off –o exec=off zroot/usr/src
Create the rest of the file file systems (You can customize this to your needs).
# zfs create –o compression=lzjb –o setuid=off –o exec=off zroot/var/crash
# zfs create –o compression=off –o setuid=off –o exec=off zroot/var/db
# zfs create –o compression=lzjb –o setuid=off –o exec=on zroot/var/db/pkg
# zfs create –o compression=off –o setuid=off –o exec=off zroot/var/empty
# zfs create –o compression=lzjb –o setuid=off –o exec=off zroot/var/log
# zfs create –o compression=gzip –o setuid=off –o exec=off zroot/var/mail
# zfs create –o compression=off –o setuid=off –o exec=off zroot/var/run
# zfs create –o compression=lzjb –o setuid=off –o exec=on zroot/var/tmp
# chmod 1777 /mnt/var/tmp
Create the /usr/local file system on the zlocal disks
# zfs create –o compression=lzjb –o setuid=off –o exec=off zlocal/src
Install FreeBSD onto the new ZFS drives
# sh
# cd /usr/freebsd-dist
# export DESTDIR=/mnt
# for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz games.txz; do (cat $file | tar –-unlink –xpJf – -C ${DESTDIR:-/}); done
Make /var/empty readable and copy over the ZFS cache file
# zfs set readonly=on zroot/var/empty
# cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache
Create a basic /etc/rc.conf
# echo ‘zfs_enable=”YES”’ > /mnt/etc/rc.conf
# echo ‘hostname=”beastie.mydomain.local”’ >> /mnt/etc/rc.conf
# ‘ifconfig_em0=”DHCP”’ >> /mnt/etc/rc.conf
Create the basic loader.conf
# echo ‘zfs_load=”YES”’ > /mnt/boot/loader.conf
# echo ‘vfs.root.mountfrom=”zfs:zroot”’ >> /mnt/boot/loader.conf
Get the regular nightly status updates.
# echo ‘daily_status_zfs_enable=”YES”‘ >> /mnt/etc/periodic.conf
Finish up the Install
Create /etc/fstab to use both swap partitions and support for mounting a CD.
# cat << EOF > /mnt/etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/gpt/swap0 none swap sw 0 0
/dev/gpt/swap1 none swap sw 0 0
/dev/cd0 /cdrom cd9660 ro.noauto 0 0
EOF
Create the CDrom mount point and then unmount the ZFS file systems.
# mkdir /mnt/cdrom
# zfs umount –a
Change the mountpoints for the zroot and zlocal pools
# zfs set mountpoint=/ zroot
# zfs set mountpoint=/tmp zroot/tmp
# zfs set mountpoint=/usr zroot/usr
# zfs set mountpoint=/var zroot/var
# zfs set mountpoint=/usr/local zlocal
Reboot and remove the CD or memstick. It should boot from ZFS on the hard drives.
Final Touchups
One the system reboots into the newly install system. Login and perform the following.
Change root’s password
# passwd
Setup the local timezone
# tzsetup
Create the mail aliases file
# cd /etc/mail
# make aliases
Customize your new system to your needs. It should be ready to go.
Common ZFS usage and trouble shooting commads.
Snapshots are a way to create a quick backup of a ZFS file system. You can then do some work and restore the file system back to the time the snapshot was taken if needed. Snapshots grow over time. This means that taking a snapshot of a volume takes no time at all and doesn’t use any space initially but as the file system gets used after the snapshot and the volumes data diverges farther from what it was at the time of the snapshot, the snapshot size increases. This is why snapshots are not a good solution for long term backups but are great for a quick backup and restore. Snapshots work at the file system (ZFS volume) level so there is no easy way to restore a single file from a snapshot. Restoring the snapshot restores everything in that file system back to the point the snapshot was taken.
In the example below, assume we have a ZFS pool called zlocal and within that pool we have created a volume called zlocal/src.
Creating a snapshot of the src file system.
# zfs rollback zlocal/src@2012-07-28
Note: The information after the @ sign gives you the snapshot identifier. I usually use the current date so I know when that snapshot was taken.
Restoring back to a snapshot.
# zfs rollback zlocal/src@2012-07-28
List the available snapshots for the src file system (note. The zlocal/src file system is mounted on /usr/local/src)
# ls –al /usr/local/src/.zfs/snapshot
Total 2
dr-xr-xr-x 2 root wheel 2 Jul 28 15:12 .
dr-xr-xr-x 2 root wheel 2 Jul 27 14:51 ..
dr-xr-xr-x 2 root wheel 2 Jul 28 15:11 2012-07-28
Deleting a snapshot
# zfs destroy zlocal/src@2012-07-28
Recovering from a failed from (RAID-Z Volume).
To check in the ZFS pool are healthy
# zpool status –x
For more information on the ZFS pools
# zpool status
You can used the existing checksum information within ZFS to verify the data on a ZFS file system. This runs in the background. Note: scrub sounds bad but it is safe to run.
# zpool scrub zlocal
You can check the results of the verify with the status command.
# zpool status zlocal
A drive can be taken off line by the administrator with the following command.
# zpool offline zlocal da3
The drive can then be replaced and put back online with.
# zpool replace zlocal da3
Replacing a mirror drive (one of the boot drives).
Lets assume that da1 (/dev/gpt/disk1 is the label on that drive) is the drive that failed.
Take the drive offline.
# zpool offline zroot /dev/gpt/disk1
Remove the drive and replace with the new one. Reboot.
Create the partitions on the new drive and make it bootable.
# gpart create –s gpt da1
# gpart add –b 34 –s 64k –t freebsd-boot da1
# gpart add –s 3G –t freebsd-swap –l swap1 da1
# gpart add –t freebsd-zfs –l disk1 da1
# gpart bootcode –b /boot/pmbr –p /boot/gptzfsboot –i 1 da1
Aligning to 4K clusters is not required on the new drive as the alignment is on a ZFS pool basis and the 4K alignment is already stored in the metadata for the ZFS pool.
You can check the alignment is correct with the following command.
# zdb zroot | grep ashift
The above command should return 12 for a pool that is 4K aligned.
Replace the drive and add it back into the ZFS pool.
# zpool replace zroot /dev/gpt/disk1
Replacing a RAID-Z (zlocal) drive (not a main boot drive).
Lets assume that da3 (/dev/gpt/disk3 is the label on that drive) is the drive that failed.
Take the drive offline.
# zpool offline zlocal /dev/gpt/disk3
Remove the drive and replace with the new one. Reboot.
Create the partition on the new drive.
# gpart create –s gpt da3
# gpart add –t freebsd-zfs –l disk3 da3
Aligning to 4K clusters is not required on the new drive as the alignment is on a ZFS pool basis and the 4K alignment is already stored in the metadata for the ZFS pool.
You can check the alignment is correct with the following command.
# zdb zlocal | grep ashift
The above command should return 12 for a pool that is 4K aligned.
Replace the drive and add it back into the ZFS pool.
# zpool replace zlocal /dev/gpt/disk3