Index: geom/chapter.sgml
===================================================================
--- geom/chapter.sgml	(revision 38972)
+++ geom/chapter.sgml	(working copy)
@@ -244,143 +244,370 @@
       or access interruption, and even be physically stored in a data
       safe.</para>
 
-    <para>To begin, ensure the system has two disk drives of equal
-      size, these exercises assume they are direct access (&man.da.4;)
-      <acronym>SCSI</acronym> disks.</para>
-
     <sect2>
       <title>Mirroring Primary Disks</title>
 
-      <para>Assuming &os; has been installed on the first,
-	<devicename>da0</devicename> disk device, &man.gmirror.8;
-	should be told to store its primary data there.</para>
+      <para>In this example, &os; has already been installed on a
+	single disk, <devicename>ada0</devicename>.  A new disk,
+	<devicename>ada1</devicename>, has been added to the system.
+	A new mirror will be created first on the new single disk, the
+	existing system copied to it, and then the mirror will be
+	configured to use the both disks.  This slightly complex
+	procedure is needed because the mirror configuration needs a
+	512-byte information at the end of each disk though the
+	<devicename>ada0</devicename> uses the whole disk in most
+	cases.  Partitions on the new mirror can be larger or smaller
+	than those on the existing disk, but they must be large enough
+	to hold the data already present on
+	<devicename>ada0</devicename>.</para>
 
-      <para>Before building the mirror, enable additional debugging
-	information and opening access to the device by setting the
-	<varname>kern.geom.debugflags</varname> &man.sysctl.8; option
-	to the following value:</para>
+      <para>It is traditional and often desirable for the two mirror
+	drives to be identical in model and capacity, but it is not
+	required.  Mirrors created with dissimilar drives will have a
+	capacity equal to that of the smallest drive in the mirror.
+	Drives inserted into the mirror later must have at least as
+	much capacity as the smallest drive already in the
+	mirror.</para>
 
-      <screen>&prompt.root; <userinput>sysctl kern.geom.debugflags=17</userinput></screen>
-
-      <para>Now create the mirror.  Begin the process by storing
-	meta-data information on the primary disk device,
-	effectively creating the
-	<filename class="devicefile">/dev/mirror/gm</filename> device
-	using the following command:</para>
-
       <warning>
-	<para>Creating a mirror out of the boot drive may result in
-	  data loss if any data has been stored on the last sector of
-	  the disk.  This risk is reduced if creating the mirror is
-	  done promptly after a fresh install of &os;.  The following
-	  procedure is also incompatible with the default installation
-	  settings of &os;&nbsp;9.<replaceable>X</replaceable> which
-	  use the new <acronym>GPT</acronym> partition scheme.  GEOM
-	  will overwrite <acronym>GPT</acronym> metadata, causing data
-	  loss and possibly an unbootable system.</para>
+	<para>The mirroring procedure shown here is non-destructive,
+	  but as with any major disk operation, a complete backup
+	  should be completed and verified first.</para>
       </warning>
 
-      <screen>&prompt.root; <userinput>gmirror label -vb round-robin gm0 /dev/da0</userinput></screen>
+      <note>
+	<para>MBR partitioning is used here because full disks with
+	  GPT partition tables cannot be mirrored with
+	  &man.gmirror.8;.  In that situation, both GPT and
+	  &man.gmirror.8; store metadata at the end of the disk, and
+	  one will overwrite the other.  It is possible to instead
+	  mirror individual GPT partitions.  &man.gmirror.8; will keep
+	  metadata inside each partition, and will not interfere with
+	  the GPT metadata at the end of the disk.  That technique
+	  will not be covered here, but it is not much more complex
+	  than the MBR setup shown.</para>
+      </note>
 
-      <para>The system should respond with:</para>
+      <para>&man.gmirror.8; requires a kernel module,
+	<filename>geom_mirror.ko</filename>, either built into the
+	kernel or loaded at boot- or run-time.  Manually load the
+	kernel modules now:</para>
 
-      <screen>Metadata value stored on /dev/da0.
-Done.</screen>
+      <screen>&prompt.root; <userinput>gmirror load</userinput></screen>
 
-      <para>Initialize GEOM, this will load the
-	<filename>/boot/kernel/geom_mirror.ko</filename> kernel
-	module:</para>
+      <para>If any mirror or label metadata, or partition table
+	already exists on the new drives, it can be erased with
+	&man.glabel.8;, &man.gmirror.8;, or &man.dd.1;.</para>
 
-      <screen>&prompt.root; <userinput>gmirror load</userinput></screen>
+      <screen>&prompt.root; <userinput>glabel clear -v /dev/ada1</userinput>
+&prompt.root; <userinput>gmirror clear -v /dev/ada1</userinput>
+&prompt.root; <userinput>dd if=/dev/zero of=/dev/ada1 bs=512 count=1024</userinput></screen>
 
-      <note>
-	<para>When this command completes successfully, it creates the
-	  <devicename>gm0</devicename> device node under the
-	  <filename class="directory">/dev/mirror</filename>
-	  directory.</para>
-      </note>
+      <para>Device names and numbers vary depending on where and how
+	the drive is connected.  <devicename>ada0</devicename> may
+	become <devicename>ada6</devicename> if it is plugged into a
+	different port, or the name may change from
+	<devicename>ada</devicename> to <devicename>da</devicename> if
+	it is moved from a SATA port to a USB SATA adapter.</para>
 
-      <para>Enable loading of the <filename>geom_mirror.ko</filename>
-	kernel module during system initialization:</para>
+      <para>First, check the media size of the primary disk by using
+	&man.gpart.8;.</para>
 
-      <screen>&prompt.root; <userinput>echo 'geom_mirror_load="YES"' &gt;&gt; /boot/loader.conf</userinput></screen>
+      <screen>&prompt.root; <userinput>gpart list ada0 | grep size | tail -2</userinput>
+Mediasize: 1000204821504 (931G)
+Sectorsize: 512</screen>
 
-      <para>Edit the <filename>/etc/fstab</filename> file, replacing
-	references to the old <devicename>da0</devicename> with the
-	new device nodes of the <devicename>gm0</devicename> mirror
-	device.</para>
+      <para>Create a mirror on the new single disk.  The new disk must
+	be equal to or larger than the above media size.</para>
 
-      <note>
-	<para>If &man.vi.1; is your preferred editor, the following is
-	  an easy way to accomplish this task:</para>
+	<screen>&prompt.root; <userinput>geom zero load</userinput>
+&prompt.root; <userinput>gnop -s 1000204821504 gzero</userinput>
+&prompt.root; <userinput>gmirror label -v gm0 gzero.nop ada1</userinput>
+&prompt.root; <userinput>gmirror forget gm0</userinput></screen>
 
-	<screen>&prompt.root; <userinput>vi /etc/fstab</userinput></screen>
+      <para>As the result of the above commands,
+	<devicename>mirror/gm0</devicename> with the same size as
+	<devicename>ada0</devicename> is created by using
+	<devicename>ada1</devicename>.  Note that the <option>-s
+	1000204821504</option> in the second line must be equal to the
+	media size of <devicename>ada0</devicename> shown in the
+	output of &man.gpart.8;.</para>
 
-	<para>In &man.vi.1; back up the current contents of
-	  <filename>fstab</filename> by typing
-	  <userinput>:w /etc/fstab.bak</userinput>.  Then
-	  replace all old <devicename>da0</devicename> references
-	  with <devicename>gm0</devicename> by typing
-	  <userinput>:%s/da/mirror\/gm/g</userinput>.</para>
-      </note>
+      <para>The technical details are as follows.  In the above
+	commands, the kernel module <filename>geom_zero.ko</filename>
+	is loaded, a pseudo block device
+	<filename>/dev/gzero.nop</filename> which has the same size as
+	<devicename>ada0</devicename> is created, and a mirror
+	<devicename>mirror/gm0</devicename> is configured by using
+	<devicename>/dev/gzero.nop</devicename> and
+	<devicename>ada1</devicename>.  The
+	<devicename>/dev/gzero.nop</devicename> is not a real block
+	device and just used to limit the capacity of
+	<devicename>mirror/gm0</devicename> (&man.gmirror.8; uses the
+	smallest one in the member disks as the capacity of
+	<devicename>mirror/gm0</devicename>).  The fourth command
+	removes the pseudo device from
+	<devicename>mirror/gm0</devicename>.</para>
 
-      <para>The resulting <filename>fstab</filename> file should look
-	similar to the following.  It does not matter if the disk
-	drives are <acronym>SCSI</acronym> or <acronym>ATA</acronym>,
-	the <acronym>RAID</acronym> device will be
-	<devicename>gm</devicename> regardless.</para>
+      <para>Other than the device name,
+	<devicename>mirror/gm0</devicename> acts just like a single
+	drive.  MBR and bsdlabel partition tables can now be created
+	on the new mirror with &man.gpart.8;.  This example shows a
+	traditional split-filesystem layout, but a single
+	<filename>/</filename> filesystem and a swap partition will
+	work just as well.</para>
 
-      <programlisting># Device		Mountpoint	FStype	Options		Dump	Pass#
-/dev/mirror/gm0s1b	none		swap	sw		0	0
-/dev/mirror/gm0s1a	/		ufs	rw		1	1
-/dev/mirror/gm0s1d	/usr		ufs	rw		0	0
-/dev/mirror/gm0s1f	/home		ufs	rw		2	2
-#/dev/mirror/gm0s2d	/store		ufs	rw		2	2
-/dev/mirror/gm0s1e	/var		ufs	rw		2       2
-/dev/acd0		/cdrom		cd9660	ro,noauto	0	0</programlisting>
+      <para>After creating <devicename>mirror/gm0</devicename>, check
+	the partition table on <devicename>ada0</devicename>.</para>
 
-      <para>Reboot the system:</para>
+      <screen>&prompt.root; <userinput>gpart show ada0</userinput>
+=>        63  1953525104        ada0  MBR  (931G)
+          63  1953525042           1  freebsd  [active]  (931G)
+  1953525105          62              - free -  (31k)</screen>
 
-      <screen>&prompt.root; <userinput>shutdown -r now</userinput></screen>
+      <para>This output is an example of a 1&nbsp;TB drive.  If there is
+	some free part at the last of the drive, it is possible to
+	copy the contents from <devicename>ada0</devicename> with an
+	additional 512-byte information at the end of the disk.</para>
 
-      <para>During system initialization, the
-	<devicename>gm0</devicename> should be used in place of the
-	<devicename>da0</devicename> device.  Once fully initialized,
-	this may be checked by visually inspecting the output from
-	the <command>mount</command> command:</para>
+      <para>However, if the output is something like the following,
+	the whole disk is already allocated for &os;.  It means there
+	is no space to store the 512-byte information to configure the
+	mirror.</para>
 
-      <screen>&prompt.root; <userinput>mount</userinput>
-Filesystem         1K-blocks    Used    Avail Capacity  Mounted on
-/dev/mirror/gm0s1a   1012974  224604   707334    24%    /
-devfs                      1       1        0   100%    /dev
-/dev/mirror/gm0s1f  45970182   28596 42263972     0%    /home
-/dev/mirror/gm0s1d   6090094 1348356  4254532    24%    /usr
-/dev/mirror/gm0s1e   3045006 2241420   559986    80%    /var
-devfs                      1       1        0   100%    /var/named/dev</screen>
+      <screen>&prompt.root; <userinput>gpart show ada0</userinput>
+=>        63  1953525105        ada0  MBR  (931G)
+          63  1953525105           1  freebsd  [active]  (931G)</screen>
 
-      <para>The output looks good, as expected.  Finally, to begin
-	synchronization, insert the <devicename>da1</devicename> disk
-	into the mirror using the following command:</para>
+      <para>In this case, edit the partition table and reduce the
+	capacity by one sector on <devicename>mirror/gm0</devicename>.
+	The detail procedure will be explained later.</para>
 
-      <screen>&prompt.root; <userinput>gmirror insert gm0 /dev/da1</userinput></screen>
+      <para>In either case, partition tables on the primary disk
+	should be copied first.  It can be done by using &man.gpart.8;
+	<command>backup</command> and <command>restore</command>
+	subcommands.</para>
 
-      <para>As the mirror is built the status may be checked using
-	the following command:</para>
+      <screen>&prompt.root; <userinput>gpart backup ada0 &gt; table.ada0</userinput>
+&prompt.root; <userinput>gpart backup ada0s1 &gt; table.ada0s1</userinput></screen>
 
-      <screen>&prompt.root; <userinput>gmirror status</userinput></screen>
+      <para>The above commands will create two files,
+	<filename>table.ada0</filename> and
+	<filename>table.ada0s1</filename>.  The following is an
+	example of a 1&nbsp;TB drive.</para>
 
-      <para>Once the mirror has been built and all current data
-	has been synchronized, the output from the above command
-	should look like:</para>
+      <screen>&prompt.root; <userinput>cat table.ada0</userinput>
+MBR 4
+1 freebsd         63 1953525105   [active]</screen>
 
-      <screen>      Name    Status  Components
-mirror/gm0  COMPLETE  da0
-                      da1</screen>
+      <screen>&prompt.root; <userinput>cat table.ada0s1</userinput>
+BSD 8
+1  freebsd-ufs          0    4194304  
+2 freebsd-swap    4194304   33554432  
+4  freebsd-ufs   37748736   50331648  
+5  freebsd-ufs   88080384   41943040  
+6  freebsd-ufs  130023424  838860800  
+7  freebsd-ufs  968884224  984640881</screen>
 
-      <para>If there are any issues, or the mirror is still
-	completing the build process, the example will show
-	<literal>DEGRADED</literal> in place of
-	<literal>COMPLETE</literal>.</para>
+      <para>If the whole disk was used in the output of &man.gpart.8;
+	<command>show</command>, the capacity in these partition
+	tables must be reduced by one sector.  Edit the two files like
+	the following.</para>
+
+      <screen>&prompt.root; <userinput>cat table.ada0</userinput>
+MBR 4
+1 freebsd         63 <emphasis>1953525104</emphasis>   [active]</screen>
+
+      <screen>&prompt.root; <userinput>cat table.ada0s1</userinput>
+BSD 8
+1  freebsd-ufs          0    4194304  
+2 freebsd-swap    4194304   33554432  
+4  freebsd-ufs   37748736   50331648  
+5  freebsd-ufs   88080384   41943040  
+6  freebsd-ufs  130023424  838860800  
+7  freebsd-ufs  968884224  <emphasis>984640880</emphasis></screen>
+
+      <para>If not, these two files can be used without modification.</para>
+
+      <para>Now restore the partition table into
+	<devicename>mirror/gm0</devicename>.</para>
+
+      <screen>&prompt.root; <userinput>gpart restore mirror/gm0 &lt; table.ada0</userinput>
+&prompt.root; <userinput>gpart restore mirror/gm0s1 &lt; table.ada0s1</userinput></screen>
+
+      <para>Check the partition table by using &man.gpart.8;
+	<command>show</command>.  This example has
+	<devicename>gm0s1a</devicename> for <filename>/</filename>,
+	<devicename>gm0s1d</devicename> for <filename>/var</filename>,
+	<devicename>gm0s1e</devicename> for <filename>/usr</filename>,
+	<devicename>gm0s1f</devicename> for <filename>/data1</filename>, and
+	<devicename>gm0s1g</devicename> for <filename>/data2</filename>.</para>
+
+      <screen>&prompt.root; <userinput>gpart show mirror/gm0</userinput>
+=>        63  1953525104  mirror/gm0  MBR  (931G)
+          63  1953525042           1  freebsd  [active]  (931G)
+  1953525105          62              - free -  (31k)
+
+&prompt.root; <userinput>gpart show mirror/gm0s1</userinput>
+=>         0  1953525042  mirror/gm0s1  BSD  (931G)
+           0     2097152             1  freebsd-ufs  (1.0G)
+     2097152    16777216             2  freebsd-swap  (8.0G)
+    18874368    41943040             4  freebsd-ufs  (20G)
+    60817408    20971520             5  freebsd-ufs  (10G)
+    81788928   629145600             6  freebsd-ufs  (300G)
+   710934528  1242590514             7  freebsd-ufs  (592G)
+  1953525042          63                - free -  (31k)</screen>
+
+      <para>Two of the partition tables should have some free part at
+	the end of each disk.</para>
+
+      <para>Then, create new filesystems on these partitions.  The
+	number of partitions depends on the primary disk.</para>
+
+      <screen>&prompt.root; <userinput>newfs -U /dev/mirror/gm0s1a</userinput>
+&prompt.root; <userinput>newfs -U /dev/mirror/gm0s1d</userinput>
+&prompt.root; <userinput>newfs -U /dev/mirror/gm0s1e</userinput>
+&prompt.root; <userinput>newfs -U /dev/mirror/gm0s1f</userinput>
+&prompt.root; <userinput>newfs -U /dev/mirror/gm0s1g</userinput></screen>
+
+      <para>Make the mirror bootable by installing bootcode in the MBR
+	and bsdlabel and setting the active slice:</para>
+
+      <screen>&prompt.root; <userinput>gpart bootcode -b /boot/mbr mirror/gm0</userinput>
+&prompt.root; <userinput>gpart set -a active -i 1 mirror/gm0</userinput>
+&prompt.root; <userinput>gpart bootcode -b /boot/boot mirror/gm0s1</userinput></screen>
+
+      <para>Adjust the <filename>/etc/fstab</filename> to use the
+	new partitions on the mirror.  Back up this file first by copying it to
+	<filename>/etc/fstab.orig</filename>.</para>
+
+      <screen>&prompt.root; <userinput>cp /etc/fstab /etc/fstab.orig</userinput></screen>
+
+      <para>Then edit <filename>/etc/fstab</filename>.  Replace the
+	all of <devicename>/dev/ada0</devicename> with
+	<devicename>mirror/gm0</devicename>.</para>
+
+      <programlisting># Device		Mountpoint	FStype	Options	Dump	Pass#
+/dev/mirror/gm0s1a	/		ufs	rw	1	1
+/dev/mirror/gm0s1b	none		swap	sw	0	0
+/dev/mirror/gm0s1d	/var		ufs	rw	2	2
+/dev/mirror/gm0s1e	/usr		ufs	rw	2	2
+/dev/mirror/gm0s1f	/data1		ufs	rw	2	2
+/dev/mirror/gm0s1g	/data2		ufs	rw	2	2</programlisting>
+
+      <para>Also, in case the &man.gmirror.8; kernel module has not
+	been built into the kernel,
+	<filename>/boot/loader.conf</filename> is edited to load
+	it:</para>
+
+      <programlisting>geom_mirror_load="YES"</programlisting>
+
+      <para>Filesystems from the original disk can now be copied onto
+	the mirror with &man.dump.8; and &man.restore.8;.  Note that
+	it may take some time to create a snapshot for each filesystem
+	by the &man.dump.8; command with <option>-L</option>
+	option.</para>
+
+      <screen>&prompt.root; <userinput>mount /dev/mirror/gm0s1a /mnt</userinput>
+&prompt.root; <userinput>dump -C16 -b64 -0aL -f - /    | (cd /mnt && restore -rf -)</userinput>
+&prompt.root; <userinput>mount /dev/mirror/gm0s1d /mnt/var</userinput>
+&prompt.root; <userinput>mount /dev/mirror/gm0s1e /mnt/usr</userinput>
+&prompt.root; <userinput>mount /dev/mirror/gm0s1f /mnt/data1</userinput>
+&prompt.root; <userinput>mount /dev/mirror/gm0s1g /mnt/data2</userinput>
+&prompt.root; <userinput>dump -C16 -b64 -0aL -f - /usr | (cd /mnt/usr && restore -rf -)</userinput>
+&prompt.root; <userinput>dump -C16 -b64 -0aL -f - /var | (cd /mnt/var && restore -rf -)</userinput>
+&prompt.root; <userinput>dump -C16 -b64 -0aL -f - /data1 | (cd /mnt/data1 && restore -rf -)</userinput>
+&prompt.root; <userinput>dump -C16 -b64 -0aL -f - /data2 | (cd /mnt/data2 && restore -rf -)</userinput></screen>
+
+      <para>Reboot the system.  If everything works fine, the system
+	should boot by using <devicename>mirror/gm0</devicename>.  The
+	<devicename>mirror/gm0</devicename> should have the same
+	contents as the <devicename>ada0</devicename> had.  Just after
+	the reboot, the system is loaded from
+	<devicename>mirror/gm0</devicename> and
+	<devicename>ada0</devicename> is no longer used.  Note that
+	the mirror still consists of <devicename>ada1</devicename>
+	only.</para>
+
+      <para>If the boot stopped with the following message, there
+	was something wrong.</para>
+
+      <screen>Mounting from ufs:/dev/mirror/gm0s1a failed with error 19.
+
+Loader variables:
+  vfs.root.mountfrom=ufs:/dev/mirror/gm0s1a
+  vfs.root.mountfrom.options=rw
+
+Manual root filesystem specification:
+  &lt;fstype&gt;:&lt;device&gt; [options]
+      Mount &lt;device&gt; using filesystem &lt;fstype&gt;
+      and with the specified (optional) option list.
+
+    eg. ufs:/dev/da0s1a
+        zfs:tank
+        cd9660:/dev/acd0 ro
+          (which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)
+
+  ?               List valid disk boot devices
+  .               Yield 1 second (for background tasks)
+  &lt;empty line&gt;    Abort manual input
+
+mountroot&gt;</screen>
+
+      <para>To recover from this, enter
+	<literal>ufs:/dev/ada0s1a</literal> to the prompt.  Although
+	the system should boot by using <devicename>ada0</devicename>,
+	another prompt to select a shell program appears because
+	<filename>/etc/fstab</filename> is incorrect.  Just hit the
+	enter key at the prompt.  It is possible to restore the
+	modifications so far by reverting
+	<filename>/etc/fstab</filename>.  Reboot the system and try
+	the procedure again.</para>
+
+      <screen>Enter full pathname of shell or RETURN for /bin/sh:
+&prompt.root; <userinput>cp /etc/fstab.orig /etc/fstab</userinput>
+&prompt.root; <userinput>reboot</userinput></screen>
+
+      <para>If the system booted on
+	<devicename>mirror/gm0</devicename> successfully, the final
+	step is adding <devicename>ada0</devicename> as a spare disk.</para>
+
+      <important>
+	<para>Adding <devicename>ada0</devicename> as a spare disk
+	  means the contents on that disk will be erased completely.
+	  Double-check if <devicename>mirror/gm0</devicename> has the
+	  same contents as <devicename>ada0</devicename>.  If there is
+	  something wrong with the contents copied by &man.dump.8; and
+	  &man.restore.8;, revert <filename>/etc/fstab</filename>,
+	  reboot, and try the whole procedure again.</para>
+      </important>
+
+      <screen>&prompt.root; <userinput>gmirror insert mirror/gm0 /dev/ada0</userinput>
+GEOM_MIRROR: Device gm0: rebuilding provider ada0</screen>
+
+      <para>Synchronization between the two disks will start
+	immediately.  The &man.gmirror.8; <command>status</command>
+	command shows the progress.</para>
+
+      <screen>&prompt.root; <userinput>gmirror status</userinput>
+      Name    Status  Components
+mirror/gm0  DEGRADED  ada1 (ACTIVE)
+                      ada0 (SYNCHIRONIZING, 64%)</screen>
+
+      <para>After a while, the synchronization will finish.</para>
+
+      <screen>GEOM_MIRROR: Device gm0: rebuilding provider ada0 finished.
+&prompt.root; <userinput>gmirror status</userinput>
+      Name    Status  Components
+mirror/gm0  COMPLETE  ada1 (ACTIVE)
+                      ada0 (ACTIVE)</screen>
+
+      <para>The <devicename>mirror/gm0</devicename> now consists of
+	the two disks, <devicename>ada0</devicename> and
+	<devicename>ada1</devicename>, and the contents are
+	automatically synchronized with each other.
+	In use, the mirror <devicename>mirror/gm0</devicename>
+	will behave just like the original single drive.</para>
     </sect2>
 
     <sect2>
