Saturday, October 12, 2013

How to increase ZFS root pool


In ZFS, you cannot extend the root pool by adding new disks. But it has some logic too. For an example, if you are having more than one hard disk in root zpool, loss of one disk result be un-bootable system. To eliminate these kind of situations ,its better to keep the rpool in one disk and mirror it instead of spreading over the multiple disks.

But sometimes project team will commit mistake by keeping /var, /usr, /home under rpool and system may running out of root disk space.

EFI labels are not supported for Solaris rpool,rpool disk must be SMI labeled and all the sectors to be set on partition 0 like the below one.
Part      Tag    Flag     First Sector       Size       Last Sector
  0        usr    wm               256   1015.86MB        2080733
  1 unassigned    wm                 0         0             0
  2 unassigned    wm                 0         0             0
  3 unassigned    wm                 0         0             0
  4 unassigned    wm                 0         0             0
  5 unassigned    wm                 0         0             0
  6 unassigned    wm                 0         0             0
  8   reserved    wm           2080734      8.00MB        2097117
For a test, Here i am trying to extend the rpool using new disk,
# zpool add rpool c1t8d0s0
cannot add to 'rpool': root pool can not have multiple vdevs or separate logs

Solution:

There is only one way to extend the rpool space. Please see the below steps:
1.Add a bigger size root hard disk.(i.e. If current root disk is 40GB,then add 80GB hard disk)
2.Mirror the rpool using the new 80GB hard disk
3.Installboot block in new disk
4.Detach the old disk from rpool

My setup:
# zpool list rpool
NAME    SIZE  ALLOC   FREE    CAP  HEALTH  ALTROOT
rpool  11.9G  8.56G  3.38G    71%  ONLINE  -
# zpool status rpool
  pool: rpool
 state: ONLINE
 scrub: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t0d0s0  ONLINE       0     0     0
errors: No known data errors

Here i am going to extend my rpool to 20GB. My new 20GB hard disk is c1t9d0.
# format c1t9d0
selecting c1t9d0
[disk formatted]
FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> p
WARNING - This disk may be in use by an application that has
          modified the fdisk table. Ensure that this disk is
          not currently in use before proceeding to use fdisk.
format> fdisk
No fdisk table exists. The default partition for the disk is:
  a 100% "SOLARIS System" partition
Type "y" to accept the default partition,  otherwise type "n" to edit the
 partition table.
y
format> p
PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition>
partition> 0
Part      Tag    Flag     Cylinders        Size            Blocks
  0 unassigned    wm       0               0         (0/0/0)           0

Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]:1
Enter partition size[0b, 0c, 0e, 0.00mb, 0.00gb]: 2606c
partition> p
Current partition table (unnamed):
Total disk cylinders available: 2607 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders        Size            Blocks
  0 unassigned    wm       1 - 2605       19.96GB    (2606/0/0) 41865390
  1 unassigned    wm       0               0         (0/0/0)           0
  2     backup    wu       0 - 2606       19.97GB    (2607/0/0) 41881455
  3 unassigned    wm       0               0         (0/0/0)           0
  4 unassigned    wm       0               0         (0/0/0)           0
  5 unassigned    wm       0               0         (0/0/0)           0
  6 unassigned    wm       0               0         (0/0/0)           0
  7 unassigned    wm       0               0         (0/0/0)           0
  8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
  9 unassigned    wm       0               0         (0/0/0)           0
partition> l
Ready to label disk, continue? y
partition> q
format> q
#


Now we are going to mirror the rpool with new disk.
# zpool attach rpool c1t0d0s0 c1t9d0s0
Please be sure to invoke installgrub(1M) to make 'c1t9d0s0' bootable.
Make sure to wait until resilver is done before rebooting.
bash-3.00# zpool status rpool
  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 0.08% done, 7h6m to go
config:
        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c1t0d0s0  ONLINE       0     0     0
            c1t9d0s0  ONLINE       0     0     0  6.82M resilvered
errors: No known data errors

Once the mirroring is done, we are good to detach the old disk from zpool.But make sure you have made the new disk bootable by updating boot block.
bash-3.00# zpool detach rpool c1t0d0s0
bash-3.00# zpool status rpool
  pool: rpool
 state: ONLINE
 scrub: resilver completed after 0h31m with 0 errors on Tue May  7 02:05:56 2013
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t9d0s0  ONLINE       0     0     0  8.56G resilvered

errors: No known data errors
bash-3.00# df -h /
Filesystem             size   used  avail capacity  Mounted on
rpool/ROOT/rpooldataset
                        12G   5.6G   1.6G    78%    /
bash-3.00# installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c1t9d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 273 sectors starting at 50 (abs 16115)

Installgrub will work only on X86 servers. For SPARC serves,please use installboot.
# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t9d0s0

You may wondering after doing all the stuff's root disk space is not increased ? You have to do one small thing to activate the trick. Perform the scrub on rpool and set autoexpand to get the new size.
# zpool scrub rpool
# zpool status rpool
  pool: rpool
 state: ONLINE
 scrub: scrub in progress for 0h9m, 70.77% done, 0h4m to go
config:
        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t9d0s0  ONLINE       0     0     0

# zpool status rpool
  pool: rpool
 state: ONLINE
 scrub: scrub completed after 0h19m with 0 errors on Tue May  7 02:36:21 2013
config:
        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t9d0s0  ONLINE       0     0     0

errors: No known data errors
bash-3.00# df -h /
Filesystem             size   used  avail capacity  Mounted on
rpool/ROOT/rpooldataset
                        12G   5.6G   1.6G    78%    /
bash-3.00# zpool set autoexpand=on rpool
bash-3.00# df -h /
Filesystem             size   used  avail capacity  Mounted on
rpool/ROOT/rpooldataset
                        20G   5.6G   9.5G    38%    /
Now you can see root filesystem has been extended to 20GB.

VxVM commands with examples


Disk group related tasks


vxdiskadd or vxdiskadm
vxdiskadd and vxdiskadm are interactive programs. It may ask you series of questions.
Answer to these questions according to your requirement.

# vxdiskadd [ all | c0t0d0 c0t1d0]
OR
# vxdiskadm

To add disk c1t1d0 to a diskgroup
# vxdiskadd c1t1d0 ( answer the questions appropriately)
or
# vxdiskadm

To find out maximum size that a volume could be created in the disk group (free space in a DG)

# vxassist -g appdg maxsize
Maximum volume size: 31170560 (15220Mb)

To list the available diskgroups

# vxdg list
NAME STATE ID
tmpdg enabled,cds 1144189106.24.ultra5
testdg enabled,cds 1144135936.13.ultra5

To remove disk disk01 from diskgroup dg01
# vxdg –g dg01 rmdisk disk01
To get the detailed information about dg01 diskgroup
# vxdg list dg01
To display free space available in each disk in a disk group “dg01”
# vxdg -g tmpdg free
DISK DEVICE TAG OFFSET LENGTH FLAGS
tmpdg01 c1t2d0s2 c1t2d0 4194304 31171664 -
tmpdg02 c1t3d0s2 c1t3d0 0 35365968 -
The actual space available is LENGTH in 512b
To deport a diskgroup dg01
# vxdg deport dg01
To import a diskgroup dg01
# vxdg import dg01
To import a diskgroup with a newname
# vxdg -n <new_dgname> import <old_dgname>
To destroy a diskgroup dg01
# vxdg destroy dg01

Disk related tasks

To display the disks information
# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 auto:none – – online invalid
c1t2d0s2 auto:cdsdisk tmpdg01 tmpdg online
c1t3d0s2 auto:cdsdisk tmpdg02 tmpdg online
c1t4d0s2 auto:none – – online invalid
c1t5d0s2 auto:cdsdisk testdg01 testdg online
c1t10d0s2 auto:none – – online invalid
c1t11d0s2 auto:none – – online invalid
c1t12d0s2 auto:none – – online invalid
To remove the disk c1t1d0 from volume manager control
# vxdisk rm c1t1d0
To make a disk as spare for hot relocation spare
# vxedit set spare=on disk03
To remove a spare disk disk03 from a hot relocation spare
# vxedit set spare=off disk03
To rename disk01 to disk03
# vxedit rename disk01 disk03
To make disk01 offline
# vxdisk offline disk01
To display detailed disk information about disk01
# vxdisk -g tmpdg list testdg01
VxVM vxdisk ERROR V-5-1-558 Disk testdg01: Disk not in the configuration ultra5 [/tmpfs]
 # vxdisk -g tmpdg list tmpdg01
Device: c1t2d0s2
devicetag: c1t2d0
type: auto
hostid: ultra5
disk: name=tmpdg01 id=1144189105.22.ultra5
group: name=tmpdg id=1144189106.24.ultra5
info: format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/c1t2d0s2 char=/dev/vx/rdmp/c1t2d0s2
version: 3.1
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=2 offset=2304 len=35365968 disk_offset=0
private: slice=2 offset=256 len=2048 disk_offset=0
update: time=1144283192 seqno=0.7
ssb: actual_seqno=0.0
headers: 0 240
configs: count=1 len=1280
logs: count=1 len=192
Defined regions:
config priv 000048-000239[000192]: copy=01 offset=000000 enabled
config priv 000256-001343[001088]: copy=01 offset=000192 enabled
log priv 001344-001535[000192]: copy=01 offset=000000 enabled
lockrgn priv 001536-001679[000144]: part=00 offset=000000
Multipathing information:
numpaths: 1
c1t2d0s2 state=enabled
To find out the free space in a particular disk “tmpdg02″
# vxdg -g tmpdg free tmpdg01
DISK DEVICE TAG OFFSET LENGTH FLAGS
tmpdg01 c1t2d0s2 c1t2d0 4194304 31171664 -
The actual space available is LENGTH in 512b
To display multipath to a VM Disk
# vxdisk path
SUBPATH DANAME DMNAME GROUP STATE
c1t0d0s2 c1t0d0s2 mydg01 mydg ENABLED
c4t0d0s2 c1t0d0s2 mydg01 mydg ENABLED
c1t1d0s2 c1t1d0s2 mydg02 mydg ENABLED
c4t1d0s2 c1t1d0s2 mydg02 mydg ENABLED
vxdisk list command also shows the multipathing information

Sub-Disk related tasks

To create a subdisk disk01-01 on disk disk01 of size 500MB
# vxmake sd disk01-01 disk01,0,500m
Note: If a second subdisk is created on the same vmdisk, the offset value 0 can not be
given. It will overlap with the existing subdisk. We can find out the free offset value
by giving the command vxdg –g (group-name) free which is used to find out the free
disk space in diskgroups
To associate subdisks disk01-01, disk02-01 to an existing plex plex01
# vxsd assoc plex01 disk01-01 disk02-01
To dissociate subdisk disk01-01 from a plex
# vxsd dis disk01-01
To remove a subdisk sd01
# vxedit rm sd01
To move data from subdisk sd01 to subdisk sd02
# vxsd mv sd01 sd02
To join two subdisks sd01 and sd02 and make it as single subdisk sd03
# vxsd join sd01 sd02 sd03
To split subdisk sd01 into two subdisks called sd04 sd05
# vxsd –s size split sd01 sd04 sd04 (-s zize – size of the first subdisk to be created)
Note:- if sub disk sd01 is associated with a plex before split , after split,
both subdisks will be associated with the same plex. become
To display detailed information about all subdisks
# vxprint –ls

Plex related tasks


To create a plex and associate subdisks disk01-01 and disk03-01 to the plex
# vxmake plex plex01 sd=disk01-01,disk03-01
To create a striped plex plex01
# vxmake plex plex01 layout=stripe stwidth=32 ncolumn=2 sd=disk01-01,disk02-01
To create a RAID5 plex called plex01
# vxmake plex plex01 layout=raid5 stwidth=32 ncolumn=2 sd=disk01-01, disk02-01, disk03-01
To associate (attach) a plex to an existing volume
# vxplex att vol01 plex01
To create a new volume and associate plex to that volume
# vxmake –U fsgen vol vol01 plex=plex01,plex02
To Dissociate a plex from a volume
# vxplex dis plex02
To temporarily detach a plex from a volume
# vxplex det plex02
To remove a plex
# vxedit -r rm plex02
To dissociate and remove a plex from a volume (removing mirror)
# vxplex –o rm dis plex02
To display detailed information about all plexes
# vxprint –lp
To display detailed information about a specific plex
# vxprint –l plex01
To make a plex called plex01 to clean state from stale state
# vxmend fix clean plex01
To make a plex from the disabled state to stale state
# vxmend on plex01

Volume related tasks

To create a volume vol01 with attached plexes plex01 & plex02
# vxmake –U fsgen vol vol01 plex=plex01,plex02
Note:- -U Usage type. It should be “fsgen”, if file system to be created on that volume
 To enable (start) a volume
# vxvol start vol01
 To disable (stop) a volume
# vxvol stop vol01
 To stop all enabled volumes
# vxvol stopall
 To set the state of the volume vol01 to CLEAN
# vxvol init clean vol01
 To display detailed information about all volumes
# vxprint –lv
 To start a volume vol01, vol02 which can not be enabled
# vxrecover –s vol01 vol02
 To increase or decrease the size of the volume & file system to 500MB
# vxresize vol01 500m
 To list all the tasks currently running on the system
# vxtask list
 To remove the volume appvol1
# vxedit -g appvg -rf rm appvol1
vxprint
vxprint command is used to display the various information about disk, diskgroups, subdisk,plex, volume and etc… It displays the informatin from Veritas volume manager configuration.

To display all the available diskgroups
# vxprint -G
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg dg01 dg01 – – – – – -
dg dg02 dg02 – – – – – -

To display details about a diskgroup
# vxprint -g dg01
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg dg01 dg01 – – – – – -
dm disk01 c1t2d0s2 – 35365968 – – – -
dm disk02 c1t3d0s2 – 35365968 – – – -
v vol01 fsgen DISABLED 2097152 – ACTIVE – -
pl vol01-01 vol01 DISABLED 2097152 – ACTIVE – -
sd sd01 vol01-01 ENABLED 2097152 0 – –
-
TO display short note of volume vol1
# vxprint -g dg01 -v vol01
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v vol01 fsgen DISABLED 2097152 – ACTIVE – -

To get the detailed output about a volume
# vxprint -g dg01 -lv vol01
Volume: vol01
info: len=2097152
type: usetype=fsgen
state: state=ACTIVE kernel=DISABLED cdsrecovery=0/0 (clean)
assoc: plexes=vol01-01
policies: read=SELECT (round-robin) exceptions=NO_OP
flags: closed writeback
logging: type=REGION loglen=0 serial=0/0 mapalign=0 maplen=0 (disabled)
apprecov: seqno=0/0
recovery: mode=default
recov_id=0
device: minor=34000 bdev=272/34000 cdev=272/34000 path=/dev/vx/dsk/dg01/vol01
perms: user=root group=root mode=0600
guid: {5d61a4fe-1dd2-11b2-8e6d-080020b5a36b}

vxedit
The vxedit utility can be used to rename/remove the disks, subdisks, plex and volume
information from the veritas configuration.
vxedit [-dpPsvV ] [-g diskgroup] rename oldname newname
vxedit [-dfpPrsvV ] [-g diskgroup] rm name…
vxedit [-dfGpPrsvV ] [-e pattern] [-g diskgroup] set
attribute=value… [name...]
where:
-d for disk operation
-p for plex operation
-s for subdisk operation
-v for volume operation
-G for Disk group operation
-f Force an operation
-r Operate recursively on records associated with the selected records
For selected volume records, this affects associated plex, subdisk,and subvolumes records. For selected plex records, this affects associated subdisk and subvolume records.

To remove appvol1 including the associated plex and subdisks
# vxedit -g appvg -rf rm appvol1
Note: if the above command tried with out the ‘-r’ option, it will fail saying there are
associated plexes.

# vxedit -g dg01 -f rm vol01
VxVM vxedit ERROR V-5-1-1189 Volume vol01 has associated plexes

To remove a plex plex01 along with associated sub-disks
# vxedit -g appvg -rf rm plex01

To rename a plex name from plex-old to plex-new
# vxedit -g dg01 -p rename plex-old plex-new

 Useful commands

:vxtask command is used to administer operations on VxVM tasks that are running on
the system. Operations include listing tasks, modifying the state of a task (pausing, resuming,
aborting) and modifying the rate of progress of a task.
vxtask abort taskid
vxtask pause taskid
vxtask resume taskid

To list all tasks currently running on the system
# vxtask list

To print tasks hierarchically, with child tasks following the parent tasks
# vxtask -h list

LINUX Vxdisk command tips
# vxdisk list

DEVICE TYPE DISK GROUP STATUS
sda auto:sliced rootdisk rootdg online
sdb auto:sliced appdg01 appdg online

# vxdisk list sda
Device: sda
devicetag: sda
type: auto
hostid: system7044
disk: name=rootdisk id=1188098145.7.system7044
group: name=rootdg id=1188098145.9.sytem7044
info: format=sliced,privoffset=1,pubslice=3,privslice=6
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/sda3 char=/dev/vx/rdmp/sda3
privpaths: block=/dev/vx/dmp/sda6 char=/dev/vx/rdmp/sda6
version: 2.1
iosize: min=512 (bytes) max=256 (blocks)
public: slice=3 offset=0 len=71665902 disk_offset=63
private: slice=6 offset=1 len=2047 disk_offset=71663917
update: time=1188098305 seqno=0.8
ssb: actual_seqno=0.0
headers: 0 248
configs: count=1 len=1481
logs: count=1 len=224
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-001498[001250]: copy=01 offset=000231 enabled
log priv 001499-001722[000224]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 2
sda state=enabled
sdc state=enabled