First aid if VCSA root partition turns out to be too small
I recent times I frequently see vCenter server appliances (VCSA), whose root partitions ran out of free space. As a result services are unable to start after reboot. There are some tricks to free some space on root but on the long run you should increase the partition size.
Sounds simple – but it’s quite tricky and a bit dangerous. Don’t try this at home! 😉
Find out device
We first have to find out, on which device our root partition is located.
vc:~ # lsblk NAME MAJ:MIN RM SIZE RO MOUNTPOINT sda 8:0 0 20G 0 ├─sda1 8:1 0 132M 0 /boot ├─sda2 8:2 0 1G 0 [SWAP] └─sda3 8:3 0 10.9G 0 / sdb 8:16 0 1.4G 0 sdc 8:32 0 25G 0 └─swap_vg-swap1 (dm-8) 253:8 0 25G 0 [SWAP] sdd 8:48 0 50G 0 └─core_vg-core (dm-7) 253:7 0 50G 0 /storage/core sde 8:64 0 10G 0 └─log_vg-log (dm-6) 253:6 0 10G 0 /storage/log sdf 8:80 0 10G 0 └─db_vg-db (dm-5) 253:5 0 10G 0 /storage/db sdg 8:96 0 5G 0 └─dblog_vg-dblog (dm-4) 253:4 0 5G 0 /storage/dblog sdh 8:112 0 25G 0 └─seat_vg-seat (dm-3) 253:3 0 25G 0 /storage/seat sdi 8:128 0 1G 0 └─netdump_vg-netdump (dm-2) 253:2 0 1016M 0 /storage/netdump sdj 8:144 0 10G 0 └─autodeploy_vg-autodeploy (dm-1) 253:1 0 10G 0 /storage/autodeploy sdk 8:160 0 10G 0 └─invsvc_vg-invsvc (dm-0) 253:0 0 10G 0 /storage/invsvc fd0 2:0 1 4K 0 sr0 11:0 1 1024M 0
As we can see it is located on sda3. Now we have to find the corresponding SCSI node.
vc:~ # lsscsi [0:0:0:0] disk VMware Virtual disk 1.0 /dev/sda [0:0:1:0] disk VMware Virtual disk 1.0 /dev/sdb [0:0:2:0] disk VMware Virtual disk 1.0 /dev/sdc [0:0:3:0] disk VMware Virtual disk 1.0 /dev/sdd [0:0:4:0] disk VMware Virtual disk 1.0 /dev/sde [0:0:5:0] disk VMware Virtual disk 1.0 /dev/sdf [0:0:6:0] disk VMware Virtual disk 1.0 /dev/sdg [0:0:8:0] disk VMware Virtual disk 1.0 /dev/sdh [0:0:10:0] disk VMware Virtual disk 1.0 /dev/sdi [0:0:12:0] disk VMware Virtual disk 1.0 /dev/sdj [0:0:14:0] disk VMware Virtual disk 1.0 /dev/sdk [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR00 1.00 /dev/sr0
We can see sda is on the first SCSI node (which is not surprising).
Increase VMDK size
The default size of vDisk1 is 11 GB. We will enlarge that to 20 GB. At this point our VM mustn’t have a snapshot. Handy functions like LVM and autogrow don’t work on the root partition.
We need to restart the VM to apply changed sizes of vDisk1.
shutdown -r now
Let’s have a look on device sda
fdisk -l /dev/sda
vc:~ # fdisk -l /dev/sda Disk /dev/sda: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x547c469c Device Boot Start End Blocks Id System /dev/sda1 2048 272383 135168 83 Linux /dev/sda2 272384 2377727 1052672 82 Linux swap / Solaris /dev/sda3 * 2377728 25165823 11394048 83 Linux
An asterisk indicates sda a bootable device.
How much free space do we have?
vc:~ # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 11G 5.1G 5.1G 51% / udev 7.9G 164K 7.9G 1% /dev tmpfs 7.9G 40K 7.9G 1% /dev/shm /dev/sda1 128M 38M 84M 31% /boot /dev/mapper/core_vg-core 50G 185M 47G 1% /storage/core /dev/mapper/log_vg-log 9.9G 8.7G 695M 93% /storage/log /dev/mapper/db_vg-db 9.9G 4.0G 5.4G 43% /storage/db /dev/mapper/dblog_vg-dblog 5.0G 2.8G 1.9G 60% /storage/dblog /dev/mapper/seat_vg-seat 25G 20G 4.1G 83% /storage/seat /dev/mapper/netdump_vg-netdump 1001M 18M 932M 2% /storage/netdump /dev/mapper/autodeploy_vg-autodeploy 9.9G 151M 9.2G 2% /storage/autodeploy /dev/mapper/invsvc_vg-invsvc 9.9G 297M 9.1G 4% /storage/invsvc
The size of sda is still 11GB, yet the disk is bigger.
Now it’s time to make a snapshot of the VM. Those of you who are using Veeam Backup & Replication should also create a replicate of vCenter.
Use all instruction on your own risk
We’re going to destroy partition sda3 and rebuild it in new boundaries. If something goes wrong, your vCenter is lost. Read the last sentence again.
Delete partition sda3
Partition /dev/sda3 will be deleted by pressing [d] and partition number .
Command (m for help): d Partition number: 3
Partition table hasn’t been written yet. It is still possible to cancel with Ctrl-C.
Now we’ll create a new partition [n]. Type is primary [p]. Number is . From now on we accept default values.
Command (m for help): n Command action e extended p primary partition (1-4)
Make sda3 bootable
The new partition sda3 is not yet bootable. We can achieve that by setting the bootable flag [a] and choose partition number .
Command (m for help): a Partition number (1-4): 3
Check partition table
With the command print [p] we can have a look at the partition table.
Command (m for help): p
It’s time to make changes persistent by writing our changes [w].
Command (m for help): w
We need a reboot to apply changes.
shutdown -r now
After reboot we can reconnect by SSH or console.
The filesystem still has its old size. Only the partition has grown. We can change that by resize2fs.
df -h /dev/sda3
If everything works as expected, don’t forget to delete the snapshot. 😉