Update: You can find a much better version of this as an article I wrote for the Odroid Magazine:https://magazine.odroid.com/article/linux-logical-volume-manager-lvm2/

I’ve been planning to upgrade my LVM2 2 disk striped setup with a new disk for a while. I am currently on the testing phase but I think I have a good way to do it without losing any data.

I plan to implement this on my Odroid XU4 based Cloudshell 2 which is running Ubuntu 16.0.4.

I used this guide as inspiration: LVM fun

I recommend reading that guide first if you are not familiar with LVM.

Phase 1: Testing with loop devices

The plan is as follows:

• Connect new spare disk via the second USB3.0 port using an external enclosure(the cloudshell only supports 2 SATA drives and both ports are occupied right now)
• Create a PV(physical volume) on the new disk
• Add new PV to existing VG(volume group)
• unmount all VG volumes and/or freeze allocation on the PV to migrate
• pvmove one of the 2 existing PVs onto this new PV
• Leave it overnight since it is going to copy sector by sector a 2 TB disk
• Reduce VG by removing old PV(the one moved at the previous step)
• Shutdown and swap out old disk with new one
• Boot and check that the LVs(logical volumes) are correctly mapped to the PVs

OK, here is how I tested.

First, create 3 loopback devices backed by 3 empty files.

dd if=/dev/zero of=/tmp/hdd1.img bs=1G count=1
dd if=/dev/zero of=/tmp/hdd2.img bs=1G count=1
# twice the space
dd if=/dev/zero of=/tmp/hdd3.img bs=1G count=2

losetup -f

losetup /dev/loop0 /tmp/hdd1.img
losetup /dev/loop1 /tmp/hdd2.img
losetup /dev/loop2 /tmp/hdd3.img

Now, create PVs on all these devices

pvcreate /dev/loop0
pvcreate /dev/loop1
pvcreate /dev/loop2

Add the first 2 volumes to a VG:

vgcreate vgtest /dev/loop0 /dev/loop1

Check stuff with:

pvs
vgs

pvdisplay

Create 2 striped volumes as such:

lvcreate -i2 -I4 -L1G -nlvdata1 vgtest
lvcreate -i2 -I4 -l100%FREE -nlvdata2 vgtest

The first one will have 1Gb and it will be 50/50 split across the 2 PVs.

The second one will fill up the existing space and also be split between the 2 PVs(size will be around 2.9Gb).

OK, so this is the initial setup, we have 2 disks with 2 logical volumes spread among them using striping.

Format the “disks” and write some files to them for testing( I used mkfs.ext4 on the virtual devices to create filesystems). When done, unmount them and let’s migrate the data.

Migrating the data to disk 3 from disk 2:

# Add new device to VG
vgextend vgtest /dev/loop2

# Migrate data
pvmove /dev/loop1 /dev/loop2

In the real world this is the step that takes forever. I noticed that if  interrupted there is a good chance that the process can be resumed. So power outages might not result in data loss.

Resize PV to fill space on /dev/loop2(it has a size of 2GB)

pvresize /dev/loop2

Bonus round: create an unequal stripe by using the extra space at the end of /dev/loop2(perfomance loss, auch…)

lvextend -i1 -l +100%FREE /dev/vgtest/lvdata2

You will end up with this setup:

lvdisplay -m /dev/vgtest/lvdata2
  --- Logical volume ---
  LV Path                /dev/vgtest/lvdata2
  LV Name                lvdata2
  VG Name                vgtest
  LV UUID                vDefWQ-1ugy-1Sp5-T1JL-8RQo-BWqJ-Sldyr2
  LV Write Access        read/write
  LV Creation host, time odroid, 2018-03-06 17:43:44 +0000
  LV Status              available
  # open                 0
  LV Size                1.99 GiB
  Current LE             510
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:2

  --- Segments ---
  Logical extents 0 to 253:
    Type                striped
    Stripes             2
    Stripe size         4.00 KiB
    Stripe 0:
      Physical volume   /dev/loop0
      Physical extents  128 to 254
    Stripe 1:
      Physical volume   /dev/loop2
      Physical extents  128 to 254

  Logical extents 254 to 509:
    Type                linear
    Physical volume     /dev/loop2
    Physical extents    255 to 510

 

Now, we are going to throw away /dev/loop1.

vgreduce vgtest /dev/loop1

losetup -d /dev/loop1

In the real world this would be: unmap the PV and pull the physical disk, this is where you edit your fstab so the OS does not try to mount automatically(for now) and reboot.

We are now going to simulate a device name reallocation:

# Disk 3 becomes offline
losetup -d /dev/loop2

# Disk comes back as different device
losetup /dev/loop1 /tmp/hdd3.img

If you run pvs, vgs and lvs your volumes should be intact!

Now mount the volumes and check that your test files are still there.

In theory, this is the process but I have yet to try it on real data.

# pvs
  PV         VG         Fmt  Attr PSize    PFree
  /dev/loop0 vgtest     lvm2 a--  1020.00m    0
  /dev/loop1 vgtest     lvm2 a--     2.00g    0