LVM2 (Logical Volume Management) is pretty amazing, but when something goes wrong, it’s not easy to troubleshoot. This is not the fault of the tools, but a reflection that LVM is relatively new in Linux, and not widely understood.
What I Tried to Do
I tried to increase the size of a logical volume with the lvextend command:
lvextend --extents 100%FREE /dev/VolumeGroup1/var
This form of the command is supposed to use all of the free space in the volume group.
What Failed
The command responded with an error message that contained this text:
device-mapper: reload ioctl failed: Invalid argument
Running lvdisplay showed the following status for the volume:
LV Status suspended
LVM could not activate the volume with the vgchange command:
vgchange -ay
Attempts to run fsck.ext4 resulted in an error message about a bad superblock. The volume also became unresponsive, and the system would not run certain commands (even though the volume was not mounted, the device mapper process was probably corrupted). I also saw kernel messages like this in the system log:
kernel: device-mapper: device 8:16 too small for target
How I Undid the Damage
When you have an LVM problem, don’t panic. There is built-in undo/revert/restore feature in LVM2! Every time you change something, a file is automatically saved to /etc/lvm/archive (at least on an Ubuntu 14 system). I have NO IDEA why this feature is not more widely known! The command is vgcfgrestore:
vgcfgrestore --file /etc/lvm/archive/name_of_archive_file VolumeGroupName
NOTE: if you are running LVM commands when your system is booted from a recovery CD, the undo files are saved to the /etc directory on the RAM disk, which is wiped out when the system is rebooted. Be sure to copy all files from /etc/lvm/archive to another drive, such as a USB drive, so they will be available if you have to reboot.
How I Resized the Volume
Rodrigo Tamada figured out why lvextend sometimes fails when it attempts to automatically determine the amount of free space on the volume. The problem is that sometimes LVM does not correctly calculate the amount of free space on the physical volume. LVM then extends the logical volume beyond the extents of the physical volume. I followed the procedure suggested by Rodrigo, and it worked. First, look at the output of the lvdisplay command to find the name of the physical volume that the logical volume resides on. Then, run the pvresize command, no size specified, to change the physical volume size to match the size of the device:
pvresize /dev/mapper/your_volume_name
Run pvdisplay to verify that the physical volume size has changed. Now, you should be able to extend a volume to use all the available space.
your solution is fantastic.it helps a lot.
It literally saved me. Thanks a ton!
Great information, definitely something to have in mind before panic
vgcfgrestore saved my ass from a HUGE problem.. thanks!!! But I have 3 servers using the same PV (it’s a NAS) and I need to specify how much the LV should grow on each server. I can’t just let it take all available space. Any ideas?
Luis, the lvextend command has many options that allow you to specific the exact amount by which to extend the logical volume: https://linux.die.net/man/8/lvextend Is that what you’re asking?
Thanks Craig, this info is a life saver indeed. I wonder why lvm config restore option is not very popular among other forums too.