LVM device-mapper: reload ioctl failed: Invalid argument

LVM2 (Logical Volume Management) is pretty amazing, but when something goes wrong, it’s not easy to troubleshoot. This is not the fault of the tools, but a reflection that LVM is relatively new in Linux, and not widely understood.

What I Tried to Do

I tried to increase the size of a logical volume with the lvextend command:

lvextend --extents 100%FREE /dev/VolumeGroup1/var

This form of the command is supposed to use all of the free space in the volume group.

What Failed

The command responded with an error message that contained this text:

device-mapper: reload ioctl failed: Invalid argument

Running lvdisplay showed the following status for the volume:

LV Status    suspended

LVM could not activate the volume with the vgchange command:

vgchange -ay

Attempts to run fsck.ext4 resulted in an error message about a bad superblock. The volume also became unresponsive, and the system would not run certain commands (even though the volume was not mounted, the device mapper process was probably corrupted). I also saw kernel messages like this in the system log:

kernel: device-mapper: device 8:16 too small for target

How I Undid the Damage

When you have an LVM problem, don’t panic. There is built-in undo/revert/restore feature in LVM2! Every time you change something, a file is automatically saved to /etc/lvm/archive (at least on an Ubuntu 14 system). I have NO IDEA why this feature is not more widely known! The command is vgcfgrestore:

vgcfgrestore --file /etc/lvm/archive/name_of_archive_file VolumeGroupName

NOTE: if you are running LVM commands when your system is booted from a recovery CD, the undo files are saved to the /etc directory on the RAM disk, which is wiped out when the system is rebooted. Be sure to copy all files from /etc/lvm/archive to another drive, such as a USB drive, so they will be available if you have to reboot.

How I Resized the Volume

Rodrigo Tamada figured out why lvextend sometimes fails when it attempts to automatically determine the amount of free space on the volume. The problem is that sometimes LVM does not correctly calculate the amount of free space on the physical volume. LVM then extends the logical volume beyond the extents of the physical volume. I followed the procedure suggested by Rodrigo, and it worked. First, look at the output of the lvdisplay command to find the name of the physical volume that the logical volume resides on. Then, run the pvresize command, no size specified, to change the physical volume size to match the size of the device:

pvresize /dev/mapper/your_volume_name

Run pvdisplay to verify that the physical volume size has changed. Now, you should be able to extend a volume to use all the available space.

Comments

Comment #1 by william

william - Mar 1, 2016

your solution is fantastic.it helps a lot.

Comment #2 by Konstantin Khankin

Konstantin Khankin - May 3, 2016

It literally saved me. Thanks a ton!

Comment #3 by Jorge G

Jorge G - Jul 4, 2016

Great information, definitely something to have in mind before panic

Comment #4 by Luis

Luis - Apr 1, 2017

vgcfgrestore saved my ass from a HUGE problem.. thanks!!! But I have 3 servers using the same PV (it’s a NAS) and I need to specify how much the LV should grow on each server. I can’t just let it take all available space. Any ideas?

Comment #5 by Craig

Craig - Apr 5, 2017

Luis, the lvextend command has many options that allow you to specific the exact amount by which to extend the logical volume: https://linux.die.net/man/8/lvextend Is that what you’re asking?

Comment #6 by balpay

balpay - May 0, 2018

Thanks Craig, this info is a life saver indeed. I wonder why lvm config restore option is not very popular among other forums too.