0

We are having an Ubuntu server with 1 HDD in a volume group, this HDD got filled,so added an iscsi disk , before adding to the volume group, did restart and tested and the iscsi disk was showing under disk -l , now added this disk to the volume group and expanded the logical volume,and all was working fine until the server restarted, once the server restarted it will not boot ,it cannot mount the root volume as the iscsi disk is not visible I think hence the volume group is not coming online.and get to initramfs

What to do?

I tried using the live cd and was able to connect to iscsi manually and was able to see the volume group and the physical disks especially the iscsi one. and was able to mount the volume group to see it's contents.

But how to fix the booting issue , how to check where the iscsi is visible during boot, how to make it available so that it can mount it for / to work

Please help

Improve this question

1 Answer 1

Reset to default
0

You will need to load the drivers into the kernel, and also ensure you have the mod file needed in grub.

I have never tried to combine different types of block devices into the one volume group, and I would strongly recommend you do not mix types of drives in the one VG.

In your case it is also more difficult, because iscsi, is a layer 3 protocol block device. I am not sure you would be able to get the iscsi initiator to connect to the drive, and then lvm to bring up the VG. I may be wrong, but I dont think it can be done easily, and would not be stable even if it was done.

I would suggest that you use a live cd as you have done, boot and mount the current VG. Ideally move some files to a usb drive, or network store temporally. Then resize the logical volume and the VG back to how it was.

Create a new pv/vg and logical volume for the iscsi drive, and then create the partitions you require and mount those partitions in non-boot required directories, eg /home or if it is a web server maybe /var/www. This will allow for the server to boot with only the main drive, and then during the boot processes, assuming iscsi config is saved, it will mount the non critical drive.

The main directories the server needs to be boot are: /etc, /usr/sbin, /usr/bin, /var/, /usr/, /lib/

The /usr/sbin and /usr/bin gets mounted via symbolic link to /sbin and /bin

If you /boot is not on its own partition, then that would also need to be on the main lvm.

Also ensure you fstab, has the partitions on the iscsi drive listed, once you have moved the data over.

Once you have booted off the live CD mount the volume group as you have it currently. to /mnt/newroot

Then run:

mount -R /run /mnt/newroot/run
mount -R /dev /mnt/newroot/dev
mount -R /proc /mnt/newroot/proc
mount -R /sys /mnt/newroot/sys

chroot /mnt/newroot

This will change the root be the root of the Volume group, you will then be able to access all your directories etc as if you have just booted into that os.

run mount -a

This will mount all the mount points that are in the brokensetup.

you will need to add the options in the kernel line of grub similar to, the easiest way is to add the line, into /etc/default/grub as the GRUB_CMDLINE_LINUX_DEFAULT for example:

GRUB_CMDLINE_LINUX_DEFAULT="ISCSI_INITIATOR=iqn.2024-06.au.com:client ISCSI_TARGET_NAME=iqn.2024-06.au.com:server ISCSI_TARGET_IP=10.0.0.250 ISCSI_TARGET_PORT=3260  root=UUID=6c816f51-0613-45e7-a15b-bc2d5cd00f88 ip=10.0.0.100::10.0.0.1:255.255.255.0:client:eth0:off "

ISCSI server in this example is 10.0.0.250
ISCSI client is 10.0.0.100
Gateway 10.0.0.1 
Subnet 255.255.255.0

You will need to change these to match your case or use DHCP

The ip variable sets a static IP for the machine, if you want to use dhcp, you need to have ip=dhcp the root=UID=

Once you do all these changes, you will need to run update-grub, and cross your fingers.

This should initiate iscsi before trying to running the lvm provision script, and hopefully, it will allow you to work as normal.

**note this is theory only, as stated earlier, I have never tried to have a vg, with a iscsi drive, and a internal SATA/SAS drive.

Improve this answer
25
  • Thank you Luke for the response, could you please let me know how to do this part using live Cd?(You will need to load the drivers into the kernel, and also ensure you have the mod file needed in grub.) Also read somewhere that adding _netdev in fstab will it make any difference? Hope not ,as in fstab see only the entry for the volume group,boot I think not for this iscsi disk,but was wondering no way this iscsi could get connected before trying to load / ? Like any script? .iscsi settings iscsiconfig are made to connect automatically, but again that can happen only after loading /? plz help.
    – JoBo
    Commented Jun 16 at 10:33
  • I have updated the main answer, it will be interesting to see how you go. adding netdev to fstab wont help, because you need iscsi to discover before LVM is provisioned. If we are able to get that to work, then fstab just needs to mount the LVM partition. Hopefully it will work. Alternatively you could try using a ipxe or pxe script that uses sanboot, to initiate the iscsi drive, and then it should mount with lvm as normal, but that can be a little tricky, and the exact way will change based on the hardware/manufacturer.
    – Luke Attard
    Commented Jun 16 at 18:01
  • Thank you very much will try this,to clarify root=UID= I should get the / uuid from the fstab right? Which is currently failing to load and this IP that we are setting does it work as pxe only?
    – JoBo
    Commented Jun 16 at 21:26
  • If you are using pxe only, then I would chainload ipxe, then use sanboot to initiate iscsi mount, then LVM should be able to see it, and your grub will work fine.
    – Luke Attard
    Commented Jun 18 at 2:32
  • You said you could load from a live cd, then mount the iscsi, and then see the VG? If you do this, then chroot to the mounted VG, you will be able to look at mtab (the live running version of fstab) and get the uuid from there, or you can just look in /dev/disk/by-uuid/. If you do a ls -la /dev/disk/by-uuid/ you will see what uuid links to what drive
    – Luke Attard
    Commented Jun 18 at 2:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .