VM still running when NFS datastore crash

Dear all,

I’m using NFS system datastore for my setup. The FreeNAS provide this NFS system. But one day, NFS server crash and hang, need to hard reboot. At the time NFS is hang, the vm is still running (linux & windows vm). When NFS server is back, Linux VM remount the disk on NFS as read-only and need to restart the vm and do check disk to fix error before run again. The problem is there are some vm cannot fix the disk correctly, so that it get error and cannot boot anymore.

So is there a way to make opennebula auto stop the vm if detect NFS server is down or the way to ensure the system running safely of NFS has to restart

Thank you all

Versions of the related components and OS (frontend, hypervisors, VMs):
I’m using:

  • OpenNebula 5.6 - FreeNAS 11.1
  • Hosting server OS: ubuntu 18.04
  • Guest OS: Ubuntu, CentOS, Windows

Steps to reproduce:

  • Setup Opennebula system with FreeNAS as NFS system datastore
  • Disconnect FreeNAS, shutdown or restart it
  • Check VM which running Linux OS

If OpenNebula detects a failed NFS server,it will already be too late to stop the VMs, the damage is already done. By default, on your fstab, you should already have configured the system to auto scan and correct issues related to the file system. There are limits to this mechanism, if the damage is too big, the partition (partition table to be exact) might be lost and more complicated procedures will have to be used to restore lost data.

1 Like