Kernel panic when booting up a RHEL 6.6 virtual machine

Hi!

I’m getting a kernel panic when i’m trying to boot up a RHEL 6.6 virtual machine:

mount: wrong fs type, bad option, bad superblock on /dev/mapper/vgSys-root_lv, missing codepage or helper program, or other error. In some cases useful info is found in syslog - try dmesg | tail or so

I’ve done the same process for RHEL 6.5 and no problem has arisen, I don’t know if there are any packages i this release than can cause this problem.

Some extra info:

/etc/fstab

/dev/mapper/vgSys-root_lv / ext4 defaults 1 1
/dev/mapper/vgMon-wva_lv /WVA ext4 defaults 1 2
UUID=a5336b69-7438-4876-81f5-ff6bc4f413c2 /boot ext4 defaults 1 2
/dev/mapper/vgMon-ctm_lv /ctmagent ext4 defaults 1 2
/dev/mapper/vgSys-home_lv /home ext4 defaults 1 2
/dev/mapper/vgSys-opt_lv /opt ext4 defaults 1 2
/dev/mapper/vgMon-itm_lv /opt/IBM/ITM ext4 defaults 1 2
/dev/mapper/vgMon-omv_lv /opt/OmniVision ext4 defaults 1 2
/dev/mapper/vgSys-tmp_lv /tmp ext4 defaults 1 2
/dev/mapper/vgMon-bl_lv /usr/local/bl_agent ext4 defaults 1 2
/dev/mapper/vgSys-var_lv /var ext4 defaults 1 2
/dev/mapper/vgSys-varlog_lv /var/log ext4 defaults 1 2
/dev/mapper/vgSys-audit_lv /var/log/audit ext4 defaults 1 2
/dev/mapper/vgSwap-swap_lv swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/vgSoftware/tomcat_lv /opt/tomcat ext4 rw 0 0
/dev/vgSoftware/wily_lv /opt/wily ext4 rw 0 0
/dev/vgSoftware/gestionWAS_lv /opt/gestionWAS ext4 rw 0 0
/dev/vgSoftware/export_lv /export ext4 rw 0 0
/dev/vgSoftware/sds_lv /SDS ext4 rw 0 0
/dev/vgSoftware/zipapps_lv /ZipAplicacionesLogicas ext4 rw 0 0

And template info:

TEMPLATE 145 INFORMATION
ID : 145
NAME : JBOSS-JWS-3.3-RHEL66.X64.ATF.PRB
USER : oneadmin
GROUP : oneadmin
REGISTER TIME : 11/06 09:03:09

PERMISSIONS
OWNER : um-
GROUP : —
OTHER : —

TEMPLATE CONTENTS
CONTEXT=[
BL_AREA="$BL_AREA",
FILES_DS="$FILE[IMAGE_ID=417]",
INIT_SCRIPTS=“atf-init.sh”,
NETWORK=“YES”,
NTP="$NTP",
SEARCH_DOMAIN="$SEARCH_DOMAIN",
SET_HOSTNAME="$SET_HOSTNAME",
SSH_PUBLIC_KEY="$USER[SSH_PUBLIC_KEY]" ]
CPU="2"
DISK=[
DEV_PREFIX=“vd”,
DRIVER=“qcow2”,
IMAGE=“JBOSS-JWS-3.3-RHEL66.X64.ATF.PRB.disk1”,
IMAGE_UNAME=“oneadmin” ]
DISK=[
DEV_PREFIX=“vd”,
DRIVER=“qcow2”,
IMAGE=“JBOSS-JWS-3.3-RHEL66.X64.ATF.PRB.disk2”,
IMAGE_UNAME=“oneadmin” ]
DISK=[
DEV_PREFIX=“vd”,
DRIVER=“qcow2”,
IMAGE=“JBOSS-JWS-3.3-RHEL66.X64.ATF.PRB.disk3”,
IMAGE_UNAME=“oneadmin” ]
DISK=[
DEV_PREFIX=“vd”,
DRIVER=“qcow2”,
IMAGE=“JBOSS-JWS-3.3-RHEL66.X64.ATF.PRB.disk4”,
IMAGE_UNAME=“oneadmin” ]
FROM_APP="54f985c1a1d05513be000049"
FROM_APP_NAME="TIBCO-BW-5.11-RHEL-65.X64.ATF.PRB"
GRAPHICS=[
KEYMAP=“es”,
LISTEN=“0.0.0.0”,
TYPE=“VNC” ]
MEMORY="4096"
NIC=[
NETWORK=“ManagementNetwork_VLAN309”,
NETWORK_UNAME=“oneadmin” ]
NIC=[
NETWORK=“ServiceNetwork_VLAN409”,
NETWORK_UNAME=“oneadmin” ]

Can you give a hand, guys? Thanks in advance.

hi @esierrap ,

I’ve seen such problems on images created with disk clone/save-as on running VM.

Try to attach(I don’t know what datastore IM_MAD you are using) the root disk on baremetal and check with the partition table with fdisk. Then try to add the VG and do fsck on root lv.

Kind Regards,
Anton Todorov

Thank you @atodorov_storpool,

Could it be due to a kernel bug? The current virtual machine has version 2.6.32-504.8.1.el6.x86_64 whereas on a VM with kernel 2.6.32-504.3.3.el6.x86_64, the error doesn’t occur…

Hi @esierrap,

Everything is possible :smile:

But probably it is initrd issue than kernel bug.
I’ve just saw that you are using qcow2 files for disk image containers. As next step you can try to mount the qcow image via fuse and check/compare the initrd files inside. The initrd for kernel version 2.6.32-504.8.1.el6.x86_64 could have missing kernel module or something other…

Kind Regards,
Anton Todorov

Theses are the kernel modules that i have:

Module Size Used by
vsock 44145 0
ppdev 8537 0
microcode 112653 0
vmware_balloon 7199 0
parport_pc 22658 0
parport 36209 2 ppdev,parport_pc
sg 29318 0
vmci 42456 1 vsock
i2c_piix4 11776 0
i2c_core 29964 1 i2c_piix4
ata_generic 3837 0
shpchp 29130 0
ext4 378476 19
jbd2 93427 1 ext4
mbcache 8193 1 ext4
sd_mod 36998 9
crc_t10dif 1305 1 sd_mod
vmxnet3 43524 0
mptspi 16411 5
mptscsih 36636 1 mptspi
mptbase 93647 2 mptspi,mptscsih
scsi_transport_spi 25447 1 mptspi
pata_acpi 3701 0
dm_mirror 14384 0
dm_region_hash 12085 1 dm_mirror
dm_log 9930 2 dm_mirror,dm_region_hash
dm_mod 95622 59 dm_mirror,dm_log

Do yo miss one in particular?

Here, kernel modules compiled into the kernel:

[root@vmrhel66ok ~]# cat /boot/config-2.6.32-504.3.3.el6.x86_64 | grep -i virt
CONFIG_PARAVIRT_GUEST=y
CONFIG_PARAVIRT=y

CONFIG_PARAVIRT_SPINLOCKS is not set

CONFIG_PARAVIRT_CLOCK=y

CONFIG_PARAVIRT_DEBUG is not set

CONFIG_VIRT_TO_BUS=y
CONFIG_NET_9P_VIRTIO=m
CONFIG_VIRTIO_BLK=m
CONFIG_SCSI_VIRTIO=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM_VIRTIO=m

CONFIG_REGULATOR_VIRTUAL_CONSUMER is not set

CONFIG_FB_VIRTUAL=m
CONFIG_SND_VIRTUOSO=m

CONFIG_DEBUG_VIRTUAL is not set

CONFIG_VIRTUALIZATION=y
CONFIG_VIRTIO=m
CONFIG_VIRTIO_RING=m
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m

Thanks for all your support! @atodorov_storpool

Hi @esierrap,

Lets check another things:
What is your kernel cmdline?
Do you have rd.auto cmdline parameters, or any rd.lvm.*?
is there a difference in the kernel cmdline for the working and non-working kernel?

Kind Regards,
Anton Todorov

Hi!

Working kernel:

kernel /vmlinuz-2.6.32-504.3.3.el6.x86_64 ro root=/dev/mapper/vgSys-root_lv rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vgSys/root_lv KEYBOARDTYPE=pc KEYTABLE=es rd_NO_MD nofb quiet splash=quiet SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vgSwap/swap_lv rd_NO_DM rhgb quiet

Non-working kernel:

Kernel /vmlinuz-2.6.32-504.8.1.el6.x86_64 ro root=/dev/mapper/vgSys-root_lv rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vgSys/root_lv KEYBOARDTYPE=pc KEYTABLE=es rd_NO_MD nofb quiet splash=quiet SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vgSwap/swap_lv rd_NO_DM rhgb quiet rdblacklist=ata_piix rdblacklist=ata_generic

I’ve deleted last two parameters but the result is the same.

Thank you, once again.