LVM commands are stuck forever and Sunstore is unable to create any vms

Hi Team,

We’ve been using open nebula 5.10 with HP SAN. At the time of setup, the snapshots were not supported when using SAN with open nebula 5.10. We were planning an upgrade but we did not have any backup. So we manually created snapshots for each VM to be prepared for the worse.

The script did below -

for i in $(onevm list | awk '{ print $1 }')
do
        if [[ $i =~ $re ]] ; then
		node=$(onevm show $i | grep "HOST                :" | awk '{ print $3 }')

		ssh $node sudo mkdir /mnt/cloud_snaps/$backupDate/$i

		ssh $node sudo cp /var/lib/one/datastores/101/$i/deployment.* /mnt/cloud_snaps/$backupDate/$i/

		lvmvolume=$(ssh $node sudo lvdisplay | grep /dev/vg-one-101/lv-one-$i | awk '{ print $3 }')

                for a in $lvmvolume
                do
                        echo `date` >> remote-backup.log
						echo $node lvcreate -L10G -s -n $snapshot-$node $a >> $backuplog
						ssh $node sudo lvcreate -L10G -s -n $snapshot-$node $a
						
						volumeId=`echo $a | awk '{print substr($0,length($0),1)}'`
						

						echo "$node dd if=/dev/vg-one-101/$snapshot-$node of=/mnt/cloud_snaps/$backupDate/$i/disk.$volumeId" >> $backuplog
						ssh $node sudo dd if=/dev/vg-one-101/$snapshot-$node of=/mnt/cloud_snaps/$backupDate/$i/disk.$volumeId
						

						echo $node lvremove /dev/vg-one-101/$snapshot-$node -y >> $backuplog
						ssh $node sudo lvremove /dev/vg-one-101/$snapshot-$node -y
				done
		fi
done

We were able to take two successful snapshots of all vms until this problem started. We’re unable to run any LVM commands. Every command seems to be stuck forever on something.

lvs -vvvv

root@cloud-node1:~# lvs -vvvv
#lvmcmdline.c:2787         Parsing: lvs -vvvv
#lvmcmdline.c:1868         Using command index 93 id lvs_general enum 73.
#config/config.c:1468       devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ]
#libdm-config.c:997       Setting global/locking_type to 1
#libdm-config.c:1061       Setting global/use_lvmetad to 0
#commands/toolcontext.c:1703   WARNING: Not using lvmetad because config setting use_lvmetad=0.
#commands/toolcontext.c:1704   WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
#libdm-config.c:1061       Setting global/use_lvmpolld to 1
#libdm-config.c:1061       Setting devices/sysfs_scan to 1
#filters/filter-sysfs.c:327         Sysfs filter initialised.
#filters/filter-internal.c:77         internal filter initialised.
#filters/filter-type.c:56         LVM type filter initialised.
#filters/filter-usable.c:192         Usable device filter initialised.
#libdm-config.c:1061       Setting devices/multipath_component_detection to 1
#filters/filter-mpath.c:291         mpath filter initialised.
#filters/filter-partitioned.c:59         Partitioned filter initialised.
#libdm-config.c:1061       Setting devices/md_component_detection to 1
#filters/filter-md.c:73         MD filter initialised.
#libdm-config.c:1061       Setting devices/fw_raid_component_detection to 0
#filters/filter-composite.c:104         Composite filter initialised.
#libdm-config.c:1061       Setting devices/ignore_suspended_devices to 0
#libdm-config.c:1061       Setting devices/ignore_lvm_mirrors to 1
#libdm-config.c:966       Setting devices/cache_dir to /run/lvm
#libdm-config.c:966       Setting devices/cache_file_prefix to
#libdm-config.c:975       devices/cache not found in config: defaulting to /run/lvm/.cache
#filters/filter-persistent.c:368         Persistent filter initialised.
#libdm-config.c:1061       Setting devices/write_cache_state to 1
#libdm-config.c:1061       Setting global/use_lvmetad to 0
#libdm-config.c:966       Setting activation/activation_mode to degraded
#libdm-config.c:1074       metadata/record_lvs_history not found in config: defaulting to 0
#lvmcmdline.c:2853         DEGRADED MODE. Incomplete RAID LVs will be processed.
#libdm-config.c:1061       Setting activation/monitoring to 1
#lvmcmdline.c:2859         Processing: lvs -vvvv
#lvmcmdline.c:2860         Command pid: 36378
#lvmcmdline.c:2861         system ID:
#lvmcmdline.c:2864         O_DIRECT will be used
#libdm-config.c:997       Setting global/locking_type to 1
#libdm-config.c:1061       Setting global/wait_for_locks to 1
#locking/locking.c:129       File-based locking selected.
#libdm-config.c:1061       Setting global/prioritise_write_locks to 1
#libdm-config.c:966       Setting global/locking_dir to /run/lock/lvm
#libdm-config.c:1061       Setting global/use_lvmlockd to 1
#daemon-client.c:33         /run/lvm/lvmlockd.socket: Opening daemon socket to lvmlockd for protocol lvmlockd version 1.
#daemon-client.c:52         Sending daemon lvmlockd: hello
#libdm-config.c:966       Setting response to OK
#libdm-config.c:966       Setting protocol to lvmlockd
#libdm-config.c:997       Setting version to 1
#locking/lvmlockd.c:94         Successfully connected to lvmlockd on fd 3.
#libdm-config.c:975       report/output_format not found in config: defaulting to basic
#libdm-config.c:1074       log/report_command_log not found in config: defaulting to 0
#libdm-config.c:1074       report/aligned not found in config: defaulting to 1
#libdm-config.c:1074       report/buffered not found in config: defaulting to 1
#libdm-config.c:1074       report/headings not found in config: defaulting to 1
#libdm-config.c:975       report/separator not found in config: defaulting to
#libdm-config.c:1074       report/prefixes not found in config: defaulting to 0
#libdm-config.c:1074       report/quoted not found in config: defaulting to 1
#libdm-config.c:1074       report/columns_as_rows not found in config: defaulting to 0
#libdm-config.c:975       report/lvs_sort not found in config: defaulting to vg_name,lv_name
#libdm-config.c:975       report/lvs_cols_verbose not found in config: defaulting to lv_name,vg_name,seg_count,lv_attr,lv_size,lv_major,lv_minor,lv_kernel_major,lv_kernel_minor,pool_lv,origin,data_percent,metadata_percent,move_pv,copy_percent,mirror_log,convert_lv,lv_uuid,lv_profile
#libdm-config.c:975       report/compact_output_cols not found in config: defaulting to
#locking/lvmlockd.c:1528         lockd global mode sh

The same goes for other commands pvs, vgscan etc

dmsetup info -c output

Name                                    Maj Min Stat Open Targ Event  UUID
vg--one--101-lv--one--277--2            253  65 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFtC85aJh9pAsFMHuk0McmYuBo3P0Pq0NS
vg--one--101-lv--one--190--0            253  35 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF7B7ZnuqCrCnjaYCKtFUmDDECdWVnTfw7
vg--one--101-lv--one--346--0            253  85 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF3dbNx9biLPWdhj2xs2YvFOvNfEnEagy5
vg--one--101-lv--one--331--0            253  81 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFkxP28LTRCa0Xwy3cEbdelgReXBMiA07I
vg--one--101-lv--one--162--0            253  32 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFs3cV742s68SsH82XcE9UZTDyJFn5ffzC
vg--one--101-lv--one--85--0             253  12 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFpmKj1Q3ZPWVRibVxe1b6VICfOGqufO6r
vg--one--101-lv--one--241--0            253  49 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF9vqwYzKxqUhP1pJolTcf7Gae1xtLZc9H
vg--one--101-lv--one--227--3            253  56 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFoTE2s07rVNfvLr2IXwI4d3BC8P8a22lu
vg--one--101-lv--one--166--0            253  33 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFOoH0YWRWqmG3zGPF2NFhOqwkQLJeGzDq
vg--one--101-lv--one--151--0            253  23 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFfxdsc0u3p1HulwTeGjiKiyadc3o3RNc6
vg--one--101-lv--one--1--0              253   6 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFUxHmPFR4vZ55SrSuvg1Rfr9xLVtT9Hef
3600c0ff00050c1c45d3bce5e01000000       253   1 L--w    1    1      0 mpath-3600c0ff00050c1c45d3bce5e01000000
vg--one--101-lv--one--288--0            253  72 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF7Cb4gg3DakeGB897Y2UnjA5DESLHIs05
vg--one--101-lv--one--273--0            253  61 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFWirQqlhEhFyMPFWxLv4Ew30u7Jn0Glcd
vg--one--101-lv--one--352--0            253  90 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFg56RxHQhgFnFn9PvP8WgQ1QONoApmxst
vg--one--101-lv--one--217--0            253  43 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFWOyUepj2RzxX49CRkb0neELNAtXPjYh9
vg--one--101-lv--one--202--0            253  41 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFifSP17MXiXoTa512L0lLS6mwVUrcrWUf
vg--one--101-lv--one--198--0            253  38 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFtkW9LnZV1YqTBSIcvDyYM6YVab0rrBDo
vg--one--101-lv--one--339--0            253  83 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFXt6s6lPKfjJwMJTdSetijnMB8QnLIHKM
vg--one--101-lv--one--227--2            253  55 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFM2g4FMOnuTUIU4y8HIZmwysG8fqzInK2
vg--one--101-lv--one--277--0            253  64 L--w    2    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFGOk3d1ncu8BK1FTYfljVUpt0MsPVCNtA
vg--one--101-lv--one--112--0            253  15 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFMImfrREdRrOwiTGQqe7CxRcDVPk9d9I3
vg--one--101-lv--one--366--2            253  93 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFUj7HT3c2aUco2YVcLDfz1EIh1jzg4Q8F
vg--one--101-lv--one--159--0            253  31 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFKJfp29FOekfsaXSycBfgzNMqdbKiKmy3
3600c0ff00050c1c45c3bce5e01000000-part1 253   5 L--w   91    1      0 part1-mpath-3600c0ff00050c1c45c3bce5e01000000
vg--one--101-lv--one--266--0            253  57 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFKqPmHLpW326MHq75Qmqp5xLt5PwTcIAE
vg--one--101-lv--one--84--0             253  11 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFCORatpvja3Sr7Ign6y8rFuNp8LYD8srZ
vg--one--101-lv--one--297--3            253  94 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFLrrUwEcfcDC4RteqJacVWZhyU5Rr4pDV
3600c0ff00050c1c45c3bce5e01000000       253   2 L--w    1    1      0 mpath-3600c0ff00050c1c45c3bce5e01000000
vg--one--101-lv--one--227--0            253  48 L--w    0    3      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF5y42uftiGuUEKMjMeATDGCCwQPriZdfe
vg--one--109-lv--one--52--0             253  96 L--w    0    1      0 LVM-Tm4MEczTmotO1VcAj4izfTPT71Tg3Xyh8ucA2yzuSmvc22MZ0Jcwig4qzKdACe0a
vg--one--101-lv--one--287--0            253  71 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFdPJP3O9324W5oLFWZdnYcLF1104c91DL
vg--one--101-lv--one--272--0            253  60 L--w    2    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFQTxiYK8HW90orlY0djpQHWLsldLnI0v4
vg--one--101-lv--one--209--2            253  44 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFk4HSMK4p3ehGXgBpieCItDcvZMfxWcPb
vg--one--101-lv--one--244--0            253  51 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFJxX4CNyGbzEgseDtEECfp82yqDbLd2Rl
vg--one--101-lv--one--351--0            253  89 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFN6XyrpXDQKpRXU0pZ8VoeLMuZQMxgl0c
vg--one--101-lv--one--201--0            253  40 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFmZAoyyzaD20bvrEm32ylliAJuuZqvkvz
vg--one--101-lv--one--197--0            253  37 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFt6iJdMrGU5w2LHQMVcveH8mzQPVPz1Tw
vg--one--101-lv--one--154--0            253  27 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF0bWKri50GgGQ2qVJ3yVEkoIw4B1Ob17A
3600c0ff00050c1c45d3bce5e01000000-part1 253   4 L--w    0    1      0 part1-mpath-3600c0ff00050c1c45d3bce5e01000000
vg--one--109-lv--one--56--0             253  97 L--w    0    1      0 LVM-Tm4MEczTmotO1VcAj4izfTPT71Tg3XyhMqbTvXHJO00yeX4KpYo9Wevlmp9bwKCT
vg--one--101-lv--one--340--0            253  84 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFrLpdBFVP3Hy4lIASr7ZOy3sjWW7VrhWN
vg--one--101-lv--one--243--2            253  50 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFhhKUmyGjdC75cjKwW3ycAWKlFaqTdYu0
vg--one--101-lv--one--366--2-real       253  28 L--w    1    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFUj7HT3c2aUco2YVcLDfz1EIh1jzg4Q8F-real
vg--one--101-lv--one--158--0            253  30 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFNFL7P51PS1C3yeb6KIAVMotEucZm4b6c
vg--one--101-lv--one--153--2            253  26 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF9uro1WlkkuTTmLsJEbv8RRWoqjL6BPXb
vg--one--101-lv--one--222--0            253  46 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFpX6sGZTFRQdwHeR0yAmLIbDIZdHFpG0c
vg--one--101-lv--one--316--0            253  79 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF2m1DdNRF0hdhlZdBPX8Oh4kqWggLbIlH
vg--one--101-lv--one--301--0            253  77 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFKdOZmn1zg0QTZB3dmfBCQ9EY8F1iyCtd
vg--one--101-lv--one--282--0            253  68 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFNfHi18dweDaErbE9kDspgWfsrTHAPWNi
vg--one--101-lv--one--191--3            253  76 L--w    2    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF2G0X30oDrgNJId7i2rlGXAiewqJK6Yyf
vg--one--101-lv--one--226--0            253  47 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF83RKaWLreiWo8lFaXmE0nX9F4caQyfPD
vg--one--101-lv--one--192--0            253  36 L--w    2    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFm7ps9H3ZiP7EP58CSzgMRShohMWPlcqp
3600c0ff00050c1c45e3bce5e01000000-part1 253   3 L--w    3    1      0 part1-mpath-3600c0ff00050c1c45e3bce5e01000000
vg--one--101-lv--one--286--0            253  70 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFi6sOx8nZtxsfeePhMqE9XGX0ee4ZsXtD
vg--one--101-lv--one--270--3            253  67 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFi1eFqRIHeUqU7A03z4QZvzZ9baLr7uGQ
vg--one--101-lv--one--365--0            253  92 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF7GGN2iBIYnHnBGvNJtkx8EjcZL6fRpe4
vg--one--101-lv--one--153--0            253  25 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFVlkWuxEXkR30eWeOmT7mZhZ6G0TyHUeO
vg--one--109-lv--test                   253  95 L--w    0    1      0 LVM-Tm4MEczTmotO1VcAj4izfTPT71Tg3XyhD3luh4SAl9DAI3MNU2FeRvoxV9AHzGjB
vg--one--101-lv--one--275--0            253  63 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF4iNLh1GXmXrCW7vlhIh2Aeaxkibom1Qq
vg--one--101-lv--one--246--3            253  99 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFWks361OZrvwFvXd2rkY4bejeQypLmILj
vg--one--101-lv--one--270--2            253  66 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFoUsd2XUwu5lTLfLpiEn2kv5f0LiO2E5G
vg--one--101-lv--one--93--0             253  14 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF0iuo73Bage7pFjbrPnBYU6AW5wgTOgX4
vg--one--101-lv--one--292--0            253  74 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFk5MiTbf4vuft4WsOiH5J2t1ChUMHZXIf
vg--one--101-lv--one--157--0            253  29 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFZcHSJztgFI9vZEFxCfmE5EkOUhkeozTx
vg--one--101-lv--one--371--0            253  98 L--w    2    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFGMNK1FDClvCU9I2OCBxlKYChnU8fKsVy
vg--one--101-lv--one--246--2            253  53 L--w    2    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF8o7Noon44fXBodFFHbTWEJ5rykwS4Jfa
vg--one--101-lv--one--82--0             253  10 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFrgt9yrqVi3jVnbNE3V9Lr85sIa0wA69U
vg--one--101-lv--one--315--0            253  78 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFOzpbMfWFMLi9ihwzD3ijNIPPbcoAhjHx
vg--one--101-lv--one--199--2            253  39 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFfSKm5j9VWGgp6UqYcLqHM0KvdYnQw3wW
vg--one--101-lv--one--130--3            253  22 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFPCxq29B3qx24JdKc7QHiLOH4jJUSmZ3i
vg--one--101-lv--one--263--2            253  58 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFd2F2UHhWJRCelEwm1lX0yCxidnQ6DMY5
vg--one--101-lv--one--86--0             253  13 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFdsE1ZseJQw0n0isrl4CnusOiEPeGRlql
vg--one--101-lv--one--47--1             253   9 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF0Eodt4Uzxn8PfkcC9FjAdwa1z1LJNDfW
vg--one--101-lv--snapshot10             253 100 L-sw   16    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFLhilzQ2aLeiCxfi08cMmRc78EEamUtEN
vg--one--101-lv--snapshot1-cow          253  80 L-sw    1    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFRTC7eStB3oVzNvbrRrl71vqE1OGNwD2g-cow
vg--one--101-lv--one--285--0            253  69 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFlPGoNLFz2XNPz5dSqLe19eyCFF4ae2gr
vg--one--101-lv--one--270--0            253  59 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFAs8QI7anV4so8cfV0c93K2TwB7c25U53
vg--one--101-lv--one--130--2            253  21 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFydtbAmBKgDs4cI4ezf0DCH9WBSszcpmC
vg--one--101-lv--one--364--0            253  91 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFapZAZSZejq4QxH1F8b6cH5rstT04AB15
vg--one--101-lv--one--214--0            253  42 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFF8gmnGrn7Zoe1GbLURFDA8MHinegO6co
vg--one--101-lv--one--117--2            253  18 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFZeFTd9WHVC2jZpSsKko2gIoFrj0SuLDV
vg--one--101-lv--one--336--0            253  82 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFG4e4qOffwjpfRjthWGyeNl8KfqLqd3uf
vg--one--101-lv--one--167--0            253  34 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFMuwQfbSXZEkw5Wv3zL6BZpWL1jBTNzPB
vg--one--101-lv--one--152--0            253  24 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFBYTeDEYfgaI6kIhtmPiy834h6GBGdQq8
vg--one--101-lv--one--274--0            253  62 L--w    2    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFaKOWuU1BTn3Xi1zLPPoYpJHZTCTSA7ht
vg--one--101-lv--one--32--0             253   8 L--w    0    3      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFyzOgiE7hef00uAvk3nmo89t9lnpJkGF6
vg--one--101-lv--one--246--0            253  52 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFr3W3rhVbsjvqwX2ABtOXvAOyyivwGZG5
3600c0ff00050c1c45e3bce5e01000000       253   0 L--w    1    1      0 mpath-3600c0ff00050c1c45e3bce5e01000000
vg--one--101-lv--snapshot1              253  87 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFRTC7eStB3oVzNvbrRrl71vqE1OGNwD2g
vg--one--101-lv--one--241--2            253  88 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFetOFmfr8AL1K3CBQNdQkmSXU4D3iBPmL
vg--one--101-lv--one--218--0            253  45 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFIL7fgOUDu6MZ0aCq2hsfVm31p7N7d2fB
vg--one--101-lv--one--291--0            253  73 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFfODmHiqSPstu2ab0eHuO0xM5prJzVeqH
vg--one--101-lv--one--263--0            253  54 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFZXUoyLkNulimnGFClyc9DBh4XgQB4dEX
vg--one--101-lv--one--128--0            253  19 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFjnVp86hs4rAvRuRuiBqXXDthWt6UXFUZ
vg--one--101-lv--one--113--0            253  16 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFAsWsme2ZQSVZFTEaLq1ICbFIZvL4xKln
vg--one--101-lv--one--295--0            253  75 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFSgAysDyosnAWpTuyyalFhicgcISz3C1g
vg--one--101-lv--one--130--0            253  20 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccF83tNqLVSchmuCdh3zieK4MwpvIFc2dCZ
vg--one--101-lv--one--290--2            253  86 L--w    0    2      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFjamkpzhHPJJnGQZKxCDnnJnZiIq8w3ir
vg--one--101-lv--one--117--0            253  17 L--w    2    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFO8DJoKt7AReAdAk4sXaofpRcAUOpkKCs
vg--one--101-lv--one--25--0             253   7 L--w    0    1      0 LVM-OLJcAfQjbMWcmMMX8j6Bk1dqeInUqccFH22y2uhLCXMd8RC2UcZfZszdaYBes0ik```

Hi @shahidcodes,

It seems that the a lock is hold for the resources [1]:

#locking/lvmlockd.c:1528         lockd global mode sh

Maybe you can check that there are not lv* commands stuck holding a lock.

[1] lvmlockd(8) - Linux manual page

Hello @cgonzalez

We ultimately had to reboot each node to get rid of this issue.

Thanks for the help