Not able to deploy more than 9 LXC containers on LXC host

Hello,

I have been experimenting with the 6.0 version for a few weeks now. All is going well, I have 5 identical machines 1xFE, 3xKM, and 1xLXC hosts. I have multiple Local storage and multiple NFS Datastores. I’m able to deploy 100 VMs in a few minutes.

LXC Issue #1 - I can’t deploy more than 9 containers
I’m am not able to deploy more than 9 LXC containers on my LXC host. The 10th+ always fails. It feels like a configuration issue but I have been unable to find it yet. I’m using Ubuntu 20.04 throughout.

Here is the oned.log:

Wed Oct 20 21:44:28 2021 [Z0][VMM][I]: ExitCode: 0
Wed Oct 20 21:44:28 2021 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/104/271/vm.xml.
Wed Oct 20 21:44:28 2021 [Z0][VMM][I]: ExitCode: 0
Wed Oct 20 21:44:28 2021 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/104/271/ds.xml.
Wed Oct 20 21:44:29 2021 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/lxc/deploy '/var/lib/one//datastores/104/271/deployment.1' 'one-lxc01' 271 one-lxc01
Wed Oct 20 21:44:29 2021 [Z0][VMM][I]: There was an error starting the container.
Wed Oct 20 21:44:29 2021 [Z0][VMM][I]: ExitCode: 255
Wed Oct 20 21:44:29 2021 [Z0][VMM][I]: ExitCode: 0
Wed Oct 20 21:44:29 2021 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Wed Oct 20 21:44:29 2021 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Wed Oct 20 21:44:29 2021 [Z0][IPM][D]: Message received: DEPLOY FAILURE 271 -

Sometimes it shows RUNNING with no error and then fails a minute later to POWEROFF

Here are the logs of the VM (Container) itself:

Wed Oct 20 23:25:49 2021 [Z0][VM][I]: New state is ACTIVE
Wed Oct 20 23:25:49 2021 [Z0][VM][I]: New LCM state is PROLOG
Wed Oct 20 23:25:51 2021 [Z0][VM][I]: New LCM state is BOOT
Wed Oct 20 23:25:52 2021 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/271/deployment.0
Wed Oct 20 23:25:55 2021 [Z0][VM][I]: New LCM state is RUNNING
Wed Oct 20 23:27:50 2021 [Z0][LCM][I]: VM running but monitor state is POWEROFF
Wed Oct 20 23:27:50 2021 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Wed Oct 20 23:27:50 2021 [Z0][VM][I]: New state is POWEROFF
Wed Oct 20 23:27:50 2021 [Z0][VM][I]: New LCM state is LCM_INIT
Wed Oct 20 23:45:42 2021 [Z0][VM][I]: New state is ACTIVE
Wed Oct 20 23:45:42 2021 [Z0][VM][I]: New LCM state is BOOT_POWEROFF
Wed Oct 20 23:45:42 2021 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/271/deployment.1
Wed Oct 20 23:45:44 2021 [Z0][VM][I]: New LCM state is RUNNING
Wed Oct 20 23:46:06 2021 [Z0][LCM][I]: VM running but monitor state is POWEROFF
Wed Oct 20 23:46:06 2021 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Wed Oct 20 23:46:06 2021 [Z0][VM][I]: New state is POWEROFF
Wed Oct 20 23:46:06 2021 [Z0][VM][I]: New LCM state is LCM_INIT

LXC Issue #2 - I can’t properly delete LXC Containers
Terminating powered off or running LXC seems to keep the container in EPILOG state for an indefinite amount of time. Subsequently doing a recover-delete removes the container but leaves it has a zombie VM. These VM show as Done on the Front end. It is easy enough to remove them with the onedb purge-done but how do I remove the Zombies from the LXC host. The /dev/loopX definition for the Zombie containers also remain.

Any pointers would help.

I found a way to delete the LXC container Zombies using:

#lxc-destroy one-{lxc-id}

on the LXC host

Hello @AntoineB

Look for the LXC-level container logs at /var/log/lxc/<container_name> in the virtualization node where the container was deployed to.

LXC Issue #2 - I can’t properly delete LXC Containers

The way the container storage is handled is as follows:

  • the container images at /var/lib/one/datastores/<sys_ds_id>/<vm_id>/ are mapped into block devices depending on the storage driver used.

  • the block devices are then mounted at /var/lib/one/datastores/<sys_ds_id>/<vm_id>/mapper/disk.<disk_id>

  • the block device mountpoint is then id-shifted with bindfs to match the container id-mapping requirements. This shift is mirrored to /var/lib/lxc-one/<vm_id>/disk.<disk_id>

You need to revert all of this in order to manually clean the container

I tried to deploy 12 containers this time 9 succeeded and 3 failed. the /var/log/lxc directory is empty

Here is the container host syslog. vm_id =< 330 succeeded > 330 failed

Oct 22 00:05:20 one-lxc01 kernel: [17458.394942] brPrivNat4: topology change detected, propagating

Oct 22 00:05:30 one-lxc01 kernel: [17468.770184] EXT4-fs (loop20): mounted filesystem with ordered data mode. Opts: (null)

Oct 22 00:05:30 one-lxc01 kernel: [17468.926018] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:05:30 one-lxc01 kernel: [17468.926196] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:05:30 one-lxc01 kernel: [17468.926290] ISO 9660 Extensions: RRIP_1991A

Oct 22 00:05:30 one-lxc01 kernel: [17469.025570] brPrivNat4: port 9(one-329-0) entered blocking state

Oct 22 00:05:30 one-lxc01 kernel: [17469.025573] brPrivNat4: port 9(one-329-0) entered disabled state

Oct 22 00:05:30 one-lxc01 kernel: [17469.025690] device one-329-0 entered promiscuous mode

Oct 22 00:05:30 one-lxc01 systemd-udevd[192130]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

Oct 22 00:05:30 one-lxc01 systemd-udevd[192130]: Using default interface naming scheme 'v245'.

Oct 22 00:05:30 one-lxc01 networkd-dispatcher[5809]: WARNING:Unknown index 33 seen, reloading interface list

Oct 22 00:05:30 one-lxc01 kernel: [17469.026799] eth0: renamed from vethcXvnZg

Oct 22 00:05:31 one-lxc01 systemd-networkd[5648]: one-329-0: Link UP

Oct 22 00:05:32 one-lxc01 kernel: [17470.362876] audit: type=1400 audit(1634861132.327:70): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=192359 comm="(d-logind)" flags="rw, rslave"

Oct 22 00:05:32 one-lxc01 systemd-networkd[5648]: one-329-0: Gained carrier

Oct 22 00:05:32 one-lxc01 kernel: [17470.623162] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Oct 22 00:05:32 one-lxc01 kernel: [17470.623274] IPv6: ADDRCONF(NETDEV_CHANGE): one-329-0: link becomes ready

Oct 22 00:05:32 one-lxc01 kernel: [17470.623432] brPrivNat4: port 9(one-329-0) entered blocking state

Oct 22 00:05:32 one-lxc01 kernel: [17470.623434] brPrivNat4: port 9(one-329-0) entered listening state

Oct 22 00:05:34 one-lxc01 systemd-networkd[5648]: one-329-0: Gained IPv6LL

Oct 22 00:05:34 one-lxc01 kernel: [17472.612487] audit: type=1400 audit(1634861134.575:71): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=192610 comm="(networkd)" flags="rw, rslave"

Oct 22 00:05:34 one-lxc01 kernel: [17472.638301] brPrivNat4: port 9(one-329-0) entered learning state

Oct 22 00:05:34 one-lxc01 kernel: [17472.706335] audit: type=1400 audit(1634861134.671:72): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=192629 comm="(resolved)" flags="rw, rslave"

Oct 22 00:05:35 one-lxc01 kernel: [17473.515577] audit: type=1400 audit(1634861135.479:73): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=192702 comm="(ostnamed)" flags="rw, rslave"

Oct 22 00:05:36 one-lxc01 kernel: [17474.650208] brPrivNat4: port 9(one-329-0) entered forwarding state

Oct 22 00:05:36 one-lxc01 kernel: [17474.650225] brPrivNat4: topology change detected, propagating

Oct 22 00:05:44 one-lxc01 kernel: [17482.832864] EXT4-fs (loop22): mounted filesystem with ordered data mode. Opts: (null)

Oct 22 00:05:44 one-lxc01 kernel: [17483.024707] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:05:44 one-lxc01 kernel: [17483.024960] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:05:44 one-lxc01 kernel: [17483.025120] ISO 9660 Extensions: RRIP_1991A

Oct 22 00:05:45 one-lxc01 networkd-dispatcher[5809]: WARNING:Unknown index 34 seen, reloading interface list

Oct 22 00:05:45 one-lxc01 systemd-networkd[5648]: one-330-0: Link UP

Oct 22 00:05:45 one-lxc01 systemd-udevd[193179]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

Oct 22 00:05:45 one-lxc01 systemd-udevd[193179]: Using default interface naming scheme 'v245'.

Oct 22 00:05:45 one-lxc01 kernel: [17483.174081] brPrivNat4: port 10(one-330-0) entered blocking state

Oct 22 00:05:45 one-lxc01 kernel: [17483.174085] brPrivNat4: port 10(one-330-0) entered disabled state

Oct 22 00:05:45 one-lxc01 kernel: [17483.174340] device one-330-0 entered promiscuous mode

Oct 22 00:05:45 one-lxc01 kernel: [17483.176063] eth0: renamed from vethVBb1H6

Oct 22 00:05:46 one-lxc01 kernel: [17484.678070] audit: type=1400 audit(1634861146.644:74): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=193418 comm="(d-logind)" flags="rw, rslave"

Oct 22 00:05:46 one-lxc01 kernel: [17484.849110] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Oct 22 00:05:46 one-lxc01 kernel: [17484.849253] IPv6: ADDRCONF(NETDEV_CHANGE): one-330-0: link becomes ready

Oct 22 00:05:46 one-lxc01 kernel: [17484.849382] brPrivNat4: port 10(one-330-0) entered blocking state

Oct 22 00:05:46 one-lxc01 kernel: [17484.849385] brPrivNat4: port 10(one-330-0) entered listening state

Oct 22 00:05:46 one-lxc01 systemd-networkd[5648]: one-330-0: Gained carrier

Oct 22 00:05:48 one-lxc01 systemd-networkd[5648]: one-330-0: Gained IPv6LL

Oct 22 00:05:48 one-lxc01 kernel: [17486.485482] audit: type=1400 audit(1634861148.448:75): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=193620 comm="(networkd)" flags="rw, rslave"

Oct 22 00:05:48 one-lxc01 kernel: [17486.575414] audit: type=1400 audit(1634861148.540:76): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=193639 comm="(resolved)" flags="rw, rslave"

Oct 22 00:05:48 one-lxc01 kernel: [17486.702088] audit: type=1400 audit(1634861148.668:77): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=193643 comm="(resolved)" flags="rw, rslave"

Oct 22 00:05:48 one-lxc01 kernel: [17486.805982] audit: type=1400 audit(1634861148.772:78): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=193646 comm="(resolved)" flags="rw, rslave"

Oct 22 00:05:48 one-lxc01 kernel: [17486.873649] brPrivNat4: port 10(one-330-0) entered learning state

Oct 22 00:05:48 one-lxc01 kernel: [17486.912581] audit: type=1400 audit(1634861148.876:79): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=193649 comm="(resolved)" flags="rw, rslave"

Oct 22 00:05:48 one-lxc01 kernel: [17486.995741] audit: type=1400 audit(1634861148.960:80): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=193652 comm="(resolved)" flags="rw, rslave"

Oct 22 00:05:49 one-lxc01 kernel: [17487.785518] audit: type=1400 audit(1634861149.748:81): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=193771 comm="(ostnamed)" flags="rw, rslave"

Oct 22 00:05:50 one-lxc01 kernel: [17488.889631] brPrivNat4: port 10(one-330-0) entered forwarding state

Oct 22 00:05:50 one-lxc01 kernel: [17488.889647] brPrivNat4: topology change detected, propagating

Oct 22 00:06:00 one-lxc01 kernel: [17498.132485] EXT4-fs (loop24): mounted filesystem with ordered data mode. Opts: (null)

Oct 22 00:06:00 one-lxc01 kernel: [17498.330383] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:06:00 one-lxc01 kernel: [17498.330752] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:06:00 one-lxc01 kernel: [17498.330916] ISO 9660 Extensions: RRIP_1991A

Oct 22 00:06:00 one-lxc01 networkd-dispatcher[5809]: WARNING:Unknown index 35 seen, reloading interface list

Oct 22 00:06:00 one-lxc01 systemd-udevd[194053]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

Oct 22 00:06:00 one-lxc01 systemd-udevd[194053]: Using default interface naming scheme 'v245'.

Oct 22 00:06:00 one-lxc01 systemd-networkd[5648]: one-331-0: Link UP

Oct 22 00:06:00 one-lxc01 kernel: [17498.441488] brPrivNat4: port 11(one-331-0) entered blocking state

Oct 22 00:06:00 one-lxc01 kernel: [17498.441491] brPrivNat4: port 11(one-331-0) entered disabled state

Oct 22 00:06:00 one-lxc01 kernel: [17498.441669] device one-331-0 entered promiscuous mode

Oct 22 00:06:00 one-lxc01 kernel: [17498.443293] eth0: renamed from vethjbVMNa

Oct 22 00:06:00 one-lxc01 systemd-networkd[5648]: one-331-0: Link DOWN

Oct 22 00:06:00 one-lxc01 kernel: [17498.565854] brPrivNat4: port 11(one-331-0) entered disabled state

Oct 22 00:06:00 one-lxc01 kernel: [17498.567371] device one-331-0 left promiscuous mode

Oct 22 00:06:00 one-lxc01 kernel: [17498.567375] brPrivNat4: port 11(one-331-0) entered disabled state

Oct 22 00:06:00 one-lxc01 systemd-networkd[5648]: rtnl: received neighbor for link '35' we don't know about, ignoring.

Oct 22 00:06:00 one-lxc01 systemd-networkd[5648]: rtnl: received neighbor for link '35' we don't know about, ignoring.

Oct 22 00:06:12 one-lxc01 systemd[1]: Started Session 72 of user oneadmin.

Oct 22 00:06:16 one-lxc01 kernel: [17514.287424] EXT4-fs (loop26): mounted filesystem with ordered data mode. Opts: (null)

Oct 22 00:06:16 one-lxc01 kernel: [17514.493843] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:06:16 one-lxc01 kernel: [17514.494055] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:06:16 one-lxc01 kernel: [17514.494221] ISO 9660 Extensions: RRIP_1991A

Oct 22 00:06:16 one-lxc01 systemd-udevd[194287]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

Oct 22 00:06:16 one-lxc01 systemd-udevd[194287]: Using default interface naming scheme 'v245'.

Oct 22 00:06:16 one-lxc01 networkd-dispatcher[5809]: WARNING:Unknown index 36 seen, reloading interface list

Oct 22 00:06:16 one-lxc01 systemd-networkd[5648]: one-332-0: Link UP

Oct 22 00:06:16 one-lxc01 kernel: [17514.625558] brPrivNat4: port 11(one-332-0) entered blocking state

Oct 22 00:06:16 one-lxc01 kernel: [17514.625562] brPrivNat4: port 11(one-332-0) entered disabled state

Oct 22 00:06:16 one-lxc01 kernel: [17514.625719] device one-332-0 entered promiscuous mode

Oct 22 00:06:16 one-lxc01 kernel: [17514.627438] eth0: renamed from veth9W8VNQ

Oct 22 00:06:16 one-lxc01 systemd-networkd[5648]: one-332-0: Link DOWN

Oct 22 00:06:16 one-lxc01 kernel: [17514.777181] brPrivNat4: port 11(one-332-0) entered disabled state

Oct 22 00:06:16 one-lxc01 kernel: [17514.779038] device one-332-0 left promiscuous mode

Oct 22 00:06:16 one-lxc01 kernel: [17514.779047] brPrivNat4: port 11(one-332-0) entered disabled state

Oct 22 00:06:16 one-lxc01 systemd-networkd[5648]: rtnl: received neighbor for link '36' we don't know about, ignoring.

Oct 22 00:06:16 one-lxc01 systemd-networkd[5648]: rtnl: received neighbor for link '36' we don't know about, ignoring.

Oct 22 00:06:30 one-lxc01 kernel: [17528.636419] EXT4-fs (loop28): mounted filesystem with ordered data mode. Opts: (null)

Oct 22 00:06:30 one-lxc01 kernel: [17528.819347] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:06:30 one-lxc01 kernel: [17528.819617] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:06:30 one-lxc01 kernel: [17528.819758] ISO 9660 Extensions: RRIP_1991A

Oct 22 00:06:30 one-lxc01 networkd-dispatcher[5809]: WARNING:Unknown index 37 seen, reloading interface list

Oct 22 00:06:30 one-lxc01 systemd-udevd[195064]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

Oct 22 00:06:30 one-lxc01 systemd-udevd[195064]: Using default interface naming scheme 'v245'.

Oct 22 00:06:30 one-lxc01 systemd-networkd[5648]: one-333-0: Link UP

Oct 22 00:06:30 one-lxc01 kernel: [17528.948449] brPrivNat4: port 11(one-333-0) entered blocking state

Oct 22 00:06:30 one-lxc01 kernel: [17528.948453] brPrivNat4: port 11(one-333-0) entered disabled state

Oct 22 00:06:30 one-lxc01 kernel: [17528.948691] device one-333-0 entered promiscuous mode

Oct 22 00:06:30 one-lxc01 kernel: [17528.950306] eth0: renamed from veth0qE3LD

Oct 22 00:06:31 one-lxc01 systemd-networkd[5648]: one-333-0: Link DOWN

Oct 22 00:06:31 one-lxc01 kernel: [17529.107903] brPrivNat4: port 11(one-333-0) entered disabled state

Oct 22 00:06:31 one-lxc01 kernel: [17529.109956] device one-333-0 left promiscuous mode

Oct 22 00:06:31 one-lxc01 kernel: [17529.109962] brPrivNat4: port 11(one-333-0) entered disabled state

Oct 22 00:06:31 one-lxc01 systemd-networkd[5648]: rtnl: received neighbor for link '37' we don't know about, ignoring.

Oct 22 00:06:31 one-lxc01 systemd-networkd[5648]: rtnl: received neighbor for link '37' we don't know about, ignoring.

Oct 22 00:09:50 one-lxc01 systemd[1]: Started Session 73 of user oneadmin.

Oct 22 00:11:10 one-lxc01 systemd[1]: session-73.scope: Succeeded.

Oct 22 00:11:57 one-lxc01 systemd[1]: Started Session 74 of user oneadmin.

Oct 22 00:11:59 one-lxc01 systemd[1]: var-lib-lxc\x2done-331-disk.0.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[181218]: var-lib-lxc\x2done-331-disk.0.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[6171]: var-lib-lxc\x2done-331-disk.0.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[1]: var-lib-one-datastores-0-331-mapper-disk.0.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[181218]: var-lib-one-datastores-0-331-mapper-disk.0.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[6171]: var-lib-one-datastores-0-331-mapper-disk.0.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[6171]: var-lib-lxc\x2done-331-disk.1.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[1]: var-lib-lxc\x2done-331-disk.1.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[181218]: var-lib-lxc\x2done-331-disk.1.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[1]: var-lib-one-datastores-0-331-mapper-disk.1.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[181218]: var-lib-one-datastores-0-331-mapper-disk.1.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 systemd[6171]: var-lib-one-datastores-0-331-mapper-disk.1.mount: Succeeded.

Oct 22 00:11:59 one-lxc01 kernel: [17857.876379] EXT4-fs (loop24): mounted filesystem with ordered data mode. Opts: (null)

Oct 22 00:11:59 one-lxc01 kernel: [17858.003919] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:11:59 one-lxc01 kernel: [17858.004362] ISO 9660 Extensions: Microsoft Joliet Level 3

Oct 22 00:11:59 one-lxc01 kernel: [17858.004579] ISO 9660 Extensions: RRIP_1991A

Oct 22 00:12:00 one-lxc01 systemd-udevd[202095]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

Oct 22 00:12:00 one-lxc01 systemd-udevd[202095]: Using default interface naming scheme 'v245'.

Oct 22 00:12:00 one-lxc01 networkd-dispatcher[5809]: WARNING:Unknown index 38 seen, reloading interface list

Oct 22 00:12:00 one-lxc01 systemd-networkd[5648]: one-331-0: Link UP

Oct 22 00:12:00 one-lxc01 kernel: [17858.134595] brPrivNat4: port 11(one-331-0) entered blocking state

Oct 22 00:12:00 one-lxc01 kernel: [17858.134600] brPrivNat4: port 11(one-331-0) entered disabled state
Oct 22 00:12:00 one-lxc01 kernel: [17858.134789] device one-331-0 entered promiscuous mode
Oct 22 00:12:00 one-lxc01 kernel: [17858.136690] eth0: renamed from vethWkLMTp
Oct 22 00:12:00 one-lxc01 systemd-networkd[5648]: one-331-0: Link DOWN
Oct 22 00:12:00 one-lxc01 kernel: [17858.276074] brPrivNat4: port 11(one-331-0) entered disabled state
Oct 22 00:12:00 one-lxc01 kernel: [17858.278106] device one-331-0 left promiscuous mode
Oct 22 00:12:00 one-lxc01 kernel: [17858.278115] brPrivNat4: port 11(one-331-0) entered disabled state
Oct 22 00:12:00 one-lxc01 systemd-networkd[5648]: rtnl: received neighbor for link '38' we don't know about, ignoring.
Oct 22 00:12:00 one-lxc01 systemd-networkd[5648]: rtnl: received neighbor for link '38' we don't know about, ignoring.
Oct 22 00:14:50 one-lxc01 systemd[1]: Started Session 75 of user oneadmin.
Oct 22 00:16:10 one-lxc01 systemd[1]: session-75.scope: Succeeded.
Oct 22 00:17:01 one-lxc01 CRON[208808]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

I could not figure it out. I re-imaged the server to firecracker - all is well. I will comeback to it later on.