# Multi-tiered local storage for VMs

**URL:** <https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950>\
**Category:** Installation & Configuration\
**Created:** [November 15, 2022, 10:52am UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950 "2022-11-15T10:52:36Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![spa90](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@spa90](https://forum.opennebula.io/u/spa90)\
**Post date:** [November 15, 2022, 10:52am UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950/1 "2022-11-15T10:52:37Z")

</div>

Hello,

I’m running OpenNebula 6.4 installed via minione as front-end coupled with two Dell servers running Rocky Linux 8.6 with KVM as hypervisor, installed via the official AlmaLinux OpenNebula repository following the [official docs](https://docs.opennebula.io/6.4/open_cluster_deployment/kvm_node/kvm_node_installation.html).

I’m trying to deploy a VM using **local storage** using the _ssh_ transfer driver configuration but would like to place:

- the OS disk on a slower (hdd) datastore
- the data disk on a faster (ssd) datastore

On the frontend I’ve created a cluster together with:

- a datastore for the slow disks
- another one for the fast disks

On the hypervisors I’ve mounted the slow & fast disks under their respective `/var/lib/one/datastores/<ID>` paths.

I’ve downloaded the OpenNebula Marketplace Alpine Linux 3.15 image and launched a VM with a single OS disk pointing to the slow datastore.  
Upon launch I tried adding a new data disk but found I couldn’t point to another datastore.  
Doing the reverse also works i.e., placing the VM on the fast datastore, so I am sure both datastores are working fine.

I’ve tried various scenarios like:

- specifying both datastores during VM deployment under `Deploy VM in a specific Datastore` and later trying to attach another disk from the secondary datastore
- creating a `DATABLOCK` image in the image datastore and trying to attach it thru the Storage tab of the new VM as `Image`
- creating a new volatile image hoping I’ll be able to specify its location

I also searched the forums for information for similar scenarios but couldn’t find anything relevant except for [this](https://forum.opennebula.io/t/how-to-use-a-second-datastore/7323/16) thread; the path to fixing the issue unfortunately doesn’t match my scenario.

So my question is: is it possible to use OpenNebula-managed KVM hypervisors with Local Storage while splitting the OS and data disks of a VM on different datastores?

Thank you

---

<div class="post-metadata">

**Author:** ![rdziwinski](https://avatars.discourse-cdn.com/v4/letter/r/a4c791/32.png) [@rdziwinski](https://forum.opennebula.io/u/rdziwinski)\
**Post date:** [November 16, 2022, 9:26am UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950/2 "2022-11-16T09:26:02Z")

</div>

Hello,  
i have similar problem. I was tried add disk from another DS via pyone and this way is also wrong.

> [@Unable to add disk from other TM\_MAD](https://forum.opennebula.io/t/unable-to-add-disk-from-other-tm-mad/10949):
>
> Hi community slight_smile I have template with disk in local storage DISK\_TYPE=FILE. Image is download from another DS (ceph) via ssh deploy mode. I created VM with it and it’s fine. Next. I’m trying to add custom image from ceph storage (other tm\_mad, tm\_mad\_system, datastore). I don’t find it in Sunstone so i was tried via pyone. Unfortunately it’s not working and i still can’t add new image. My comand: id = 427 disk='DISK=[IMAGE\_ID=147, CLONE=YES, CLONE\_TARGET=SELF, DISCARD=unmap, IO=na…

---

<div class="post-metadata">

**Author:** ![rdziwinski](https://avatars.discourse-cdn.com/v4/letter/r/a4c791/32.png) [@rdziwinski](https://forum.opennebula.io/u/rdziwinski)\
**Post date:** [October 30, 2023, 12:00pm UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950/3 "2023-10-30T12:00:50Z")

</div>

Hello, any information about it? I think multiple DS in one VM will be very useful.

---

<div class="post-metadata">

**Author:** ![vpalma](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/vpalma/32/8065_2.png) [@vpalma](https://forum.opennebula.io/u/vpalma)\
**Post date:** [October 30, 2023, 2:53pm UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950/4 "2023-10-30T14:53:57Z")

</div>

Hi @rdziwinski 👋

You can use images from different datastores in a single virtual machine, but you need the IMAGE and SYSTEM datastore to use the same `TM_MAD` for each disk.

---

<div class="post-metadata">

**Author:** ![spa90](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@spa90](https://forum.opennebula.io/u/spa90)\
**Post date:** [November 17, 2023, 9:19am UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950/5 "2023-11-17T09:19:40Z")

</div>

Hey, Victor,

Thanks for the insight; my initial question was whether placing a VMs disks on multiple datastores is possible.

Am I correct in understanding that what you are saying is: that you are able to pull a VMs disks from different image datastores; however, said VMs disks will be placed on the same system datastore?

Thanks!

---

<div class="post-metadata">

**Author:** ![vpalma](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/vpalma/32/8065_2.png) [@vpalma](https://forum.opennebula.io/u/vpalma)\
**Post date:** [November 24, 2023, 8:48am UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950/6 "2023-11-24T08:48:12Z")

</div>

Hi @spa90,

Let me explain better.

You can have different disks of different datastores in a virtual machine, the only exception is that each disk must share the same type of `TM_MAD` for its IMAGE and SYSTEM datastore.

So, let’s assume the following. You have a VM with this configuration:

- A first disk coming from an SSH IMAGE datastore.
- A second disk coming from an LVM IMAGE datastore.

For both cases, on the host where you are going to deploy the VM, you need to have configured an SSH SYSTEM datastore and an LVM SYSTEM datastore. Otherwise, it will not work properly. Note also that using different datastores within the same virtual machine greatly limits the operations to be performed on it. In this case for example, even if you have a disk in an LVM SYSTEM datastore, you will not be able to migrate the machine in case of Host error because the other disk in the SSH SYSTEM datastore will not allow it.

You can read more about the differences between SYSTEM and IMAGE datastore [here](https://docs.opennebula.io/6.4/management_and_operations/storage_management/datastores.html).

Best,  
Victor.

---

<div class="post-metadata">

**Author:** ![Haiming\_Chen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/haiming_chen/32/8584_2.png) [@Haiming\_Chen](https://forum.opennebula.io/u/Haiming_Chen)\
**Post date:** [June 4, 2025, 11:20am UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950/7 "2025-06-04T11:20:49Z")

</div>

hi @vpalma

You misunderstood. The requirement is not to transfer images between different image datastores, but to place the **OS disk** and **data disk** on different types of hardware storage (e.g., HDD and SSD).

For example, in the following `vm-guest` XML configuration:

- `disk.0` (OS disk) is stored on an **HDD** (corresponding to system datastore ID `0`).
- `disk.2` (data disk) is stored on an **SSD** (corresponding to system datastore ID `100`).

Here’s the XML snippet:

```xml
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='writethrough' discard='unmap'/>
  <source file='/var/lib/one/datastores/0/disk.0' index='2'/>
  <backingStore/>
  <target dev='vda' bus='virtio'/>
  <boot order='1'/>
  <alias name='virtio-disk0'/>
  <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='writethrough' discard='unmap'/>
  <source file='/var/lib/one/datastores/100/disk.2' index='1'/>
  <backingStore/>
  <target dev='vdb' bus='virtio'/>
  <boot order='2'/>
  <alias name='virtio-disk2'/>
  <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</disk>

```

Does this clarify the requirement? Let me know if further details are needed.

---

<div class="post-metadata">

**Author:** ![rdziwinski](https://avatars.discourse-cdn.com/v4/letter/r/a4c791/32.png) [@rdziwinski](https://forum.opennebula.io/u/rdziwinski)\
**Post date:** [November 14, 2025, 5:00pm UTC](https://forum.opennebula.io/t/multi-tiered-local-storage-for-vms/10950/8 "2025-11-14T17:00:14Z")

</div>

Hi,

As we noted during testing, it’s not possible to have 1 disk in the local datastore (eg, qcow) and a second disk in the Ceph datastore. Something like this:

```auto

<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap' iothread='1'/>
  <source file='/var/lib/one//datastores/167/3194/disk.0' index='2'/> 
</disk>

<disk type='network' device='disk'>      
  <driver name='qemu' type='raw' cache='none' io='native' discard='unmap' iothread='1'/>      
  <auth username='nebula-public'>
    <secret type='ceph' uuid='XXXXXXXXXXXXXXXXX'/> 
  </auth>      
  <source protocol='rbd' name='nebula-public/one-1-0' index='2'>        
    <host name='10..10.10.10' port='6789'/>      
  </source>
</disk>

```

why? That is a very popular and useful case. System disk in fast, local NVMe storage and a second disk for data archive in Ceph Datastore. What is the problem with implementing that?