Live migration

Introduction

Virtual machine live migration is the key feature provided by StratoVirt. It needs to execute virtual machine migration when any of the following happens:

• Server overload: when a source server is overloaded, a set of the VMs from this server is migrated to an underloaded server using VM migration technique.
• Server maintenance: if there is a need for server maintenance, VMs from the source server are migrated to another server.
• Server fault: whenever there is server fault, VMs are migrated from the faulty server to the target server.

Transports

The migration stream can be passed over any transport as following:

• TCP mode migration: using tcp sockets to do the migration.
• UNIX mode migration: using unix sockets to do the migration.

Note: UNIX mode only supports migrate two VMs on the same host OS. TCP mode supports migrate both on the same or different host OS.

Migration

Launch the source VM:

./stratovirt \
    -machine q35 \
    -kernel ./vmlinux.bin \
    -append "console=ttyS0 pci=off reboot=k quiet panic=1 root=/dev/vda" \
    -drive file=path/to/rootfs,id=rootfs,readonly=off,direct=off \
    -device virtio-blk-pci,drive=rootfs,id=rootfs,bus=pcie.0,addr=0 \
    -qmp unix:path/to/socket1,server,nowait \
    -serial stdio \

Launch the destination VM:

./stratovirt \
    -machine q35 \
    -kernel ./vmlinux.bin \
    -append "console=ttyS0 pci=off reboot=k quiet panic=1 root=/dev/vda" \
    -drive file=path/to/rootfs,id=rootfs,readonly=off,direct=off \
    -device virtio-blk-pci,drive=rootfs,id=rootfs,bus=pcie.0,addr=0 \
    -qmp unix:path/to/socket2,server,nowait \
    -serial stdio \
    -incoming tcp:192.168.0.1:4446 \

Note:

• The destination VM command line parameter needs to be consistent with the source VM.
• If it is necessary to change the data transmission from tcp network protocol to unix socket, the parameter -incoming tcp:192.168.0.1:4446 needs to be replaced with -incoming unix:/tmp/stratovirt-migrate.socket.
• Unix socket protocol only supports migrate two VMs on the same host OS.

Start to send migration for the source VM:

$ ncat -U path/to/socket1
<- {"QMP":{"version":{"StratoVirt":{"micro":1,"minor":0,"major":0},"package":""},"capabilities":[]}}
-> {"execute":"migrate", "arguments":{"uri":"tcp:192.168.0.1:4446"}}
<- {"return":{}}

Note:

• If using unix socket protocol to migrate vm, you need to modify QMP command of "uri":"tcp:192.168.0.1:4446" to "uri":"unix:/tmp/stratovirt-migrate.socket".

When finish executing the command line, the live migration is start. in a moment, the source VM should be successfully migrated to the destination VM.

Cancel Migration

If you want to cancel the live migration, executing the following command:

$ ncat -U path/to/socket1
<- {"QMP":{"version":{"StratoVirt":{"micro":1,"minor":0,"major":0},"package":""},"capabilities":[]}}
-> {"execute":"migrate_cancel"}
<- {"return":{}}

Query migration state

Use QMP command query-migrate to check migration state:

$ ncat -U path/to/socket
<- {"QMP":{"version":{"StratoVirt":{"micro":1,"minor":0,"major":0},"package":""},"capabilities":[]}}
-> {"execute":"query-migrate"}
<- {"return":{"status":"completed"}}

Now there are 5 states during migration:

• None: Resource is not prepared all.
• Setup: Resource is setup, ready to migration.
• Active: In migration.
• Completed: Migration completed.
• Failed: Migration failed.
• Canceled: Migration canceled.

Limitations

Migration supports machine type:

• q35 (on x86_64 platform)
• virt (on aarch64 platform)

Some devices and feature don't support to be migration yet:

• vhost-net
• vhost-user-net
• vfio devices
• balloon
• mem-shared,backend file of memory
• pmu
• sve
• gic-version=2

Some device attributes can't be changed:

• virtio-net: mac
• virtio-blk: file(only ordinary file or copy file), serial_num
• device: bus, addr
• smp
• m

If hot plug device before migrate source vm, add newly replaced device command should be add to destination vm.

Before live migration:

• source and destination host CPU needs to be the same architecture.
• the VMs image needs to be shared by source and destination.
• live migration may fail if the VM is performing lifecycle operations, such as reboot, shutdown.
• the command to startup the VM needs to be consistent on source and destination host.

During live migration:

• source and destination networks cannot be disconnected.
• it is banned to operate VM lifecycle, includes using the QMP command and executing in the VM.
• live migration time is affected by network performance, total memory of VM and applications.

After live migration:

• it needs to wait for the source VM to release resources before fetching back the live migration operation.