Introduction

I regularly need to test network configurations involving multiple Linux machines. One option is to run one side of the test on my own PC and the other on a second device, such as a Raspberry Pi. This has the advantage that every node physically exists and can be examined directly, but the setup is cumbersome and slow to iterate on, and there is always the risk of leaving the laptop's own configuration in a broken state.

The ideal arrangement is a testbed of virtual machines connected to one another through a shared network segment, each running independently, with access to the internet for downloading packages and updates. As an added requirement, the setup must be reproducible and launchable with a single command from the shell.

This article describes how I built such an environment with QEMU. It was written in 2017; the parts of the setup that would be done differently today are addressed in the Modern notes section.

Installing the guest operating system

Installing a Linux guest in QEMU is straightforward and well documented online [1], so I will not cover it here.

The startup script

The script below launches a testbed consisting of one server and two clients. Each machine is attached to both the internet and a private virtual layer 2 switch [2]. Note that the script must run with root privileges for the network setup to take effect.

#!/bin/bash

# Amount of RAM given to each VM
RAM_SIZE_MB=512

# Additional QEMU options
QEMU_OPTIONS="-enable-kvm -vga std"

# Interface names
TAP_SERVER=tapSrv0
TAP1=tap1
TAP2=tap2
BRIDGE=brVrt0

# Preliminary network interface setup (requires root privileges)
sudo ip tuntap add $TAP_SERVER mode tap user `whoami`
sudo ip link set $TAP_SERVER up

sudo ip tuntap add $TAP1 mode tap user `whoami`
sudo ip link set $TAP1 up

sudo ip tuntap add $TAP2 mode tap user `whoami`
sudo ip link set $TAP2 up

sudo ip link add $BRIDGE type bridge
sudo ip link set $BRIDGE up

sudo ip link set $TAP_SERVER master $BRIDGE
sudo ip link set $TAP1 master $BRIDGE
sudo ip link set $TAP2 master $BRIDGE

# Start the testbed
qemu-system-i386 $QEMU_OPTIONS -m $RAM_SIZE_MB \
	-netdev tap,id=vlan0,ifname=$TAP_SERVER \
	-device e1000,netdev=vlan0,mac=52:54:00:00:00:01 \
	-netdev user,id=user0 -device e1000,netdev=user0 -hda server.cow &

qemu-system-i386 $QEMU_OPTIONS -m $RAM_SIZE_MB \
	-netdev tap,id=vlan0,ifname=$TAP1 \
	-device e1000,netdev=vlan0,mac=52:54:00:00:00:02 \
	-netdev user,id=user0 -device e1000,netdev=user0 -hda client1.cow &

qemu-system-i386 $QEMU_OPTIONS -m $RAM_SIZE_MB \
	-netdev tap,id=vlan0,ifname=$TAP2 \
	-device e1000,netdev=vlan0,mac=52:54:00:00:00:03 \
	-netdev user,id=user0 -device e1000,netdev=user0 -hda client2.cow &

# Wait for all background jobs to terminate
wait

# Teardown of the interfaces
sudo ip tuntap del $TAP_SERVER mode tap
sudo ip tuntap del $TAP1 mode tap
sudo ip tuntap del $TAP2 mode tap

sudo ip link del $BRIDGE

Getting the interface setup right, on both the host and the guest side, was the most difficult part; there is a surprising amount of outdated and misleading material on this topic on the web.

The key point to understand is that QEMU requires a network device instance for every netdev declared on the command line. Each netdev is therefore given an id (-netdev id=netdev_id,netdev_type) and a device is then attached to it with -device hw_type,netdev=netdev_id. In the script this pairing is done twice per machine: once for the tap netdev that connects the guest to the private switch, and once for the user netdev that provides internet access through QEMU's built-in user-mode networking. The role of each netdev type is described in QEMU's networking documentation [3].

On the host side, the desired topology is achieved by bridging the TAP interfaces together before the virtual machines are started. The relevant commands are taken from the KVM networking guide [4] and adapted to this situation.

Each virtual machine is launched as a background job (note the & at the end of each command), so the script issues a wait before tearing the network down: it halts until every job has terminated, guaranteeing that the interfaces are not removed while a guest is still using them.

Modern notes

The setup described above remains valid on current QEMU and Linux releases, but a few of the choices made in 2017 would be updated today:

• qemu-system-i386 emulates a 32‑bit x86 machine. Unless a 32‑bit guest is specifically needed, the modern equivalent is qemu-system-x86_64, which matches the architecture of current distributions.
• The .cow suffix identifies images in the legacy QCOW version 1 format, which still works but has no block device layering or many of the performance features of its successor. New images are created in the QCOW2 format (suffix .qcow2), for example with qemu-img create -f qcow2 server.qcow2 10G.
• The user netdev is QEMU's built‑in user‑mode networking (SLiRP): it is convenient because it requires no host configuration, but it is NAT‑only, so the host cannot initiate connections to the guests. For most testbeds this is acceptable; where inbound reachability is required, the host‑side bridge can be extended with an uplink and an iptables NAT rule, or the lighter‑weight slirp4netns user‑mode stack can be used in place of the user netdev.

References