bbvirt - hotplug BitBabbler devices into libvirt managed domains
program is an attempt to take some of the pain out of what is
currently required to distribute multiple USB devices between the host and
guest virtual machines. While there are several ways in which this may be
configured and managed, at present none of them actually provide a complete
and coherent solution on their own, all of them fall short of the mark in some
significant and annoying way. The aim here is to piece together enough of
those hacks to actually get all of the functionality that we want now, until
the libvirt native support for this improves enough to not need it anymore.
At present this deals with libvirt managed QEMU/KVM virtual machines.
The ideal behaviour here is pretty simple. Given some arbitrary number of
BitBabbler devices, we should be able to assign them to either the host
machine, or to a guest VM running on it, and once we do that they should
behave in the normal manner expected of any USB device.
- If they are plugged in when the guest machine is started,
they should be seen by that machine as they would be by the host.
- If they are plugged in after the machine is started, they
should be hotplugged into that machine as they would be on the host.
- If they are unplugged while the machine is running, they
should be cleanly removed from it, as they would be on the host.
Right now, libvirt gives us two ways that we can assign USB devices from the
host to a guest domain.
- We can assign them by their USB vendor and product ID. But
that only works when there is just a single device of that type in the
host. Which is pretty useless in most of the cases that we care about
here, where the host and each of the guests are likely to have one or more
BitBabbler devices of their own assigned to them.
- We can assign them by their logical address on the USB bus.
But that isn't a constant that we can statically configure for the domain.
Every time a device is plugged in, or replugged, or reset, or the host
machine is rebooted, that address is likely to change since it is
dynamically allocated when the device is enumerated on the bus.
There is a third way, but it relies on bypassing the normal libvirt
configuration to make direct use of the QEMU ability to assign a device by its
physical address on the bus. Which is better, but still not a magic bullet
since it relies on plugging exactly the same devices into exactly the same
ports every time (and on having those ports enumerated in the same way by the
host on every reboot, which isn't guaranteed either). It also forces us to
jump through other hoops, since we then need additional complication to manage
the access permissions of the device manually outside of libvirt, but still in
coordination with it.
The even bigger failing, which all of those methods have in common, is they all
depend on the device already being plugged in before the guest is started. If
it is inserted after the guest is started, or removed and replugged while the
guest is running, or if the host bus or a hub bounces causing a reconnect,
then the device will not be (re)attached to the guest. The only way to fix
that if it happens is to manually reattach the device with an arcane
incantation in XML (which relies on you knowing the new address of the
device), or to completely power down and restart the guest. Not the pinnacle
of user-friendly operation that we are looking for here.
There was a patch submitted to libvirt some years back which would have allowed
a device to be specified by both its USB product ID and its serial number, but
that got some push-back, and so far has still not been applied upstream. That
would have gone a long way toward making this both easy and clean, leaving us
only with the hotplug aspect to deal with. We'll leave grumpy snark about that
as an exercise for the reader ...
Another alternative is we can delegate finding the device's logical address to a
hotplug manager like udev
(7). This is attractive in the sense that we
can know when the address of a device changes and what it changes to, but
itself isn't very friendly to the idea of local admin
customisation (while it is possible to do, it seems to be getting increasingly
strongly discouraged) and using it still requires some external glue to
translate its events into something that libvirt can act on to configure the
program provides that glue, and a user friendly method of
assigning which devices should belong to which guest domains, and a front end
that can be invoked manually or by other admin controlled tasks to quickly and
easily add or remove BitBabbler devices from any of the running guest
But the limitation this approach has, is that it can't easily know when a guest
machine is started which should have devices that are already plugged in added
to it. In theory we could add them to its persistent domain definition, but
that has its own problems because we can only add devices by their ephemeral
logical address, and we can't guarantee that we will get called to remove them
from the domain again when that address becomes invalid (like if the host is
suddenly powered off or it is otherwise not cleanly shut down), so we could
end up with many stale entries accumulating in the persistent domain
configuration, which could later match some completely different device to
what we had wanted attached to it. Which means until that somehow gets fixed,
it's only safe to add them to a live guest domain, so that they will always be
removed again when it is halted, no matter how it ended up getting halted.
Clearly we've still got some way to go to get to our ideal here.
There appears to be only two ways that we can get notified of a guest machine
being started at present. One involves running yet another daemon process,
which would do little more than just sit around waiting for someone to start a
guest so it could tell us about that. But then we'd have yet another thing to
configure, yet another process running, and yet more problems with figuring
out how to ensure we don't lose a race when the host is booted, between
getting the initial set of device events, that process being ready and active,
and any guests that will be autostarted at boot actually starting.
The other way is to use a libvirt hook. Which in turn has the problem of not
actually allowing us to run any libvirt functions from it, which we need to do
in order to attach the device to the host. And which we can't guarantee that
we can just install by default, because there can be only one such hook on the
system, which the local admin may already be using ...
There is a third way, but that would involve requiring the local admin to start
all guest machines through a wrapper of our own, instead of via whatever
mechanism they already know and use. Which doesn't scale to support other USB
devices in the same situation, among the many ways that would be a horrible
solution to inflict on people.
But there is a loophole we can exploit. We can use the libvirt qemu hook to
trigger a change event for udev
, which can in turn invoke bbvirt
in much the same way that would happen if the device was really hotplugged,
which gives us the extra layer of indirection we need to be able to safely do
that from the hook. Rube Goldberg would be proud, and some of the pieces may
require hand-assembly, but with all of this in place, we can have something
resembling normal USB functionality in the guest machines.
It's not pretty, but it will work with what we have to work with.
To string this together, you'll need to ensure all of the following:
- The udev(7) rules from the bit-babbler package are
installed. If you installed this from the Debian packages that should
already be done. If you didn't, you will need to install the rules that
are found in debian/bit-babbler.udev from the source package to a
suitable place on your system (probably /etc/udev/rules.d).
- The bbvirt(1) script is installed in a place where
the udev rules will find it. If you didn't install this from the
Debian packages, and it isn't in /usr/bin, then you'll need to
tweak the udev rules to suit.
- The devices you wish to use in guest machines, and the
machines you wish to use them in, are specified in the bbvirt
configuration file. The default location for that is
/etc/bit-babbler/vm.conf. If you wish to use a different file you
will need to pass its location with the --config option in the
udev rules, and update the hook script use that file too. The
details of what you can put in that file are described in the
CONFIGURATION OPTIONS section below.
- The libvirt hook file is installed. If all the above is
done, then devices will be added to the running guest machines if they get
plugged in while the guest is running. This last step ensures devices
which are already plugged in will be added to newly started guests too
(which includes guests that are started automatically when the host
Until there is some safe way we can install this without conflicting with or
overwriting an existing hook, everyone will need to do this step manually.
If you have installed the Debian packages, then the example hook script
that we've provided for this can be found in
/usr/share/doc/bit-babbler/examples/qemu-hook. If you didn't it can
be found in libvirt/qemu-hook of the source package.
You will need to install that file as /etc/libvirt/hooks/qemu, or
merge its content with the existing qemu file there if you already
have that hook set. If that file did not previously exist, you will need
to restart libvirtd(8) to get it to begin using it.
That should cover all of the needed automation, but you can also attach and
detach devices manually at any time too. The details of doing that will be
described in the following section. Otherwise, with all the above done, there
is no other reason to need to invoke bbvirt
There are two primary modes of operation for bbvirt
which are selected by
the initial action option. If the action to perform is attach
then only a single device will be acted upon, and which device
that should be must be specified explicitly, even if there is only one device
present on the host at the time. When invoking bbvirt
may be specified by its serial number, its logical address on
the bus (in the form busnum
, given as decimal integers),
or its physical address on the bus (in the form
If the action to perform is attach-all
, then the
device(s) to act upon are selected by domain
association instead. If a
is explicitly specified, then all devices which are assigned to
that guest domain in the configuration file will be acted upon in the same way
as if bbvirt
was invoked for each of them individually with the
action. If no domain
is provided, then
all of the configured guest domains will be acted upon in this way.
The following additional options are available:
- -C, --config
- Specify an alternative configuration file to import the
device assignments from. If the path to the file is not provided
explicitly, then it will be looked for in the /etc/bit-babbler
directory (with a .conf suffix).
- -c, --connect=URI
- Specify the virsh(1) connection URI to use.
This will override a DOMAIN_URI set for the domain in the
configuration file. If that is not set using either of these methods then
the virsh default for the user running bbvirt will be used.
- -D, --domain=name
- Specify the libvirt domain to act upon. This may be used to
override the device allocation from the configuration file when
bbvirt is invoked manually, or to act on a device or domain that is
not currently specified in the configuration file.
- -b, --busnum=num
- Specify the USB bus number that the device is attached to.
This option is mostly used to avoid bbvirt needing to look this up
when it is already known (such as when it is called from a udev
rule). There isn't usually much reason to pass this if invoking
bbvirt manually, since you can just specify the device by its
logical or physical address instead.
- -d, --devnum=num
- Specify the USB device number that the device is currently
assigned. Together with the bus number, this forms the logical address of
the device. This option is mostly used to avoid bbvirt needing to
look this up when it is already known (such as when it is called from a
udev rule). There isn't usually much reason to pass this if
invoking bbvirt manually, since you can just specify the device by
its logical address instead.
- -n, --dry-run
- Don't attach or detach any devices, just show what would be
attempted if this was a live run. This option implies a minimal level of
--verbose, but the verbosity may be increased further by also
passing that option explicitly.
- -v, --verbose
- Make more noise about what is really going on. It may be
passed multiple times to increase the verbosity further.
- -?, --help
- Show a brief summary of the available options.
configuration file contains variable assignments using the
(1) shell syntax. It is sourced as a shell snippet, so you could in
principle construct the configuration for each domain dynamically, but most
typically a simple static assignment of devices to domains will suffice. If
you do elect to run code in it, you should be very defensive about namespacing
any other variables you use, or any other side effects you might cause to
happen. Any number of guest domains may be configured in it.
For each guest domain, two variables control the behaviour of bbvirt
- This variable is optional, and sets the virsh(1)
connection URI to use when attaching or detaching devices from the
given domain. If the --connect option is explicitly passed
to bbvirt it will override what is set here. If the connection
URI is not set using either of these methods then the virsh
default for the user running bbvirt will be used (which would
normally be root if run from udev).
- DOMAIN_RNG_domain=( device serial
numbers ... )
- This variable is required if automatic passthrough of
devices to a domain is desired. It is a bash array, populated with a space
separated list of all the device serial numbers that you want assigned to
domain. It is not an error for devices to be listed here which are
not currently plugged in. It is important to ensure that devices are only
assigned to one domain though, and that devices assigned to guest
domains will not be used by a seedd(1) instance running on the host
(which means the seedd configuration needs to be passed an explicit
list of the devices that it may use too).
The device serial number must always be used here. You cannot specify a
device by its logical or physical address on the bus (like you can in most
other places where we take a device ID).
- The default configuration file for assigning BitBabbler
devices to libvirt managed virtual machine domains.
- The default udev(7) rules granting direct device
access to users in the group bit-babbler, enabling USB autosuspend
when the device is idle, and invoking bbvirt to handle device
hotplug for virtual machines. These can be overridden by creating
/etc/udev/rules.d/60-bit-babbler.rules and populating it with your
- The libvirt hook script needed to enable cold-plugging of
already present devices into newly (re)started virtual machines.
was written by Ron <firstname.lastname@example.org>. You can send bug
reports, feature requests, praise and complaints to email@example.com.