nbdview: Add -topmost attribute.

[libguestfs-talks.git] / 2016-eng-talk / paper.tex

\documentclass[12pt,a4paper]{article}
\usepackage[utf8x]{inputenc}
\usepackage{parskip}
\usepackage{hyperref}
\usepackage{xcolor}
\hypersetup{
    colorlinks,
    linkcolor={red!50!black},
    citecolor={blue!50!black},
    urlcolor={blue!80!black}
}
\usepackage{abstract}
\usepackage{graphicx}
\DeclareGraphicsExtensions{.pdf,.png,.jpg}
\usepackage{float}
\floatstyle{boxed}
\restylefloat{figure}
\usepackage{fancyhdr}
  \pagestyle{fancy}
  %\fancyhead{}
  %\fancyfoot{}

\title{Optimizing QEMU boot time}
\author{
\large
Richard W.M. Jones
\normalsize Red Hat Inc.
\normalsize \href{mailto:rjones@redhat.com}{rjones@redhat.com}
}
\date{}

\begin{document}
\maketitle

\begin{abstract}
Everyone knows that containers are really fast and lightweight, and
full virtualization is slow and heavyweight ... Or that's what we
thought, until Intel demonstrated full Linux virtual machines booting
as fast as containers and using as little memory.  Intel's work used
kvmtool and a customized, cut down guest kernel.  Can we do the same
using libvirt, QEMU, SeaBIOS, and an off the shelf Linux distro
kernel?  The short answer is \textit{no}, but we can get pretty close,
and it was an exciting journey learning about unexpected performance
roadblocks, developing tools to measure the boot process, and shaving
off milliseconds all over the place.  The work has practical
significance because it will allow us to deploy secure containers,
protected by hardware virtualization.  Even if you never plan to use
containers, you're still benefiting from a faster QEMU experience.
\end{abstract}

\section{Intel Clear Linux}

Intel's Clear Linux means a lot of different things to different
people.  I'm only going to talk about a narrow aspect of it, usually
known as ``Clear Containers'', but if other people talk about Intel
Clear Linux they might be talking about a Linux distribution,
OpenStack or graphics technologies.

LWN has a useful and relatively recent introduction to Clear
Containers \url{https://lwn.net/Articles/644675/}.

Until recently Intel hosted a Clear Containers demo.  If you
downloaded it and ran \texttt{bash~./boot.sh} then it booted into a
full Linux VM in about 150ms, and using 20~MB of RAM.

Intel are using this technology along with a customized Docker driver
to run Docker containers safely inside a VM.  The overhead (150ms /
20~MB) is very attractive since it doesn't impact on the density that
containers give you.  It's also aligned with Intel's interests, since
they are selling chips with VT, VT-d, EPT, VPID and so on and they
need people to use those features.

The Clear Containers demo uses \texttt{kvmtool} with several
non-upstream patches such as for DAX and 64 bit guests.  Since first
demonstrating Clear Containers, Intel has worked on getting vNVDIMM
(needed for DAX) into QEMU.

The Clear Containers demo from last year uses a patched Linux kernel.
There are many non-upstream patches.  More importantly they use a
custom, cut down configuration where many subsystems not used by VMs
are cut out entirely.

\section{Real Linux distros use QEMU}

Can we do the same sort of thing in our Linux distros?  Let's talk
about some things that constrain us in Fedora.

We'd prefer to use QEMU over kvmtool.  QEMU isn't really ``bloated''.
It's featureful, but (generally) if you're not using those features
they don't slow things down.

We \textit{can't} use the heavily patched and customized kernel.
Fedora is strictly ``upstream first''.  Fedora also ships a single
kernel image for baremetal, virtual machines and all other uses, since
building and maintaining multiple kernels is a huge pain.

\section{Stating the problem}

What we want to do is to boot up and shut down a modern Linux kernel
in a KVM virtual machine on a modern Linux host.  Inside the virtual
machine we will eventually want to run our Docker container.  However
I am just concentrating on the overhead of the boot and shutdown.

\begin{samepage}
Conveniently -- and also the reason I'm interested in this problem --
libguestfs does almost the same thing.  It starts up and shuts down a
small Linux-based appliance.  If you have \texttt{guestfish}
installed, then you can try running the command below (several times
so you have a warm cache).  Add \texttt{-v~-x} to the command line to
see what's really going on.

\begin{verbatim}
$ guestfish -a /dev/null run
\end{verbatim}
\end{samepage}

\section{Measurements}

The first step to improving the situation is to build tools that can
accurately measure the time taken for each step in the boot process.

Booting a Linux kernel under QEMU using the \texttt{-kernel} option
looks like table~\ref{tab:kernel-steps}.

\begin{table}[h]
\caption{Steps run when you use QEMU \texttt{-kernel}}
\centering
\begin{tabular}{l}
  query QEMU's capabilities \\
  \hline
  run QEMU \\
  \hline
  run SeaBIOS \\
  \hline
  run the kernel \\
  \hline
  run the initramfs \\
  \hline
  load kernel modules \\
  \hline
  mount and pivot to the root filesystem \\
  \hline
  run \texttt{/init}, \texttt{udevd} etc \\
  \hline
  perform the desired task \\
  \hline
  shutdown \\
  \hline
  exit QEMU
\end{tabular}
\label{tab:kernel-steps}
\end{table}

How do you know when SeaBIOS starts or various kernel events happen?

I started out looking at various QEMU tracing options, but ended up
using a very simple technique: Attach a serial console to QEMU,
timestamp the messages as they arrive, and use regular expression
string matches to find significant events.

The three programs I wrote (two in C and one in Perl) use libguestfs
as a convenient framework, since libguestfs has the machinery already
for creating VMs, initramfses, capturing serial console output etc.
They are:

\begin{itemize}
\item \texttt{boot-benchmark}

\texttt{boot-benchmark} runs the boot up sequence repeatedly, throwing
away the first few runs (to warm the cache) and collecting the mean
test time and standard deviation.

\begin{verbatim}
$ ./boot-benchmark
Warming up the libguestfs cache ...
Running the tests ...

test version: libguestfs 1.33.29
 test passes: 10
host version: Linux moo.home.annexia.org 4.4.4-301.fc23.x86_64 #1 SMP
    host CPU: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
     backend: direct               [to change set $LIBGUESTFS_BACKEND]
        qemu: /home/rjones/d/qemu/x86_64-softmmu/qemu-system-x86_64
qemu version: QEMU emulator version 2.6.50, Copyright (c) 2003-2008
         smp: 1                    [to change use --smp option]
     memsize: 500                  [to change use --memsize option]
      append:                      [to change use --append option]

Result: 568.2ms ±8.7ms
\end{verbatim}

\item \texttt{boot-benchmark-range.pl}

\texttt{boot-benchmark-range.pl} is a wrapper script around
\texttt{boot-benchmark} which lets you benchmark across a range of
commits from some other project (eg. QEMU or the kernel).  You can
easily see which commits are causing or solving performance problems
as in the example below:

\begin{verbatim}
$ ./boot-benchmark-range.pl ~/d/qemu 3123bd8^..8e86aa8
da34fed hw/ppc/spapr: Fix crash when specifying bad[...]
        1666.8ms ±2.5ms

3123bd8 Merge remote-tracking branch 'remotes/dgibson/[...]
        1658.8ms ±4.2ms

f419a62 (origin/master, origin/HEAD, master) usb/uhci: move[...]
        1671.3ms ±17.0ms

8e86aa8 Add optionrom compatible with fw_cfg DMA version
        1013.7ms ±3.0ms ↑ improves performance by 64.9%
\end{verbatim}

\item \texttt{boot-analysis}

\begin{figure}[h]
\caption{boot-analysis timeline}
\includegraphics[width=0.9\textwidth]{boot-analysis-screenshot}
\label{fig:ba-timeline}
\end{figure}

\texttt{boot-analysis} performs multiple runs of the boot sequence.
It enables the QEMU serial console (and other events from libguestfs),
timestamps the events, and then presents the sequence graphically as
shown in figure~\ref{fig:ba-timeline}.  Also shown are mean times and standard
deviations and percentage of the total run time.

\begin{figure}[h]
\caption{boot-analysis longest activities}
\includegraphics[width=0.9\textwidth]{boot-analysis-screenshot-2}
\label{fig:ba-longest}
\end{figure}

This test also prints which activities took the longest time, see
figure~\ref{fig:ba-longest}.

\end{itemize}

The source for these tools is here:
\url{https://github.com/libguestfs/libguestfs/tree/master/utils}.

Only now that we have the right tools to hand can we work out what
activities take time.

For consistency, all times displayed by the tool are in milliseconds
(ms), and I try to use the same convention in this paper.

In this paper I'm using times based on my laptop, an
Intel\textregistered Core\texttrademark i7-5600U CPU @ 2.60GHz
(Broadwell~U).  This does of course mean that these results won't be
exactly reproducible, but it is hoped that with similar hardware you
will get times that differ only by a scale factor.

\section{glibc}

Surprisingly the first problem is glibc.  QEMU links to over 170
libraries, and that number keeps growing.  A simple
\texttt{qemu~-version} takes up to 60ms, and examining this with
\texttt{perf} showed two things:

\begin{itemize}
\item Ceph had a bug where it ran some \texttt{rdtsc} benchmarks in a
  constructor function.  This is now fixed.
\item The glibc link loader is really slow when presented with lots of
  libraries and lots of symbols.
\end{itemize}

The second problem is intractable.  We can't link to fewer libraries,
because each of those libraries represents some feature that someone
wants, like Ceph, or Gtk support (though if you remove the Gtk
dependency the link time reduces substantially).  And the link loader
is bound by all sorts of obscure ELF rules (eg. symbol interposition)
which we don't need but cannot avoid and make things slow.

When I said earlier that QEMU features don't slow things down, this is
an exception.

We can run QEMU fewer times.  There are several places where we need
to run QEMU.  Obviously one place is where we start the virtual
machine, and the overhead there cannot be avoided.  But also we
perform QEMU feature detection by running commands like
\texttt{qemu~-help} and \texttt{qemu~-devices~\textbackslash?} and
libguestfs now caches that output.

\section{QEMU}

Libguestfs, Intel Clear Containers, and any future Docker container
support we build will use \texttt{-kernel} and \texttt{-initrd} or
their equivalent.  In QEMU up to 2.6 on x86-64 this was implemented
using an interface called \texttt{fw\_cfg} and a PIO loop, and that is
very slow.  To load the kernel and very small initrd used by
libguestfs takes around 700ms.  In QEMU 2.7 we have added a pseudo-DMA
mode which makes this step almost instant.

To see debugging messages from the kernel and to collect our benchmark
results, we have to use an emulated 16550A UART (serial port).
Virtio-console exists but isn't a good replacement because it can't be
used to get BIOS and very early kernel messages.  The UART is slow.
It takes about 4µs per character, or approximately 1ms for 3 lines of
text.  Enabling debugging changes the results subtly.

To get serial console output from the BIOS, we use a
Google-contributed option ROM called SGABIOS.  It quickly became clear
that SGABIOS introduced a 260ms boot delay.  This happened because it
expects to be talking to a real serial terminal, so it sends control
sequences to query the width and height of this ``terminal''.  These
weren't being answered by the actual reader (libguestfs simply reads).
The solution was to modify libguestfs to respond to the control
sequence with a dummy reply, which reduced the delay to almost
nothing.

\section{libvirt}

Libguestfs can optionally use libvirt to manage the QEMU process.
When I did this it was obvious that libvirt was adding a (precisely)
200ms delay.  I tracked this down to a poorly implemented polling loop
in libvirt, waiting for the QEMU monitor socket to be created by QEMU.
I fixed it by changing the loop to use exponential backoff.  A better
fix would involve passing pre-created file descriptors to QEMU.

\section{SeaBIOS}

SeaBIOS wastes time probing for boot devices even though we will use
the \texttt{linuxboot} option ROM to boot (via \texttt{-kernel}).  By
building a \texttt{bios-fast.bin} variant of SeaBIOS with many unused
features disabled we can reduce the time spent inside the BIOS from
about 63ms to about 19ms.

\section{kernel}

PCI probing is slow, taking around 95ms for a guest with just two
virtio-scsi drives.  It turns out that it's not the scanning of the
PCI device space which is slow, but the initialization of each device
as it is found.  QEMU's i440fx machine model exports some legacy
devices which cannot be switched off, and that is unhelpful.

I implemented experimental support for parallel PCI probing using the
kernel ``async'' feature.  With 1~vCPU this slows things down very
slightly as expected.  With 4~vCPUs performance improved by about
30\%.  Unfortunately we can't use it because of the next point.

You would think multiple vCPUs would be better and faster than 1~vCPU,
but that is not the case.  It actually has a large negative impact on
performance.  Switching from 1 to 4~vCPUs increases the boot time by
over 200ms.  About 25ms is spent starting each secondary CPU (in
\texttt{check\_tsc\_sync\_target}).  This can be avoided by setting
\texttt{tsc.reliable=1} but no one can tell me if this is safe.  But
most of the extra time just disappears between the cracks -- for
example, PCI probing just slows down, but for no readily apparent
reason.  It seems as if the overhead of spinlocks or RCU or whatever
hurts general performance.  Or perhaps there is some scheduling
problem on the host since it only has 4 physical CPUs.

When the kernel runs, it does some BIOS stuff, and there's a long
delay (about 80ms) before \texttt{start\_kernel} is entered.

Another unavoidable overhead is \texttt{ftrace} which must modify
every function in the kernel.  This takes 20ms.  You can't disable
ftrace at run time, the only option is to compile it out, but that
breaks so many useful features that we'd never persuade a distro
kernel to do that.

If your kernel has crypto functions, then it will spend 18ms testing
them at boot.  Herbert Xu accepted my patch to add a
\texttt{cryptomgr.notests} flag which bypasses this.

As we are presenting an emulated 16550A UART,
\texttt{serial\_8250\_init} runs, and this spends 25ms checking that
the UART is really a 16650A (does it work, does it have a FIFO?), and
(unsurprisingly) yes it is.  This is a totally useless waste of time,
but I have not managed to come up with a patch or even with an
approach for how to avoid this that is acceptable upstream.

But the main problem is none of the above.  It's simply the small
amount of time taken to run many many initcalls.  For a distro kernel
this can be around 690ms (with serial debugging enabled which
exaggerates the effect somewhat).  One way to avoid that would be to
compile some sort of custom kernel, and even though this approach is
not acceptable for Fedora I did explore this, trying both a cut down
distro kernel, and also a super-minimal kernel.

\begin{itemize}
\item The cut down distro kernel works by removing any subsystem that
  has a $>$~1ms initcall overhead.  These include:
  \begin{itemize}
  \item auditing
  \item big\_key
  \item ftrace
  \item hugetlbfs
  \item input\_leds
  \item joydev
  \item joysticks
  \item keyboards
  \item kprobes
  \item libata
  \item mice
  \item microcode
  \item netlabel
  \item profiling support
  \item quota
  \item rtc\_drv\_cmos
  \item sound card support
  \item tablets
  \item touchscreens
  \item USB
  \item zbud
  \item zswap
  \end{itemize}
  That reduces the time taken running initcalls before userspace by
  about 20\%.  There is some scope for reducing this a bit more by
  going even further down the ``long tail'' of subsystems.
\item For my second test I started with an absolutely minimal kernel
  config (\texttt{allnoconfig}), and built up the configuration until
  I got something that booted.  That reduces the time taken running
  initcalls before userspace by about 60\% (down to 288ms).
\end{itemize}

With a minimal kernel, we can get total boot times down to the
500-600ms range, but not any lower.

\section{udev}

udevd takes about 130ms to populate \texttt{/dev}.

The rules are monolithic, entwined together and resist modification,
and starting a new set of rules from scratch looks like it would be a
constant game of catch up.

\section{initrd}

We use a program called supermin
(\url{http://libguestfs.org/supermin.1.html}) to construct the initrd
which is responsible for loading enough kmods to mount the real root
filesystem and pivoting into it.

Because of PIO loading of the initrd in earlier versions of QEMU, it
was very important to construct as small an initrd as possible, and
supermin was not doing a very good job of that.  However once I
started to analyze the situation there were some easy wins (now all
upstream):

\begin{itemize}
\item We were adding all virtio kmods to the appliance plus any
  dependencies, with the starting set being constructed using the
  wildcard ``\texttt{virtio*.ko}''.  The wildcard pulls in
  \texttt{virtio-gpu.ko} which depends on \texttt{drm.ko} and both are
  quite large.  Since we are only interested in non-graphical VMs, I
  was able to blacklist \texttt{virtio-gpu.ko} and that reduced the
  total size of the initrd.
\item We use a small C init program to load the kmods and mount
  the root filesystem, and this must be statically linked so we
  don't have to include a separate libc in the initrd.  However
  glibc produces enormous static binaries (800KB+).  Switching to using
  dietlibc allows us to build the same program to a
  22KB binary, about $\frac{1}{40}$th of the size.
\item We initially used xz-compressed kmods.  These are smaller,
  reducing PIO loading time (but making not a lot of difference to
  DMA) but they are very slow to decompress.  Switching to using
  uncompressed kmods produced a small reduction in boot time, and
  simplified the init code.
\item Stripping kmods (with \texttt{strip~-g}) is very important for
  reducing the size of the initrd.
\end{itemize}

The resulting initrd is about 126KB for the minimal kernel, or 347K
for the standard Fedora kernel.

\section{libguestfs}

Finally there is libguestfs itself which glues everything together and
provides the initial \texttt{/init} script.  There were several
savings to be made:

\begin{itemize}
\item When we are not debugging, we were still reading the verbose
  kernel output over the slow UART, and then throwing it away.  The
  solution was to add the \texttt{quiet} option.  That reduced boot
  time by about 1,000ms, the single largest saving.
\item We used to run the \texttt{hwclock} command.  With kvmclock it
  turns out this is not necessary, and removing it saved 300ms.
\item We used to run \texttt{qemu~-help} and \texttt{qemu~-version}.
  Drew Jones pointed out the obvious: the help output contains the
  version number, so that reduces the number of times we need to run
  QEMU and suffer the glibc slow link loader overhead (and in the
  final version of libguestfs we also memoize QEMU output, reducing it
  further).
\item We used to run SGABIOS unconditionally, but it is only necessary
  to use it when debugging.  When we're not debugging we can omit it
  and save loading it at all.
\item Running \texttt{ldconfig} in the appliance to update the link
  loader cache took 100ms, but we found a way that we don't need to
  run it at all.
\end{itemize}

\section{Memory usage and DAX}

I was pleasantly surprised that Intel had implemented a virtual
NVDIMM, and ext4 + DAX is also working in modern kernels, and it was a
relatively trivial job to implement DAX.

However I'm not certain that the benefits are clear, nor that I'm
measuring things correctly.

Inside the guest you can run \texttt{free~-m} with and without DAX:

\begin{verbatim}
              total    used    free  shared buff/cache available
Without DAX:    485       3     451       1         30       465
With DAX:       485       3     469       1         12       467
\end{verbatim}

The MaxRSS of QEMU reduces by about 5~MB when DAX is enabled.

\section{Conclusions}

\begin{minipage}{\textwidth}
This graph is just for a bit of fun:

\includegraphics[width=0.8\textwidth]{progress}
\end{minipage}

There were a few false starts at the beginning of March (2016) where I
was exploring how we might benchmark QEMU.  But once I had written the
right tools to analyze the boot process, two quick wins brought the
time down from 3.5~seconds to 1.2~seconds in the space of a few days.
It's worth noting that the libguestfs appliance had been booting in
approx.~3-4~seconds for literally half a decade.

Getting the time under 600ms took a few weeks longer, and without some
breakthrough in the kernel or udev, I cannot see us getting the time
under 500ms.

Performance is everyone's job, but it sometimes feels like few people
care about a use case which is considered esoteric.  Yet this does
affect everyone:

\begin{itemize}
\item If we can use virtualization as an extra layer of security
  around operations, whether that is Docker, or Qubes, or
  libvirt-sandbox, or libguestfs, that benefits everyone.
\item The same concerns about boot speed are raised over and over
  again by the embedded community.  If your digital camera is slow to
  switch on, it might be running initcalls for subsystems that it will
  never use.  (Many references here:
  \url{http://elinux.org/Boot_Time})
\end{itemize}

Hopefully this paper will persuade developers to think twice before
adding an unnecessary delay loop, inserting a useless boot splash
screen, or creating another initcall.

\end{document}