Import old talks from 2010 and 2011 (Boston).

[libguestfs-talks.git] / 2011-boston / handout / 2011-summit-rjones-handout-libguestfs.tex

\documentclass[12pt]{article}
\usepackage{alltt,graphicx,url}

\title{Inspecting and modifying virtual machines\\
with Red Hat Enterprise Linux 6.1}

\author{Richard W.M. Jones\\
\small Senior Software Engineer\\[-0.8ex]
\small Red Hat\\
\small \texttt{rjones@redhat.com}\\
}

\date{Wednesday May 4th 2011}

\begin{document}
\maketitle

\section{Introduction}

libguestfs\footnote{\url{http://libguestfs.org/}} is a library,
scripting language and a set of tools that let you look into virtual
machines and make changes to them without needing to boot them up.
``Inspection'' is the process of getting a formal description of
what's in a virtual machine, how it is configured, what software is
installed, the contents of the filesystem and Windows Registry and so
on.  ``Modification'' here means making repeatable changes to these
guests, to their configuration files, filesystems and Registries, from
programs and scripts.

Big advances have happened in libguestfs since an early version was
added to RHEL 6.0, and most of those changes will appear in RHEL 6.1.
For a start, RHEL 6.1 libguestfs is 4 or 5 times faster, so if you
tried libguestfs in RHEL 6.0 and were disappointed with the
performance, then try RHEL 6.1.  Hundreds of individual features have
been added, and we're only going to be able to show a handful of the
most important features in the talk today.

libguestfs is now the basis for several important projects inside and
outside Red Hat, including the Boxgrinder cloud image builder, at
least one proprietary ISP/cloud VM deployment system, and
virt-p2v/virt-v2v which my colleague Matthew Booth is going to talk
about after me.

Red Hat thinks that managing these previously ``opaque'' disk images
and virtual machines is very important, and we want our customers to
have the best possible open source tools available.  The libguestfs
project was started over two years ago and has had one or two full
time developers working on it ever since then.  Here are some stats
from the project:

\vspace{6mm}

\begin{tabular}{ r p{10cm} }
\hline
 24 & command line tools \\
 171 & pages in the manual \\
 over 300 & API calls \\
 555 & automated tests run on each release \\
 2,885 & git commits (about $3\frac{1}{2}$ commits per day, including weekends and holidays) \\
 313,247 & lines of code \\
\hline
\end{tabular}

\section{What is libguestfs?}

\begin{center}
\includegraphics[width=0.8\textwidth]{libguestfs-overview.pdf}
\end{center}

\section{Timeline}

\begin{center}
\includegraphics[width=0.8\textwidth]{libguestfs-timeline.pdf}
\end{center}

\section{Introducing guestfish}

guestfish\footnote{\url{http://libguestfs.org/guestfish.1.html}} is
a shell you can use to open up and modify disk images.  You can just
open up any libvirt guest or disk image by doing:

\begin{samepage}
\begin{verbatim}
# guestfish --ro -i -d RHEL60

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Red Hat Enterprise Linux Server release 6.0 (Santiago)
/dev/vg_rhel6brewx64/lv_root mounted on /
/dev/vda1 mounted on /boot

><fs> ll /
total 138
dr-xr-xr-x.  26 root root  4096 Apr 11 09:49 .
drwxr-xr-x   24 root root  4096 Apr 11 17:13 ..
-rw-r--r--.   1 root root     0 Apr 11 09:49 .autofsck
drwx------.   3 root root  4096 Sep 17  2010 .dbus
dr-xr-xr-x.   2 root root  4096 Nov  6 15:21 bin
dr-xr-xr-x.   5 root root  1024 Sep 18  2010 boot
drwxr-xr-x.   2 root root  4096 Jul 14  2010 cgroup
drwxr-xr-x.   2 root root  4096 Sep 17  2010 dev
[etc]
\end{verbatim}
\end{samepage}

A note about those options:

\begin{itemize}
\item[\texttt{--ro}]
This means open the disk read-only: you don't want to
make any changes to it.  Opening a disk which is in use
(eg. used by a running VM) is \emph{unsafe} unless you
use this option.
\item[\texttt{-i}]
This means ``inspect'' the disk image and mount up the
filesystems as they would be mounted if the guest was
running.  You can leave out this option and instead
look for the filesystems yourself using the \texttt{list-filesystems}
command.
\item[\texttt{-d}]
This means open the named libvirt guest.  You can get a list
of libvirt guests by doing \texttt{virsh list --all}.
You can use the \texttt{-a} option to open a disk image
file or device directly.
\end{itemize}

There are hundreds of guestfish commands for reading and writing
files, listing directories, creating partitions, extending logical
volumes and so on.  You can also use guestfish from shell scripts if
you want to make repeatable scripted changes to guests.  A few useful
commands include:

\begin{itemize}
\item[\texttt{cat}] Display small text files.
\item[\texttt{edit}] Edit a file.
\item[\texttt{less}] Display longer files.
\item[\texttt{ll}] List (long) directory.
\item[\texttt{ls}] List directory.
\item[\texttt{mkdir}] Make a directory.
\item[\texttt{rm}] Remove a file.
\item[\texttt{touch}] Touch a file.
\item[\texttt{upload}] Upload a local file to the disk.
\item[\texttt{write}] Create a file with content.
\end{itemize}

The best place to start is the guestfish man page:

\begin{verbatim}
$ man guestfish
\end{verbatim}

\noindent
or by reading the webpage
\url{http://libguestfs.org/guestfish.1.html}

guestfish doesn't normally need root.  The only time you need to run
guestfish as root is if you need root in order to be able to access
the disk images themselves.  There are some better alternatives, such
as adding users to the ``disk'' group.

\section{Introducing virt-rescue}

virt-rescue\footnote{\url{http://libguestfs.org/virt-rescue.1.html}}
is a good way to rescue virtual machines that don't boot, or just
generally make ad hoc changes to virtual machines.  It's like a rescue
CD for virtual machines.

virt-rescue is a little different from guestfish in that you get an
ordinary shell and ordinary tools.  However unlike guestfish,
virt-rescue cannot be used from shell scripts, so it's not useful if
you want to make repeatable changes to lots of your guests.

You must not use virt-rescue on running VMs.

If you had a libvirt guest called ``Fedora'' then:

\begin{samepage}
\begin{verbatim}
# virt-rescue -d Fedora
[lots of boot messages]

Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / are the rescue appliance.
You have to mount the guest's partitions under /sysroot
before you can examine them.

><rescue> lvs
  LV      VG        Attr   LSize Origin Snap%  Move Log Copy%  Convert
  lv_root vg_f13x64 -wi-a- 7.56g                                      
  lv_swap vg_f13x64 -wi-a- 1.94g                                      
><rescue> mount /dev/vg_f13x64/lv_root /sysroot/
[  107.912813] EXT4-fs (dm-0): mounted filesystem with ordered data mode.
Opts: (null)
><rescue> ls -l /sysroot/etc/fstab 
-rw-r--r--. 1 root root 781 Sep 16  2010 /sysroot/etc/fstab
><rescue> vi /sysroot/etc/fstab 
\end{verbatim}
\end{samepage}

There is a lot more information about virt-rescue in the
man page:

\begin{verbatim}
$ man virt-rescue
\end{verbatim}

\noindent
or you can read the manual online
\url{http://libguestfs.org/virt-rescue.1.html}

\section{Introducing the other virt-tools}

In the following sections I will be demonstrating some of the other
virt tools that come with RHEL 6.1.  Here I'll provide a quick
overview of the tools available.

\begin{itemize}
\item[guestfish]
Interactive and scriptable shell.
\item[guestmount]
Mount filesystems from any guest or disk image on the host.
\item[virt-cat]
Display a file from a guest.
\item[virt-copy-in]
Copy files and directories into a guest.
\item[virt-copy-out]
Copy files and directories out of a guest.
\item[virt-df]
Display disk usage of a guest.
\item[virt-edit]
Edit a file in a guest.
\item[virt-filesystems]
Display the partitions, filesystems, logical volumes etc. in a guest.
\item[virt-inspector]
The old RHEL 6.0 virt-inspector program.  Use virt-inspector2 instead.
\item[virt-inspector2]
Inspect a guest and produce a report detailing the operating system,
version, applications installed and more.
\item[virt-ls]
List a directory in a guest.
\item[virt-make-fs]
Make a new filesystem.
\item[virt-rescue]
Rescue mode for guests.
\item[virt-resize]
Resize a guest.
\item[virt-tar-in]
Copy files from a tarball into a guest.
\item[virt-tar-out]
Copy files out of a guest into a tarball.
\item[virt-win-reg]
Display and edit the Windows Registry in a guest.
\end{itemize}

To get more information about any command, read the manual page.  Type (for
example):

\begin{verbatim}
$ man virt-cat
\end{verbatim}

\noindent
or see the upstream website: \url{http://libguestfs.org/}

\section{Exercise: charting disk usage with virt-df}

The virt-df
utility\footnote{\url{http://libguestfs.org/virt-df.1.html}} displays
disk usage for virtual machines.  Normally the output looks like the
ordinary ``df'' command:

\begin{samepage}
\begin{verbatim}
# virt-df -h
Filesystem                                Size       Used  Available  Use%
cooking:/dev/sda                          3.0G       1.5G       1.3G   52%
cooking:/dev/sdb                          128M        95M        26M   75%
database:/dev/sda                         3.0G       733M       2.1G   25%
database:/dev/sdb                         128M        95M        26M   75%
database:/dev/sdc                          49G        25G        22G   51%
\end{verbatim}
\end{samepage}

However you can also get virt-df to produce comma-separated values
(CSV) output which is useful for monitoring and tracking disk usage.
CSV can be imported directly into many databases and spreadsheet
programs.

On my production server I capture virt-df CSV output every day using a
simple cron job \texttt{/etc/cron.daily/local-virt-df}:

\begin{samepage}
\begin{verbatim}
#!/bin/bash -
date=$(date +%F)
virt-df --csv > /var/local/virt-df.$date
\end{verbatim}
\end{samepage}

\begin{figure}[p]
\begin{center}
\includegraphics[width=0.6\textwidth]{df-chart.png}
\end{center}
\caption{Disk usage of a virtual machine over the 5 months starting
  with installation.  Notice the spikes when the VM was first
  installed, followed by a broad trend of very gradually increasing
  disk usage.}
\label{dfchart}
\end{figure}

I then import these files into a spreadsheet which allows me to chart
disk usage and look for trends.  Figure~\ref{dfchart} on
page~\pageref{dfchart} charts a virtual machine over a five month
period.

\section{Exercise: using guestfish -N}

In this exercise we will use the guestfish ``-N'' option to create a
new disk image from scratch containing some files and directories.
For the content I'm going to use a source tarball of
libguestfs\footnote{Source code for libguestfs is available from
  \url{http://libguestfs.org/download/} or for Red Hat subscribers
  from RHN.}.

To make this exercise more exciting I'm going to specify that I want
my files stored in an LVM logical volume inside the disk image, and I
want to format my filesystem using the smart new
btrfs\footnote{\url{https://secure.wikimedia.org/wikipedia/en/wiki/Btrfs}}
filesystem.  The files from the tarball are about 5 MB in size, so I'm
going to choose a disk image size which is easily large enough to
store them with plenty of space: 500 MB!  It turns out that the
minimum size for a btrfs filesystem is 256 MB, and both LVM and btrfs
impose a large overhead.

In effect my disk image will be wrapped up in several layers as in
this diagram:

\begin{center}
\includegraphics[width=0.8\textwidth]{nested-filesystem.pdf}
\end{center}

The guestfish ``-N'' option below creates the complex nested
filesystem structure\footnote{For more information about use of the
  ``-N'' option, type: \texttt{guestfish -N help}}.  Notice that you do
not need to run this command as root -- creating disk images is
something that everyone can do.

\begin{samepage}
\begin{alltt}
\$ guestfish -N lvfs:/dev/VG/LV:btrfs:500M
><fs> list-filesystems
/dev/VG/LV: btrfs
><fs> mount-options "" /dev/VG/LV / \hfill {\normalfont\tiny Mount the filesystem so we can write to it}
><fs> df-h \hfill {\normalfont\tiny Notice that 96 MB has been lost!}
Filesystem            Size  Used Avail Use\% Mounted on
/dev/mapper/VG-LV     496M   56K  404M   1\% /sysroot
><fs> tgz-in libguestfs-1.9.18.tar.gz / \hfill {\normalfont\tiny Unpack the tarball into the new filesystem}
><fs> ll /
total 8
dr-xr-xr-x  1 root root   34 Apr 12 14:08 .
drwxr-xr-x 24  500  500 4096 Apr 12 14:08 ..
drwxrwxr-x  1 root root 1076 Apr  9 22:25 libguestfs-1.9.18
><fs> exit

\$ file test1.img 
test1.img: x86 boot sector; partition 1: ID=0x83, starthead 1,
startsector 64, 1023873 sectors, code offset 0xb8
\end{alltt}
\end{samepage}

The output disk image is in \texttt{test1.img}.  How do you prove
that it contains a filesystem?  One way is to open it again with
guestfish:

\begin{verbatim}
$ guestfish -a test1.img -m /dev/VG/LV
\end{verbatim}

Another way is to take this disk image and attach it to a virtual
machine.

\section{Exercise: find vulnerable versions of Firefox}

In this exercise we will use virt-inspector\footnote{This example uses
  the Fedora virt-inspector program.  In RHEL 6.1 this program is
  called \texttt{virt-inspector2} so you need to change any references
  to ``virt-inspector'' to ``virt-inspector2''.  RHEL 6.1 ships with a
  known bug: it is not able to list 32 bit applications installed in a
  64 bit Windows guest (using the WOW64 emulator).  A fix for this bug
  will be included in RHEL 6.2.  (RHBZ\#692545)} to find out if any
vulnerable versions of
Firefox\footnote{\url{https://www.mozilla.org/security/known-vulnerabilities/}}
are installed in Windows guests.  At the time of writing, any version
of Firefox $<$ 3.6.16 was vulnerable, so we'd like to scan our Windows
guests to check this.

First run virt-inspector and have a look at the output:

\begin{samepage}
\begin{alltt}
# virt-inspector -d WindowsGuest
<operatingsystems>
  <operatingsystem>
    <name>windows</name> \hfill {\normalfont\tiny it's a Windows guest}
    <arch>i386</arch> \hfill {\normalfont\tiny it's 32 bit}
    <product\_name>Windows 7 Enterprise</product\_name>
    <major\_version>6</major\_version> \hfill {\normalfont\tiny ``6.1'' = Windows 7 -- blame Microsoft!}
    <minor\_version>1</minor\_version>
    ...
    <applications> \hfill {\normalfont\tiny the list of applications starts here}
      <application>
        <name>Mozilla Firefox (3.6.12)</name>
        <display\_name>Mozilla Firefox (3.6.12)</display\_name>
        <version>3.6.12 (en-GB)</version>
      ...
      </application>
    </applications>
  </operatingsystem>
</operatingsystems>
\end{alltt}
\end{samepage}

One way to extract and process XML documents is to use W3C standard
XPath expressions.  In this example I will use a short Python program
with the libxml2 library to find vulnerable versions of Firefox:

\begin{samepage}
\begin{alltt}
#!/usr/bin/python
import libxml2, re, sys
from distutils import version
\hfill {\normalfont\tiny Read the XML piped from standard input}
doc = libxml2.readFd (sys.stdin.fileno(), None, None, 0)
\hfill {\normalfont\tiny Use XPath to find all \textless{}application\textgreater nodes}
ctx = doc.xpathNewContext()
res = ctx.xpathEval ("//application")
for node in res:
\hfill {\normalfont\tiny Use XPath to find the \textless{}name\textgreater and \textless{}version\textgreater within current \textless{}application\textgreater node}
    ctx.setContextNode(node)
    name = ctx.xpathEval ("./name//text()")[0]
    ver = ctx.xpathEval ("./version//text()")[0]
\hfill {\normalfont\tiny Python StrictVersion lets me compare version numbers}
    ver = version.StrictVersion (str(ver).split(' ')[0])
    if re.search ("Mozilla Firefox", str(name)) and \textbackslash
            ver < version.StrictVersion ("3.6.16"):
        print "Vulnerable version of Firefox found (\%s)!" \% ver
\end{alltt}
\end{samepage}

Putting this together gives:

\begin{verbatim}
# virt-inspector -d WindowsGuest | ./vulnerable.py
Vulnerable version of Firefox found (3.6.12)!
\end{verbatim}

\end{document}