1 \documentclass[12pt]{article}
2 \usepackage{alltt,graphicx,url}
4 \title{Inspecting and modifying virtual machines\\
5 with Red Hat Enterprise Linux 6.1}
7 \author{Richard W.M. Jones\\
8 \small Senior Software Engineer\\[-0.8ex]
10 \small \texttt{rjones@redhat.com}\\
13 \date{Wednesday May 4th 2011}
18 \section{Introduction}
20 libguestfs\footnote{\url{http://libguestfs.org/}} is a library,
21 scripting language and a set of tools that let you look into virtual
22 machines and make changes to them without needing to boot them up.
23 ``Inspection'' is the process of getting a formal description of
24 what's in a virtual machine, how it is configured, what software is
25 installed, the contents of the filesystem and Windows Registry and so
26 on. ``Modification'' here means making repeatable changes to these
27 guests, to their configuration files, filesystems and Registries, from
30 Big advances have happened in libguestfs since an early version was
31 added to RHEL 6.0, and most of those changes will appear in RHEL 6.1.
32 For a start, RHEL 6.1 libguestfs is 4 or 5 times faster, so if you
33 tried libguestfs in RHEL 6.0 and were disappointed with the
34 performance, then try RHEL 6.1. Hundreds of individual features have
35 been added, and we're only going to be able to show a handful of the
36 most important features in the talk today.
38 libguestfs is now the basis for several important projects inside and
39 outside Red Hat, including the Boxgrinder cloud image builder, at
40 least one proprietary ISP/cloud VM deployment system, and
41 virt-p2v/virt-v2v which my colleague Matthew Booth is going to talk
44 Red Hat thinks that managing these previously ``opaque'' disk images
45 and virtual machines is very important, and we want our customers to
46 have the best possible open source tools available. The libguestfs
47 project was started over two years ago and has had one or two full
48 time developers working on it ever since then. Here are some stats
53 \begin{tabular}{ r p{10cm} }
55 24 & command line tools \\
56 171 & pages in the manual \\
57 over 300 & API calls \\
58 555 & automated tests run on each release \\
59 2,885 & git commits (about $3\frac{1}{2}$ commits per day, including weekends and holidays) \\
60 313,247 & lines of code \\
64 \section{What is libguestfs?}
67 \includegraphics[width=0.8\textwidth]{libguestfs-overview.pdf}
73 \includegraphics[width=0.8\textwidth]{libguestfs-timeline.pdf}
76 \section{Introducing guestfish}
78 guestfish\footnote{\url{http://libguestfs.org/guestfish.1.html}} is
79 a shell you can use to open up and modify disk images. You can just
80 open up any libvirt guest or disk image by doing:
84 # guestfish --ro -i -d RHEL60
86 Welcome to guestfish, the libguestfs filesystem interactive shell for
87 editing virtual machine filesystems.
89 Type: 'help' for help on commands
90 'man' to read the manual
91 'quit' to quit the shell
93 Operating system: Red Hat Enterprise Linux Server release 6.0 (Santiago)
94 /dev/vg_rhel6brewx64/lv_root mounted on /
95 /dev/vda1 mounted on /boot
99 dr-xr-xr-x. 26 root root 4096 Apr 11 09:49 .
100 drwxr-xr-x 24 root root 4096 Apr 11 17:13 ..
101 -rw-r--r--. 1 root root 0 Apr 11 09:49 .autofsck
102 drwx------. 3 root root 4096 Sep 17 2010 .dbus
103 dr-xr-xr-x. 2 root root 4096 Nov 6 15:21 bin
104 dr-xr-xr-x. 5 root root 1024 Sep 18 2010 boot
105 drwxr-xr-x. 2 root root 4096 Jul 14 2010 cgroup
106 drwxr-xr-x. 2 root root 4096 Sep 17 2010 dev
111 A note about those options:
115 This means open the disk read-only: you don't want to
116 make any changes to it. Opening a disk which is in use
117 (eg. used by a running VM) is \emph{unsafe} unless you
120 This means ``inspect'' the disk image and mount up the
121 filesystems as they would be mounted if the guest was
122 running. You can leave out this option and instead
123 look for the filesystems yourself using the \texttt{list-filesystems}
126 This means open the named libvirt guest. You can get a list
127 of libvirt guests by doing \texttt{virsh list --all}.
128 You can use the \texttt{-a} option to open a disk image
129 file or device directly.
132 There are hundreds of guestfish commands for reading and writing
133 files, listing directories, creating partitions, extending logical
134 volumes and so on. You can also use guestfish from shell scripts if
135 you want to make repeatable scripted changes to guests. A few useful
139 \item[\texttt{cat}] Display small text files.
140 \item[\texttt{edit}] Edit a file.
141 \item[\texttt{less}] Display longer files.
142 \item[\texttt{ll}] List (long) directory.
143 \item[\texttt{ls}] List directory.
144 \item[\texttt{mkdir}] Make a directory.
145 \item[\texttt{rm}] Remove a file.
146 \item[\texttt{touch}] Touch a file.
147 \item[\texttt{upload}] Upload a local file to the disk.
148 \item[\texttt{write}] Create a file with content.
151 The best place to start is the guestfish man page:
158 or by reading the webpage
159 \url{http://libguestfs.org/guestfish.1.html}
161 guestfish doesn't normally need root. The only time you need to run
162 guestfish as root is if you need root in order to be able to access
163 the disk images themselves. There are some better alternatives, such
164 as adding users to the ``disk'' group.
166 \section{Introducing virt-rescue}
168 virt-rescue\footnote{\url{http://libguestfs.org/virt-rescue.1.html}}
169 is a good way to rescue virtual machines that don't boot, or just
170 generally make ad hoc changes to virtual machines. It's like a rescue
171 CD for virtual machines.
173 virt-rescue is a little different from guestfish in that you get an
174 ordinary shell and ordinary tools. However unlike guestfish,
175 virt-rescue cannot be used from shell scripts, so it's not useful if
176 you want to make repeatable changes to lots of your guests.
178 You must not use virt-rescue on running VMs.
180 If you had a libvirt guest called ``Fedora'' then:
184 # virt-rescue -d Fedora
185 [lots of boot messages]
187 Welcome to virt-rescue, the libguestfs rescue shell.
189 Note: The contents of / are the rescue appliance.
190 You have to mount the guest's partitions under /sysroot
191 before you can examine them.
194 LV VG Attr LSize Origin Snap% Move Log Copy% Convert
195 lv_root vg_f13x64 -wi-a- 7.56g
196 lv_swap vg_f13x64 -wi-a- 1.94g
197 ><rescue> mount /dev/vg_f13x64/lv_root /sysroot/
198 [ 107.912813] EXT4-fs (dm-0): mounted filesystem with ordered data mode.
200 ><rescue> ls -l /sysroot/etc/fstab
201 -rw-r--r--. 1 root root 781 Sep 16 2010 /sysroot/etc/fstab
202 ><rescue> vi /sysroot/etc/fstab
206 There is a lot more information about virt-rescue in the
214 or you can read the manual online
215 \url{http://libguestfs.org/virt-rescue.1.html}
217 \section{Introducing the other virt-tools}
219 In the following sections I will be demonstrating some of the other
220 virt tools that come with RHEL 6.1. Here I'll provide a quick
221 overview of the tools available.
225 Interactive and scriptable shell.
227 Mount filesystems from any guest or disk image on the host.
229 Display a file from a guest.
231 Copy files and directories into a guest.
233 Copy files and directories out of a guest.
235 Display disk usage of a guest.
237 Edit a file in a guest.
238 \item[virt-filesystems]
239 Display the partitions, filesystems, logical volumes etc. in a guest.
240 \item[virt-inspector]
241 The old RHEL 6.0 virt-inspector program. Use virt-inspector2 instead.
242 \item[virt-inspector2]
243 Inspect a guest and produce a report detailing the operating system,
244 version, applications installed and more.
246 List a directory in a guest.
248 Make a new filesystem.
250 Rescue mode for guests.
254 Copy files from a tarball into a guest.
256 Copy files out of a guest into a tarball.
258 Display and edit the Windows Registry in a guest.
261 To get more information about any command, read the manual page. Type (for
269 or see the upstream website: \url{http://libguestfs.org/}
271 \section{Exercise: charting disk usage with virt-df}
274 utility\footnote{\url{http://libguestfs.org/virt-df.1.html}} displays
275 disk usage for virtual machines. Normally the output looks like the
276 ordinary ``df'' command:
281 Filesystem Size Used Available Use%
282 cooking:/dev/sda 3.0G 1.5G 1.3G 52%
283 cooking:/dev/sdb 128M 95M 26M 75%
284 database:/dev/sda 3.0G 733M 2.1G 25%
285 database:/dev/sdb 128M 95M 26M 75%
286 database:/dev/sdc 49G 25G 22G 51%
290 However you can also get virt-df to produce comma-separated values
291 (CSV) output which is useful for monitoring and tracking disk usage.
292 CSV can be imported directly into many databases and spreadsheet
295 On my production server I capture virt-df CSV output every day using a
296 simple cron job \texttt{/etc/cron.daily/local-virt-df}:
302 virt-df --csv > /var/local/virt-df.$date
308 \includegraphics[width=0.6\textwidth]{df-chart.png}
310 \caption{Disk usage of a virtual machine over the 5 months starting
311 with installation. Notice the spikes when the VM was first
312 installed, followed by a broad trend of very gradually increasing
317 I then import these files into a spreadsheet which allows me to chart
318 disk usage and look for trends. Figure~\ref{dfchart} on
319 page~\pageref{dfchart} charts a virtual machine over a five month
322 \section{Exercise: using guestfish -N}
324 In this exercise we will use the guestfish ``-N'' option to create a
325 new disk image from scratch containing some files and directories.
326 For the content I'm going to use a source tarball of
327 libguestfs\footnote{Source code for libguestfs is available from
328 \url{http://libguestfs.org/download/} or for Red Hat subscribers
331 To make this exercise more exciting I'm going to specify that I want
332 my files stored in an LVM logical volume inside the disk image, and I
333 want to format my filesystem using the smart new
334 btrfs\footnote{\url{https://secure.wikimedia.org/wikipedia/en/wiki/Btrfs}}
335 filesystem. The files from the tarball are about 5 MB in size, so I'm
336 going to choose a disk image size which is easily large enough to
337 store them with plenty of space: 500 MB! It turns out that the
338 minimum size for a btrfs filesystem is 256 MB, and both LVM and btrfs
339 impose a large overhead.
341 In effect my disk image will be wrapped up in several layers as in
345 \includegraphics[width=0.8\textwidth]{nested-filesystem.pdf}
348 The guestfish ``-N'' option below creates the complex nested
349 filesystem structure\footnote{For more information about use of the
350 ``-N'' option, type: \texttt{guestfish -N help}}. Notice that you do
351 not need to run this command as root -- creating disk images is
352 something that everyone can do.
356 \$ guestfish -N lvfs:/dev/VG/LV:btrfs:500M
357 ><fs> list-filesystems
359 ><fs> mount-options "" /dev/VG/LV / \hfill {\normalfont\tiny Mount the filesystem so we can write to it}
360 ><fs> df-h \hfill {\normalfont\tiny Notice that 96 MB has been lost!}
361 Filesystem Size Used Avail Use\% Mounted on
362 /dev/mapper/VG-LV 496M 56K 404M 1\% /sysroot
363 ><fs> tgz-in libguestfs-1.9.18.tar.gz / \hfill {\normalfont\tiny Unpack the tarball into the new filesystem}
366 dr-xr-xr-x 1 root root 34 Apr 12 14:08 .
367 drwxr-xr-x 24 500 500 4096 Apr 12 14:08 ..
368 drwxrwxr-x 1 root root 1076 Apr 9 22:25 libguestfs-1.9.18
372 test1.img: x86 boot sector; partition 1: ID=0x83, starthead 1,
373 startsector 64, 1023873 sectors, code offset 0xb8
377 The output disk image is in \texttt{test1.img}. How do you prove
378 that it contains a filesystem? One way is to open it again with
382 $ guestfish -a test1.img -m /dev/VG/LV
385 Another way is to take this disk image and attach it to a virtual
388 \section{Exercise: find vulnerable versions of Firefox}
390 In this exercise we will use virt-inspector\footnote{This example uses
391 the Fedora virt-inspector program. In RHEL 6.1 this program is
392 called \texttt{virt-inspector2} so you need to change any references
393 to ``virt-inspector'' to ``virt-inspector2''. RHEL 6.1 ships with a
394 known bug: it is not able to list 32 bit applications installed in a
395 64 bit Windows guest (using the WOW64 emulator). A fix for this bug
396 will be included in RHEL 6.2. (RHBZ\#692545)} to find out if any
397 vulnerable versions of
398 Firefox\footnote{\url{https://www.mozilla.org/security/known-vulnerabilities/}}
399 are installed in Windows guests. At the time of writing, any version
400 of Firefox $<$ 3.6.16 was vulnerable, so we'd like to scan our Windows
401 guests to check this.
403 First run virt-inspector and have a look at the output:
407 # virt-inspector -d WindowsGuest
410 <name>windows</name> \hfill {\normalfont\tiny it's a Windows guest}
411 <arch>i386</arch> \hfill {\normalfont\tiny it's 32 bit}
412 <product\_name>Windows 7 Enterprise</product\_name>
413 <major\_version>6</major\_version> \hfill {\normalfont\tiny ``6.1'' = Windows 7 -- blame Microsoft!}
414 <minor\_version>1</minor\_version>
416 <applications> \hfill {\normalfont\tiny the list of applications starts here}
418 <name>Mozilla Firefox (3.6.12)</name>
419 <display\_name>Mozilla Firefox (3.6.12)</display\_name>
420 <version>3.6.12 (en-GB)</version>
429 One way to extract and process XML documents is to use W3C standard
430 XPath expressions. In this example I will use a short Python program
431 with the libxml2 library to find vulnerable versions of Firefox:
436 import libxml2, re, sys
437 from distutils import version
438 \hfill {\normalfont\tiny Read the XML piped from standard input}
439 doc = libxml2.readFd (sys.stdin.fileno(), None, None, 0)
440 \hfill {\normalfont\tiny Use XPath to find all \textless{}application\textgreater nodes}
441 ctx = doc.xpathNewContext()
442 res = ctx.xpathEval ("//application")
444 \hfill {\normalfont\tiny Use XPath to find the \textless{}name\textgreater and \textless{}version\textgreater within current \textless{}application\textgreater node}
445 ctx.setContextNode(node)
446 name = ctx.xpathEval ("./name//text()")[0]
447 ver = ctx.xpathEval ("./version//text()")[0]
448 \hfill {\normalfont\tiny Python StrictVersion lets me compare version numbers}
449 ver = version.StrictVersion (str(ver).split(' ')[0])
450 if re.search ("Mozilla Firefox", str(name)) and \textbackslash
451 ver < version.StrictVersion ("3.6.16"):
452 print "Vulnerable version of Firefox found (\%s)!" \% ver
456 Putting this together gives:
459 # virt-inspector -d WindowsGuest | ./vulnerable.py
460 Vulnerable version of Firefox found (3.6.12)!