1
00:00:06,000 --> 00:00:08,500
Thank you, good afternoon.
2
00:00:12,000 --> 00:00:16,100
My name is Richard Jones and I work at Red Hat
and I'm going to be talking today about libguestfs
3
00:00:16,200 --> 00:00:19,700
which is a C library and a suite of tools
4
00:00:19,800 --> 00:00:23,300
including virt-builder, virt-sysprep, virt-sparsify
and much more
5
00:00:24,000 --> 00:00:25,900
but before I start ...
6
00:00:27,000 --> 00:00:32,200
I just wanted to give you an idea of
what exactly I'm talking about today
7
00:00:32,300 --> 00:00:35,100
So I'm going to build a Fedora 20
complete virtual machine
8
00:00:35,200 --> 00:00:38,700
It'll take about 20 seconds or so
9
00:00:38,800 --> 00:00:43,300
This is starting from a template
and cloning it
10
00:00:43,400 --> 00:00:45,200
giving it a random seed and
root password
11
00:00:45,300 --> 00:00:49,600
and we can use tools like
guestfish to open this disk image
12
00:00:49,600 --> 00:00:52,100
and make automated changes
inside it
13
00:00:55,200 --> 00:00:58,700
For example I can ..
if you wanted to
14
00:00:58,800 --> 00:01:03,300
have a virtual machine that always
started up with a particular message
15
00:01:03,400 --> 00:01:07,900
you could use something like guestfish
or our C API to make these
16
00:01:08,000 --> 00:01:10,500
automated changes to it.
17
00:01:10,600 --> 00:01:12,100
So that's what libguestfs
basically does.
18
00:01:13,800 --> 00:01:16,800
I'm going to start running
a little script
19
00:01:16,800 --> 00:01:20,000
looks a bit complicated but
all it is, is a while-loop
20
00:01:20,000 --> 00:01:21,100
which goes round and round
21
00:01:21,100 --> 00:01:23,049
and it's going to keep running
virt-builder while I'm talking
22
00:01:22,985 --> 00:01:26,887
Let's set that going.
23
00:01:26,887 --> 00:01:31,609
And we'll come back to that
in just 10 minutes.
24
00:01:33,393 --> 00:01:34,863
What does virt-builder do?
25
00:01:34,863 --> 00:01:39,772
Virt-builder can build you
a CentOS 6 guest
26
00:01:39,772 --> 00:01:42,936
It can change the format
of the output disk image
27
00:01:42,936 --> 00:01:44,804
or the size using virt-resize
28
00:01:44,804 --> 00:01:48,316
using libguesfs to go in there
and resize the partitions
29
00:01:48,316 --> 00:01:50,121
and the logical volumes
correctly
30
00:01:50,121 --> 00:01:52,961
so the guest is already resized
when it starts running.
31
00:01:52,961 --> 00:01:57,140
It can change the
hostname, timezone
32
00:01:57,140 --> 00:01:59,036
You can install packages.
33
00:01:59,036 --> 00:02:03,244
You can edit configuration files
in many different ways.
34
00:02:03,244 --> 00:02:06,940
You can run scripts at various
points during the build
35
00:02:06,940 --> 00:02:13,375
including when it's building
or if you prefer, in this case,
36
00:02:13,375 --> 00:02:15,305
when it first boots.
37
00:02:17,879 --> 00:02:20,991
Virt-builder does not install
guests from scratch
38
00:02:20,991 --> 00:02:22,420
and that's really what I'm
going to talk about today
39
00:02:22,420 --> 00:02:24,519
It doesn't run Anaconda
or debian-installer
40
00:02:24,519 --> 00:02:26,279
because that would take
far too long.
41
00:02:26,471 --> 00:02:30,503
Instead it clones
clean, cryptographically signed
42
00:02:30,503 --> 00:02:34,663
compressed templates,
and it customizes them for you.
43
00:02:34,663 --> 00:02:41,631
Libguestfs can safely and
securely modify disk images
44
00:02:41,631 --> 00:02:42,996
without requiring root.
45
00:02:42,996 --> 00:02:45,865
Virt-builder comes with a
handful of templates
46
00:02:45,865 --> 00:02:48,916
but what I'm really going to
talk about today is
47
00:02:48,916 --> 00:02:51,879
how you can prepare your own.
48
00:03:02,419 --> 00:03:06,183
Over the years we have
come up with a good way
49
00:03:06,183 --> 00:03:09,217
to prepare and distribute
small templates.
50
00:03:09,217 --> 00:03:13,468
This is the method that
we use now.
51
00:03:13,468 --> 00:03:17,745
Virt-install runs the
regular O.S. installer
52
00:03:17,745 --> 00:03:20,244
so it does run Anaconda
or debian-installer
53
00:03:20,244 --> 00:03:22,153
or Windows installers
and so on.
54
00:03:22,153 --> 00:03:24,663
You can completely automate
it using preseed
55
00:03:24,663 --> 00:03:25,997
or kickstart.
56
00:03:25,997 --> 00:03:27,875
You could also use Oz here instead.
57
00:03:29,004 --> 00:03:35,487
Virt-sysprep unconfigures the guest.
58
00:03:35,487 --> 00:03:38,137
Using libguestfs it's going to
go in there
59
00:03:38,137 --> 00:03:41,225
It can remove things like
ssh keys
60
00:03:41,225 --> 00:03:43,889
which you don't want
to have when you clone a guest
61
00:03:43,889 --> 00:03:45,932
you don't want them to all
have the same ssh key
62
00:03:45,932 --> 00:03:49,132
It can do things like
removing log files
63
00:03:49,132 --> 00:03:55,009
You don't want to give
away how your guest was created
64
00:03:55,009 --> 00:03:56,897
so it's a good idea to
scrub things like log files
65
00:03:56,897 --> 00:03:59,152
It can get rid of
persistent network configuration
66
00:03:59,152 --> 00:04:02,295
There are about 20 different
operations that virt-sysprep can do
67
00:04:03,183 --> 00:04:05,956
Virt-sparsify is the next one
68
00:04:05,956 --> 00:04:08,300
When you've installed a guest
69
00:04:08,300 --> 00:04:13,468
virt-sparsify can go in there and
make that guest disk image
70
00:04:13,468 --> 00:04:16,991
"sparse" or "thin-provisioned"
71
00:04:16,991 --> 00:04:20,279
Using libguestfs again it goes
and looks deep inside
72
00:04:20,279 --> 00:04:21,247
the filesystem.
73
00:04:21,311 --> 00:04:24,603
It can find unused space in
deleted files
74
00:04:24,603 --> 00:04:27,692
inside unused swap
75
00:04:27,692 --> 00:04:31,016
in deleted logical volumes
or partitions
76
00:04:31,016 --> 00:04:32,356
that are not used.
77
00:04:32,356 --> 00:04:36,327
It can find blocks which
although not zeroes
78
00:04:36,327 --> 00:04:38,728
in the disk image
and actually not used.
79
00:04:38,728 --> 00:04:42,860
It can give that space back
to the operating system
80
00:04:42,860 --> 00:04:44,657
by making the
disk image sparse.
81
00:04:44,657 --> 00:04:49,535
As you can see it is very
simple to use.
82
00:04:50,409 --> 00:04:52,677
The final step is xz.
83
00:04:52,677 --> 00:04:55,873
We really like xz
for a few reasons
84
00:04:55,873 --> 00:05:00,885
xz is nearly best
in class compression
85
00:05:00,885 --> 00:05:03,303
It's not perhaps the very
best compression
86
00:05:03,303 --> 00:05:06,128
but it's way better
than gzip or bzip2
87
00:05:07,188 --> 00:05:11,612
Importantly xz preserves
sparseness
88
00:05:11,612 --> 00:05:13,837
If you've gone to the trouble
of running virt-sparsify
89
00:05:13,837 --> 00:05:16,249
you don't want to decompress
your cloned disk image
90
00:05:16,249 --> 00:05:18,751
and find that your
sparseness is gone
91
00:05:18,751 --> 00:05:20,547
it's all fully
allocated again
92
00:05:20,547 --> 00:05:23,836
The other thing about xz
is it has a brilliant API
93
00:05:23,836 --> 00:05:25,697
and a brilliant file format
94
00:05:25,697 --> 00:05:27,035
It's all well-documented
95
00:05:27,035 --> 00:05:29,313
and it allows us to do
all sorts of clever stuff
96
00:05:29,313 --> 00:05:31,656
We have a sister project
called nbdkit
97
00:05:31,656 --> 00:05:34,215
It's an NBD (Network
Block Device) server
98
00:05:34,215 --> 00:05:41,088
It allows you to serve
an xz-compressed
99
00:05:41,088 --> 00:05:43,609
disk image, presented
as uncompressed
100
00:05:43,609 --> 00:05:46,847
to someone consuming NBD
101
00:05:46,847 --> 00:05:49,859
but it doesn't do that by
uncompressing the whole xz
102
00:05:49,859 --> 00:05:51,899
image before it starts.
103
00:05:51,899 --> 00:05:55,659
It just uncompresses
the bits you're reading at the time.
104
00:05:55,659 --> 00:05:59,376
It can do this because the xz API
allows random access
105
00:05:59,376 --> 00:06:01,207
to the compressed file.
106
00:06:01,207 --> 00:06:03,376
The other thing we've done with xz:
107
00:06:03,376 --> 00:06:07,595
virt-builder includes a
custom uncompressor
108
00:06:07,595 --> 00:06:10,977
which is way faster than
unxz or xzcat
109
00:06:10,977 --> 00:06:14,173
On my laptop it's
four times faster
110
00:06:14,173 --> 00:06:16,143
When you see virt-builder
running
111
00:06:16,143 --> 00:06:18,140
it has an uncompress step
that takes 10 seconds
112
00:06:18,140 --> 00:06:20,324
that normally takes
40 seconds
113
00:06:20,324 --> 00:06:22,151
if you just used xzcat
114
00:06:22,151 --> 00:06:25,111
So what's the
result of all this?
115
00:06:25,111 --> 00:06:30,653
I'm going to take just one
row of this, Fedora 20
116
00:06:30,653 --> 00:06:33,516
These are all in megabytes
117
00:06:33,516 --> 00:06:36,380
The Fedora 20 container
is 6 gigabytes
118
00:06:36,380 --> 00:06:37,591
that doesn't really mean
much
119
00:06:37,591 --> 00:06:40,637
when you're installing a guest
you just plug in a number
120
00:06:40,637 --> 00:06:42,137
in this case it was
6 gigabytes
121
00:06:42,137 --> 00:06:46,201
Most of that is empty space,
but 826 megabytes
122
00:06:46,201 --> 00:06:49,372
is used by the Fedora
core packages
123
00:06:49,372 --> 00:06:53,900
After sparsification
and xz compression
124
00:06:53,900 --> 00:06:56,684
it's down to 174
megabytes.
125
00:06:57,008 --> 00:07:02,265
This isn't the most
minimal disk image ever.
126
00:07:02,265 --> 00:07:05,217
It's possible to
go much smaller.
127
00:07:05,217 --> 00:07:09,241
However, we have done this
without doing any nasty tricks
128
00:07:09,241 --> 00:07:11,408
like deleting files
from the guest.
129
00:07:11,408 --> 00:07:12,871
We don't delete the
documentation.
130
00:07:12,871 --> 00:07:15,073
We don't remove the
package manager.
131
00:07:15,073 --> 00:07:18,100
The only things we change
are configuration files
132
00:07:18,100 --> 00:07:19,369
you're allowed to
change those
133
00:07:19,369 --> 00:07:22,020
and we've scrubbed
log files.
134
00:07:22,020 --> 00:07:24,659
So we've got a
trustworthy
135
00:07:24,659 --> 00:07:27,661
faithful representation
of that guest.
136
00:07:27,661 --> 00:07:29,777
We can distribute this
over broadband.
137
00:07:29,777 --> 00:07:31,863
Virt-builder caches the
images so you only have
138
00:07:31,863 --> 00:07:32,844
to download it once.
139
00:07:32,844 --> 00:07:38,000
You can create 1000s of F20
guests from a single download.
140
00:07:40,359 --> 00:07:46,903
I set virt-builder running
earlier on
141
00:07:46,903 --> 00:07:50,440
It's created 24 guests
142
00:07:50,440 --> 00:07:53,351
and it's working on
guest number 25 at the moment.
143
00:07:53,351 --> 00:07:59,391
You can see that uncompression
step took ~9 seconds.
144
00:07:59,391 --> 00:08:06,627
Virt-builder spits out images
every 15 .. 20 .. 25 seconds
145
00:08:06,627 --> 00:08:12,009
it really depends on
exactly how you configure the guest.
146
00:08:12,009 --> 00:08:14,857
If you install packages
it takes a bit longer.
147
00:08:14,857 --> 00:08:18,396
If you use virt-resize that
takes a bit longer too.
148
00:08:18,396 --> 00:08:25,241
But a basic guest usually
about 20 seconds or so.
149
00:08:33,441 --> 00:08:36,128
Finally, this is my
last slide
150
00:08:36,128 --> 00:08:38,228
libguestfs.org is
the website
151
00:08:38,228 --> 00:08:42,299
We have many more
tools using libguestfs
152
00:08:42,299 --> 00:08:44,039
We have a stable C API
153
00:08:44,039 --> 00:08:48,335
we have bindings in about
a dozen programming languages
154
00:08:48,335 --> 00:08:52,949
Almost certainly your Linux distro
includes libguestfs
155
00:08:52,949 --> 00:08:56,559
so it's just an "apt-get" or "yum"
install or use your package manager
156
00:08:56,559 --> 00:08:58,721
and it'll just work for you.
157
00:08:58,721 --> 00:09:01,679
Join us on the IRC channel
[#libguestfs on FreeNode]
158
00:09:01,679 --> 00:09:04,632
or the mailing list
[see libguestfs.org]
159
00:09:08,424 --> 00:09:30,705
Questions
160
00:09:50,627 --> 00:09:55,555
Q: Can you use virt-builder
with a base image
161
00:09:55,555 --> 00:10:00,824
to create an overlay,
which is much more efficient
162
00:10:00,824 --> 00:10:01,819
and much smaller?
163
00:10:02,777 --> 00:10:06,313
A: It doesn't right now, but
it will do in about 1 month.
164
00:10:06,313 --> 00:10:14,667
It is the most requested feature.
165
00:10:16,239 --> 00:10:24,004
Q: virt-diff: Is it the ordinary
diff, or a custom diff
166
00:10:24,004 --> 00:10:26,127
built on top of libguestfs?
167
00:10:32,856 --> 00:10:36,360
A: It is a special tool, but
it does use regular diff.
168
00:10:36,360 --> 00:10:41,164
It takes two disk images
and it uses the libguestfs API
169
00:10:41,164 --> 00:10:44,673
to iterate over the filesystems
in both images
170
00:10:44,673 --> 00:10:47,112
that's the custom code
171
00:10:47,112 --> 00:10:50,625
but when it finds a file
that has changed it runs
172
00:10:50,625 --> 00:10:54,708
diff on that file and
presents you with diff output.
173
00:11:31,967 --> 00:11:46,729
Q: lrzip and thin provisioning:
the results were a little bit better
174
00:11:46,729 --> 00:11:54,643
A: You're correct that lrzip ..
xz is not absolute the best
175
00:11:54,643 --> 00:11:56,516
compression tool out there.
There are ones which
176
00:11:56,516 --> 00:11:59,557
will give you a megabyte
or two better than xz.
177
00:11:59,557 --> 00:12:02,436
But it's still extremely
good compression.
178
00:12:02,436 --> 00:12:04,239
Because it has such a
well-defined API
179
00:12:04,239 --> 00:12:06,369
it's useful in many
other ways.
180
00:12:20,581 --> 00:12:24,577
Q: Will libguestfs manipulate
any sort of guest image?
181
00:12:24,577 --> 00:12:30,673
A: Yes. We use Linux kernel
code and Linux utilities
182
00:12:30,673 --> 00:12:36,483
If you could plug the disk
into a Linux computer
183
00:12:36,483 --> 00:12:39,521
and Linux could read it,
then we can read it.
184
00:12:47,473 --> 00:12:50,284
Q: You support qcow2
185
00:12:50,284 --> 00:12:54,492
A: Yes, we use KVM to
decode the qcow2 layer.
186
00:12:54,492 --> 00:12:57,797
Q: Do you support formats
like VHD?
187
00:12:57,797 --> 00:13:00,668
A: Yes we support anything
that qemu supports
188
00:13:00,668 --> 00:13:07,532
and qemu supports VHD,
VMDK, raw.
189
00:13:07,532 --> 00:13:12,009
We support everything that
qemu & Linux supports.
190
00:13:24,097 --> 00:13:27,665
Q: Do you plan to get
templates from other sources?
191
00:13:27,665 --> 00:13:30,059
A: Yes. We don't want
to supply templates.
192
00:13:30,059 --> 00:13:31,535
We only have a few
as you can see.
193
00:13:31,535 --> 00:13:33,904
What we want to do is
use the cloud images
194
00:13:33,904 --> 00:13:35,908
that distros are
already publishing.
195
00:13:35,908 --> 00:13:38,021
All we're going to do
is add metadata.
196
00:13:38,021 --> 00:13:43,583
The distros are going
to host those.
197
00:13:57,292 --> 00:14:04,873
There's a Ceph talk coming up next
and we can read images directly from Ceph.
198
00:14:04,873 --> 00:14:09,174
http://libguestfs.org/