Oracle VM VirtualBox: Ticket #3351: Normalise or repair the XML data modelshttps://www.virtualbox.org/ticket/3351
<p>
Currently the VBox metadata models are denormalised in that data elements are duplicated between the central <em>VirtualBox.xml</em> registry and the various machine.xml files. The VBox application does not enforce proper transactional integrity between these various definition files and so various program errors can cause them to get out of sync. A result of this is that VBox will then refuse to load the VM and the users are unable to recover the contents of any snapshotted HDs with understanding the XML schemas and manually patching up the XML files — not something that the average use is capable of doing. Hence certain classes of VBox error can result in major data loss for most users. There is sufficient volume of posts on the forum related to this issue to support the conclusion that this is a real problem.
</p>
<p>
As far as I can see there are three possible approaches to remove this product vulnerability:
</p>
<ul><li><strong>Renormalise the data models</strong>. For example the <em>&lt;MediaRegistry&gt;&lt;HardDisks&gt;</em> tree would only contain HDs that are not allocated to VMs. The complete set of <em>&lt;HardDisks&gt;</em> could be obtained by enumerating this tree plus the <em>&lt;HardDisks&gt;</em> in each attached machine. (This node would replace the existing <em>&lt;HardDiskAttachments&gt;</em> node). Such an approach would also mean that a VM is now a self-contained entity that can be exported or imported (a frequent user request). Yes this would require addition integrity checks to walk the VMs XML files when adding or removing HDs, but this is hardly a frequent occurrence. This is my recommended option.
</li><li><strong>Provide a repair too</strong>. This would walk the various XML files, identify inconsistencies and have a set of rules for when to recommend recovery and when to ask the user for a choice.
</li><li><strong>Move the meta data to a proper DB</strong>. The obvious one to use here is SQLlite: simple, free, and able to handle the transactional issues. Under this scenario the Machine schema would only be used for importing and exporting machines.
</li></ul><p>
This metadata is critical to the protecting the user content maintained within the various VMs, yet there are no transactional, journalling or recovery mechanisms provided to ensure the integrity of this metadata. This is unacceptable in product of VBox's maturity.
</p>
en-usOracle VM VirtualBox/graphics/vbox_logo2_gradient.pnghttps://www.virtualbox.org/ticket/3351
Trac 0.12achimhaMon, 16 Feb 2009 15:00:36 GMThttps://www.virtualbox.org/ticket/3351#comment:1
https://www.virtualbox.org/ticket/3351#comment:1
<p>
We plan to address this similar to what you propose as your first option. Repairing is difficult because you need to anticipate in your repair algorithm what could be broken. A database with SQL would just further complicate things and make it even harder to enhance the metadata format. XML allows for manual "repairs" which can be practical.
</p>
<p>
We plan to eliminate the explicit media registry (derive the information from the VM config instead). We will also move from XML schema to manual parsing and do it in a way where an older version does not mind additional data it cannot understand. This will make it easier to migrate back to an older version.
</p>
<p>
With the upcoming OVF support for VM import/export, the portability issue will be fully addressed. We therefore do not have make our internal metadata format portable across host systems.
</p>
<p>
So the XML changes will address robustness and version compatibility and OVF will give us the much demanded VM portability.
</p>
<p>
Any additional thoughts?
</p>
TicketTerryEMon, 16 Feb 2009 16:00:43 GMThttps://www.virtualbox.org/ticket/3351#comment:2
https://www.virtualbox.org/ticket/3351#comment:2
<p>
Achima, thanks for getting back to me.
</p>
<p>
My first comment is that moving from a XML / DTD conforming schema with standard parsing back to manual parsing would be a mistake. These are physical mechanisms. The people who write the standard parsers are experts in their fields and I think that you would find it difficult to do a better job of this.
</p>
<p>
The core issues here are that your logical data model for the metadata is physically denormalised and that you don't implement transactional integrity. Hence in the code you [start a logical transaction], do a lot of processing and make changes, [end a logical transaction]. At the moment the code can throw errors between this transaction start and end, and when these occur the users metadata can be left in an inconsistent state. One current consequence (with snapshots) is that this can functionally destroy the VM and its virtual disks.
</p>
<p>
Yet, this is an entirely tractable IT design issue, though perhaps not one that guys who develop VM software tend to think of. Filesystems and banking systems yes. VMMs no :-) It is a pretty straight forward piece of logical to physical design: what are your transaction and commit points? How is your normalised LDM mapped into your PDM? What is your physicalisation approach. Whether or not you make your internal metadata visible or portable, you still have these issues. A manual implementation will probably only mask these issues not address them.
</p>
<p>
Yes, using a SQL D/B involves a small increase in complication, but again this also encapsulates and addresses a whole lot of physicalisation issues at a stroke. Firefox, most Apple apps and most advanced mobile phones use SQLlite for this very reason. If it's not overkill for an iPod, I doubt that it's overkill for <a class="wiki" href="https://www.virtualbox.org/wiki/VirtualBox">VirtualBox</a>. The implementation is just a 250Kb library which you link into your apps and this handles a whole bunch of issues for you, and the D/B just looks like a single file. And yes, it comfortably handles multi-process access at the update transaction rates you see in VBox use.
</p>
<p>
In terms of repairs, most of the issues are to do with the redundancy introduced by the denormalisation and you can make simple rules to address this, for example: if the central registry and the VM are inconsistent then you update the central registry to be consistent with the VM — but again removing the redundancy is a better way. If you are reworking this area, I would suggest going to a SQLite D/B for the central metadata implementation and a method of importing and exporting VMs.
</p>
<p>
There are other specific issues with your current approach (such as your schema are linux/windows/macosx, rather than the platform neutral XML schema approach that OpenOffice.org adopts)
</p>
TicketachimhaMon, 16 Feb 2009 17:34:16 GMThttps://www.virtualbox.org/ticket/3351#comment:3
https://www.virtualbox.org/ticket/3351#comment:3
<p>
SQLite is not transactional and can royally screw up if interrupted (most of the issues we have with virtualbox.org is due to SQLite data corruption with trac). Once screwed up, you can't just use vi to edit. Also mapping the datamodel to relational is not straightforward and changing existing relations and add more data is not nice either.
</p>
<p>
OpenOffice is an example where a lot of thought is put into the format (read: many years). Also the scope of word processing is pretty well understood while virtualization is still rapidly evolving. We rely on our short innovation cycle which also means we spend less time designing the perfect (TM) file format but rather keep enhancing/changing it.
</p>
<p>
I understand your complaints and I share them. However, we are closer to resolution than you think. The static validation using XML Schema proved to be very inflexible as we are basically unable to augment the XML file and still use it with a previous version. Also the complicated referencing between the global and the VM XML files make it more likely to corrupt data and harder to fix the corruption. We will simplify that. We have made good experience with "manual" validation for the upcoming OVF support and we think that is the best solution.
</p>
<p>
We have decided to not enhance the XML format to allow for host portability but instead rely on OVF. This means we will only have to address some of the issues and not the biggest one (portability).
</p>
<p>
PS: <a class="wiki" href="https://www.virtualbox.org/wiki/VirtualBox">VirtualBox</a> was designed with pluggable configuration storage in mind, we initially intended to start with simple XML and then offer DBMS adapters. Have a look at the CFGM API. Somebody familiar with SQLite could actually give it a try and see how it works out.
</p>
TicketumoellerMon, 16 Feb 2009 18:02:41 GMThttps://www.virtualbox.org/ticket/3351#comment:4
https://www.virtualbox.org/ticket/3351#comment:4
<p>
Terry, a couple of things I'd like to add to Achim's comments, which I agree with, for what it's worth.
</p>
<p>
For one, you seem to worried that we're going to throw away stands-compliant XML parsing. This is not the case. We will continue to use our parsing based on libxml2, which works well. What doesn't work well is that we throw the VirtualBox XML files against a very strict version-specific XML schema, which makes our XML files dependent on a particular VirtualBox version and requires converting between these versions, which is not backward-compatible. This will need fixing, and we will do this according to the old principle "be liberal in what you accept and strict in what you write."
</p>
<p>
Secondly, using a database does not solve the problems we have. We have two, AFAICS: the versioning, which we'll fix as described, and possibly the transactions. Using a database locks us into using a particular table layout, which makes the versioning/conversion issue worse, not better. Also, the problems with atomic commits are independent of whether we use XML or sqlite: we can mess up the consistency either way. A transactional database only guarantees that individual commits are atomic, but for one, we have our doubts about sqlite from past experience, and secondly, we need to fix our data model, not the storage, to make commits atomic.
</p>
<p>
There's one big advantage of using databases, which is why Firefox switched to using sqlite: high-speed random access. Firefox needs to look through lots and lots of history entries, cookies and bookmarks, which is why I guess they switched. I'm not sure about sqlite and the iPod, but I know my 160 GB iPod has a lot of metadata on it, so I guess sqlite makes a lot of sense there too. But reading, parsing and writing the very small XML files sequentially is clearly not a use case for a database.
</p>
<p>
Finally, as Achim said, being able to fix issues with a text editor is a huge benefit of using XML.
</p>
<p>
All in all, sqlite can only give us speed, which we don't need. It doesn't fix the problems we need to fix. But we agree on those problems, it seems. :-)
</p>
TicketTerryEMon, 16 Feb 2009 21:04:31 GMThttps://www.virtualbox.org/ticket/3351#comment:5
https://www.virtualbox.org/ticket/3351#comment:5
<p>
Actually SQLite has had decent transaction support for some time and is suited to this sort of application embedded metadata use (much like Ff) though I am a little surprised that you use on an Apache delivered web app such as trac given the potential concurrency levels and size of D/B. However the physical implementation are quite secondary and entirely your choice, just not something to get into a tizzy about. Also if you are staying with the libxml2 parser, then ignore my related comments.
</p>
<p>
As you point out the major issue here is the logical date model and transaction design, and fixing that. We should not be separating update (and commitment) of the central registry and the machine registry by validation steps that can raise errors and get the two out of sync. An XML storage format is great for programmers and IT aware users, but as VBox's popularity extends it user base a high % of users regards XML on a par with Greek.
</p>
<p>
The main point that I am taking away from this is that the is already aware of these issues and are already planning to address them. I also support your move to OVF and away from host platform-specific DTDs. It would be useful if the main volunteers on the forum so that we can a better feel of how to handle the "I've lost everything" sort of posts that we get in the meantime.
</p>
TicketTerryEThu, 19 Feb 2009 15:12:13 GMThttps://www.virtualbox.org/ticket/3351#comment:6
https://www.virtualbox.org/ticket/3351#comment:6
<p>
Another issue which relates to transactional integrity, though not specifically the XML meta data has just been raise in <a class="ext-link" href="http://forums.virtualbox.org/viewtopic.php?t=14554"><span class="icon"> </span>Snapshot operation ruined the system</a> where a user complained that failure (in this case volume full) resulted in a snapshot delete aborting, leaving the snapshot set unusable. This is another example where the merge process is intrinsically additive (additional changed blocks are appended to the VD image file). This process would be quit easy to checkpoint and restore on failure.
</p>
<p>
The reason for my posting this here is that the VD image containers are essentially acting as hyper file systems, and there need end to end an equivalent integrity to journalling filesystems such as Ext3. Critical operations should be properly checkpointed and program failure should be trapped and the result rolled back to the checkpoint.
</p>
TicketfrankMon, 04 May 2009 14:37:07 GMTdescription changedhttps://www.virtualbox.org/ticket/3351#comment:7
https://www.virtualbox.org/ticket/3351#comment:7
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/3351?action=diff&amp;version=7">diff</a>)
</li>
</ul>
TicketDHughesThu, 23 Jun 2011 00:16:25 GMThttps://www.virtualbox.org/ticket/3351#comment:8
https://www.virtualbox.org/ticket/3351#comment:8
<p>
This problem still exists and needs to be re-visited/fixed.
</p>
<p>
I recently lost several days worth of development because of a corrupted registry and it is frustrating to find that the issue has been known about but unfixed for several years.
</p>
Ticket