As well as providing vendor-independent means of creating, retrieving, modifying and deleting XSets, the SNIA XAM v1.0 specification also defines a query language (XAM QL), based on a subset of the SQL language, for selecting and retrieving the XUIDs of XSets based on content-defined criteria.

Version 1.0 of the XAM specification defines two levels of query language support, i.e. Level 1 and Level 2. Level 1 defines queries on properties and field attributes in XSets and is mandatory. Any XSet property value that is accessible to an application program via the XAM library can be queried. Level 2 extends Level 1 to support queries on XStreams and is optional. Both levels of query are accessed through a single, defined job type that all XAM providers must support. Since no vendor that I am aware of has actually implemented level 2 queries, the remainder of this post focuses on level 1 queries.

A XAM query statement consists of a mandatory select clause followed by an optional where clause. For XAM v1.0 the only valid select clause is select '.xset.xuid'. This specifies that the application is requesting a list of XUID values. For example

select ".xset.xuid "

will return a list of every XSet that is readable at the time of the query.

The where clause is used to specify a subset of XSets to be matched. For level 1 queries it is restricted to comparisons between XSet properties and literal values and/or field attributes and literal values.

will return the list of all Xsets which were created on or after February 1st 2009.

The following table shows which field and literal types can be validly compared.[table id=14/]The XAM library validates that strings and strings liberals are conforming UTF-8 strings. Non-conforming UTF-8 string literals generate a XAM non-fatal query syntax error. Issues such as single versus multiple glyph characters and non printable characters are unspecified and are implementation and application dependent. String comparisons are case sensitive and comparison operators operate on a byte-by-byte basis. For relational operators the relationships are defined by the byte values. All data normalization is the responsibility of the application.

The supported escape sequences are the following. Any other escape sequence will generate an non-fatal error.[table id=15/]String literals must be quoted with single quotes. String literals which contain one or more single quote characters must escape each single quote character using a backslash. For example, the string literal of ’This is a string literal’ would be represented in a query as

select ".xset.xuid" where "com.example.string-property" = ’This is a string literal’

All instances of field names in a XAM query string must be quoted with double quotes. If a field name contains double quote characters, each double quote character must be escaped using a backslash. For example, the field name for the xam_boolean property com.example.”qstring” should be represented in a query as

select ".xam.xuid" where "com.example.\"qstring\"" = TRUE

If a field name contains a backslash character, the backslash itself must be escaped with another backslash. For example, the field name for the xam_double property com.example.file\ratio should be represented in a query as

select ".xam.xuid" where "com.example.file\\ratio" = 100.1

The query language accepts date-time and XUID-valued literals using the selector functions date() and xuid() respectively. The date() function takes a properly formed date-time value, specified as a literal string which is consistent with the XAM date-time specification.

The xuid() function expects an XUID which is in the form of a base64 encoded string literal. An improperly formed string literal generates a non-fatal error during query parsing.

Field attributes are accessed using the exists, typeof, readonly, binding, and length field attribute accessor functions. The exists() function tests for the existence of an named field attribute (property) in an XSet. It evaluates to TRUE if an XSet contains the named field otherwise it evaluates FALSE.

select ".xset.xuid" where exists("com.example.name")

The typeof() function returns the MIME type of a named field in an XSet which is a string-valued property.

This function can be used whenever an application could use a field reference to a string-valued property. Note that comparisons with any non-string literal value generate a non-fatal error during the parsing of the query.

The readonly() function evaluates to TRUE when a field in an XSet is marked as readonly.

select ".xset.xuid" where readonly("com.example.flag")select ".xset.xuid" where not readonly("com.example.name")

The binding() function evaluates to TRUE when a field in an XSet is marked as binding.

select ".xset.xuid" where binding("com.example.case_id")select ".xset.xuid" where not binding("com.example.subject")

The length() function returns the length, in bytes, of a named field.

select ".xset.xuid" where length("com.example.data") > 1024

When used on property fields this function returns the length as defined for stypes. Note that this function should not be used to compare the number of characters in a string as this comparison depends on the character encoding being used.

Turning now to logical operators. Subclauses within the where clause may be combined and modified by using the logical operators and, or and not. These operators are similar to their SQL counterparts.

The and operator requires both subclauses to evaluate to TRUE before including an XSet in the results. For example

selects only those XSets containing a named stream of image type JPEG or GIF. The not operator negates a boolean expression. For example

select ".xset.xuid" where not binding("com.example.property")

selects all XSets with the nonbinding property com.example.property.

As in SQL certain operators take precedence over other operators in queries. XAM QL operator precedence is as follows:[table id=16/]Operators of the same precedence are evaluated left to right within a query. However operator precedence may be overridden using parentheses as shown by the following example.

select ".xset.xuid" where not "com.example.bool-prop" and "com.example.int-prop" = 64select ".xset.xuid" where not ("com.example.bool-prop" and "com.example.int-prop" = 64)

In the first example, the not operator only applies to the com.example.bool-prop property. In the second example, the not operator applies to both com.example.bool-prop and com.example.int-prop = 42.

Now that we are somewhat familiar with XAM QL syntax, the next question to be answered is how do we tell a XAM Storage System to execute a query and return the results to us. Well it turns out that a query is executed as a particular type of XAM job called a query job. The input to a query job is an XSet which you must create. This XSet must contain two items - the field org.snia.xam.job.command which is always set to 'xam.job.query' and a XStream xam.job.query.command which is a UTF-8 text stream (MIME type 'text/plain; charset=utf-8’) containing the actual query expression string.

A query job may optionally be committed before or during the execution of the job, but this commit is not required. The ability to commit a running job is optional and implementation dependant. An application can determine if job commit is supported by checking the XSystem boolean property .xsystem.job.commit.supported. Failure to commit a query job before closing the XSet means the results of the query job are not persistently stored.

A query job is executed by invoking submitJob. If either of the two required items in the query job XSet is not correctly filled in, the standard job error field .xam.job.error is added to the query job XSet and it's value is set to either org.snia.xam::not_a_job or org.snia.xam::unspecified_command and the query job is aborted. As shown in the sample application below, the state of a query job may be determined by examining the value of .xam.job.status.

When a query job is successful, the results are stored in the query job XSet in the form of a new XStream named xam.job.query.results. This XStream has a MIME type of application/vnd.snia.xam.query.xuid_list and contains the XUIDs for the set of XSets resulting from the evaluation of the query. Each XUID is stored in a binary format as part of an 80-byte record. If an XUID is shorter than 80 bytes, the record is zero padded to 80 bytes.

A number of other properties relating to the results XStream are also set up. The xam_int property xam.job.query.results.count contains the current count of the XUID records in the results XStream. This property is updated as results are entered into the results XStream during query processing. An application can use this to provide status information to users as the query job executes. The xam_string property xam.job.query.level indicates the query level which matches the results. Its value is either org.snia.xam.job.query.level.1 or org.snia.xam.job.query.level.2.

It is important to remember that a query job operates within the roles and permissions granted to the connection. That means the results XStream only includes those XSets that are visible and accessible to an application, at least from a read perspective, according to the role under which the query job is executed.

The query job may set the following error codes in .xam.job.error:[table id=17/]When a query job is halted for whatever reason, the standard job health and status fields, i.e .xam.job.errorhealth and .xam.job.status, are set to appropriate values and the XAM Storage System may place zero or more additional XUIDs in the results XStream. Thus an application should always see complete XUID values in the results XStream. Resuming a halted query job is not supported.

Note that, unlike a traditional RDBMS, there is no locking of an XAM Storage System during the execution of a query job. Thus a query result is not an instantaneous snapshot of a XAM Storage System. The contents of a XAM Storage System may or may not change while a query job is executed. The general rule is that any XSet that has been stored before the start of the query job is included in the results XStream if it meets the query criteria, and any XSet that has been stored after the query job has completed is not be included in the results XStream. XSets that are stored during the query job execution may or may not be included. The same criteria applies to the results XStream. It may include XUIDs of XSets that are no longer in the XAM Storage System and XUIDs of XSets that no longer meet the search criteria due to a change in the XSet after the particular XSet was queried.

Here is a simple Java application that queries an XAM Storage System to determine it's level of support for XAM queries.

Note that a results XStream produced by a query job may be consumed by an application before the XAM Storage System has finished storing all of the XUIDs in the results XStream if XStream.asyncRead is used instead of XStream.read.

Well I think I have provided enough information and examples to enable you to go play with the XAM query language by yourself. It you need help the I am sure the XAM Developer Group on Google Groups will be more than happy to help you.

In parting I would point out that since XAM Storage System providers are currently only required to implement support for queries over the content metadata, I suspect that support for actual full content-based searches using XAM may not occur for some time.

IA-PC HPET (High Precision Event Timer) is a specification which was jointly developed by Intel and Microsoft in the early part of this decade.. The latest version is dated October 2004. It's stated purpose is to

initially supplement and eventually replace the legacy 8254 Programmable Interval Timer and the Real Time Clock Periodic Interrupt generation functions that are currently used as the ‘de-facto’ timer hardware for IA-PCs.

The HPET architecture defines a set of timers that can be used by the operating system. A timer block is a combination of a single counter and up to 32 comparators and match registers. The comparator compares the contents of the match register against the value of a free running monotonic up-counter. When the output of the up-counter equals the value in the match register an interrupt is generated. Each of the comparators can output an interrupt. A maximum of 8 timer blocks are supported for a total of 256 timers. Each timer block can have different clocking attributes. Specific implementations may include only a subset of these timers. A minimum of three timers is required.

The specification contains the following block diagram of the HPET architecture.

Some of the timers may be enabled to generate a periodic interrupt. If a timer is set to be periodic, its period is added to the match register each time a match occurs, thus computing the next time for this timer to generate an interrupt.. An up-counter is usually 64 bits wide but 32-bit implementations are permitted by the specification and 64-bit up-counters can also be driven in 32-bit mode. Up-counters run at a minimum of 10 MHz. which is much faster than the older RTC (Real Time Clock) and can thus produce periodic interrupts at a much higher resolution. The registers associated with these timers are mapped to memory space.

The BIOS uses ACPI ( Advanced Configuration and Power Interface) functionality to inform the operating system of the location of the HPET memory-mapped register space. Here is an example of a disassembled ACPI HPET table from an Intel DX48BT2 (AKA BoneTrail) motherboard.

See page 30 of the HPET v1.0a specification for a detailed breakdown of the individual bits in the Event Time Block (called Hardware Block by the AML disassember). Note that only one Event Timer Block need be described in the HPET table in order to bootstrap an operating system. This is the case here. For non-legacy platforms, the Event Timer Block described in the HPET is the one that provides functionality to replace the 8254/RTC Periodic Interrupt Logic.

Other Event Time Blocks are described in the ACPI namespace. Here is the relevant section from the disassembled ACPI DSDT table.

The first line is outputted when the ACPI HPET table is read. The second line is outputted when the ACPI HPET table is mapped into memory by .../arch/x86/kernel/acpi/boot.c. The next line is outputted when the HPET legacy interrupts are started and HPET is registered as the global clock. The following line is outputted when the kernel checks to ensure that at least one timer is reserved for userspace (/dev/hpet.) The next two lines of output comes from the HPET device driver (.../drivers/char/hpet.c.) It shows that 2 timers have allocated interrupts and two do not..

Here is the relevant part of the output from /proc/time_list as it relates to HPET:

The HPET driver (/dev/hpet) has a similar API to the Real Time Clock driver. It is a character device which can support any number of HPET devices. The kernel API has three interfaces exported from the driver:

Well, I think that I have provided you with enough information so that you should now be able to go away and experiment with the HPET interface yourself.

By the way, not all VMware products support HPET. Currently ESX does not provide a virtual HPET to guest operating systems and in some cases it may be necessary to disable HPET altogether because of timer drift in virtual machines. See VMware TimeKeeping for more information.

P.S. I tested the the above example on an Intel DX48BT2 motherboard running a 2.6.29.5-191 kernel.