Abstract

The Protocol for Web Description Resources (POWDER) facilitates the publication of descriptions of multiple resources such as all those
available from a Web site. This document describes how sets of IRIs can be defined such that descriptions or other data can be applied
to the resources obtained by dereferencing IRIs that are elements of the set. IRI sets are defined as XML elements with relatively
loose operational semantics. This is underpinned by the formal semantics of POWDER which include a semantic extension, defined
separately. A GRDDL transform is associated with the POWDER namespace that maps the operational to the formal semantics.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This is the Third Last Call Working Draft of this document, the Last Call period being
synchronized with two other documents in the set:
Description Resources and
Formal Semantics.
The call is being made primarily to solicit any further feedback on the issue of IRI canonicalization
(see Section 2.1.3 to Section 2.1.5). Other
sections of this document, and the entirity of the Description Resources and Formal Semantics
documents, are not expected to undergo any further substantive changes.

These three documents are expected to be advanced to Recommendation Status with Proposed Recommendation
as the next step in that process. The Working Group does not expect to issue a separate call
for implementations (Candidate Recommendation phase). The POWDER Working
Group welcomes comments through to 27 April 2009. Comments are equally welcome on
the other documents that are also available as working drafts, in particular, the
Primer [PRIMER] and Test Suite [TESTS]. Changes
to this document since the previous version
are recorded in the Change Log.

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft
document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate
to cite this document as other than work in progress.

1 Introduction

The Protocol for Web Description Resources (POWDER) facilitates the publication of descriptions of multiple resources such as all those available from a Web site. These descriptions are attributable to a named individual, organization or entity that may or may not be the creator of the described resources. This contrasts with more usual metadata that typically apply to a single resource, such as a specific document's title, which is usually provided by its author.

Description Resources (DRs) are described separately [DR]. This document sets out how groups (i.e. sets) of resources may be defined, either for use in DRs or in other contexts. Set theory has been used throughout as it provides a well-defined framework that leads to unambiguous definitions. However, it is used solely to provide a formal version of what is written in the natural language text.

POWDER uses a limited set of XML elements to define sets of resources and these have relatively loose semantics. However, a GRDDL [GRDDL] transform is associated with the POWDER root namespace through which formal semantics are accessible as RDF/OWL. This is known as Semantic POWDER or POWDER-S. The details of the GRDDL transform and the formal semantics are defined separately [FORMAL] and outlined in Section 1.4 below. The use cases, a primer, test suite and schema namespace documents complete the document set.

The POWDER schema namespace is http://www.w3.org/2007/05/powder# for which we use the prefix wdr. The POWDER-S namespace is http://www.w3.org/2007/05/powder-s# for which we use the prefix wdrs and All namespaces and prefixes used in this document are shown in the table below.

Table 1: Namespace and prefixes used in this document

Prefix

Namespace

wdr

http://www.w3.org/2007/05/powder#

wdrs

http://www.w3.org/2007/05/powder-s#

rdf

http://www.w3.org/1999/02/22-rdf-syntax-ns#

rdfs

http://www.w3.org/2000/01/rdf-schema#"

owl

http://www.w3.org/2002/07/owl#

xsd

http://www.w3.org/TR/xmlschema-2/

ex

An arbitrary prefix used to denote an 'example vocabulary'

In this document, the words MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are to be interpreted as described in RFC 2119 [RFC2119].

White space is any of U+0009, U+000A, U+000D and U+0020. A space-separated list is a string of which the items are separated by one or more space characters (in any order). The string may also be prefixed or suffixed with zero or more of those characters. To obtain the values from a space-separated list user agents MUST replace any sequence of space characters with a single U+0020 character, dropping any leading or trailing U+0020 character, and then chopping the resulting string at each occurrence of a U+0020 character, dropping that character in the process.

The (unqualified) terms POWDER, POWDER Document and Description Resource (DR) refer to operational representations and semantics. The term POWDER-S refers to documents and data that express the formal semantics of POWDER. Unqualified XML element names are in the POWDER (wdr) namespace

1.1 Design Goals and Constraints

In designing a system to define sets of resources we have drawn on earlier work [Rabin] carried out in the Web Content Label Incubator Activity [WCL-XG], and taken into account the following considerations.

It must be possible to define a set of resources, either by describing the characteristics of the IRIs of resources in the set, or by simply listing its elements.

It must be possible to determine with certainty whether a given resource is or is not an element of the Resource Set, as long as the resource's IRI is known.

The ease of creation of accurate and useful Resource Sets is important.

It should be possible to write concise Resource Set definitions.

Resource Set definitions must be easy to write, be comprehensible by humans and, as far as is possible, should avoid including or excluding resources unintentionally.

It must be possible to create software that implements Resource Set definitions primarily using standard and commonly available components and specifically must not require the creation of custom parsing components.

So far as is possible, use of processing resources should be minimized, especially by early detection of a match or failure to match.

1.2 Outline Methodology

Operationally, POWDER does not define resource sets, rather, it facilitates the definition of sets of IRIs (International Resource Identifiers) [IRIS], which can be used to denote resources in terms of their identifiers. We use the notion of IRIs instead of URIs[URIS] since IRIs are a superset of URIs. Therefore, an IRI set definition may denote a set of IRIs as well as a set of URIs.

Defining a resource set by specifying the characteristics that the identifiers of resources in the set share is clearly an indirect approach, albeit a very useful one in the real world. In a logical sense, the definition must be interpreted to arrive at the full set.

More formally, an IRI Set definition D denotes a set of IRIs IS = DI, where DI is the interpretation of D, i.e., the set of IRIs sharing the characteristics denoted by D.

We take this further and allow an IRI set definition to be built up in stages.

An IRI set IS is denoted by an IRI set definition DIS in terms of one or more characteristics that the elements of the set have in common. Each characteristic is expressed by an IRI constraintC, and IRI constraints C1, C2, … Cn give rise to IRI set definitions D, 1, D2, … Dn, so that the complete IRI set definition DIS comprises D1, D2, … Dn.

The IRI set IS is the intersection of the IRI sets denoted by the IRI set definitions in DIS.

Formally:

IS = DISI = D1I ∩ D2I ∩ … ∩ DnI = (D1 ∧ D2 ∧ … ∧ Dn)I.

For example, suppose that an IRI set IS is denoted by the following definitions:

D1: “the top level components of the host component of the IRI exactly match example.org”

D2: “the path component of the IRI begins with /foo“

Then, DIS will be defined as follows: “the top level components of the host component of the IRI exactly match example.org” AND “the path component of the IRI begins with /foo.”

Whether the IRI of a specific resource R, known as the candidate resource, is a member of IRI Set IS or not is determined by comparing its characteristics with those denoted by the set definitions used in DIS. It must be an element of the intersection of the sets defined by the interpretation of D1, D2, …, Dn to be an element of IS.

If an IRI set definition contains no constraints, then its interpretation is by definition the empty set ∅. Formally:

Let IS be an IRI Set, and let DIS be the set of IRI Set definitions denoting the IRIs in IS: if DIS = ∅, then IS = ∅.

1.3 Operational Semantics

The POWDER XML schema [WDR] defines the set of XML elements and attributes to be used for enforcing the operational semantics of an IRI set definition.

More precisely, we define an XML element iriset to take the place of the IRI set, and its child elements denote the set of IRI constraints C1, C2, …, Cn. The example reported in the previous section can therefore be written as follows:

1.4 Formal Semantics

The operational semantics described above are underpinned by formal semantics. A GRDDL [GRDDL] transform
is associated with the POWDER namespace that allows the XML data to be rendered and processed as RDF/OWL with one important
proviso — that a semantic extension is understood. Defined fully in the Formal Semantics document
[FORMAL], this allows a candidate resource's IRI to be matched against regular expressions that are values
of an OWL data type property wdrs:matchesregex(or wdrs:notmatchesregex in
the case of patterns that are to be excluded). An OWL class takes the place of the IRI set
and resources whose IRIs match all the property restrictions defined using wdrs:matchesregex and wdrs:notmatchesregexare
instances of that class. The regular expression syntax used is defined by XML schema as modified by XQuery 1.0 and
XPath 2.0 Functions and Operators [XQXP].

As shown in Example 1-1 above, the POWDER XML elements generally take strings as values. These are converted
into regular expressions as a first step in the GRDDL transform which renders POWDER documents in an intermediate format known as
POWDER-BASE. It is POWDER-BASE that is then transformed into POWDER-S. For clarity, this two-stage process is not referred to in
the main section of this document on defining a resource set which only presents POWDER and POWDER-S examples. POWDER-BASE
is, however, an important part of the extension mechanism of POWDER Resource Grouping. The Formal
Semantics document gives full details of the transformation of all elements of POWDER documents to POWDER-BASE and POWDER-S.

The result of the GRDDL transformation on Example 1-1 above is shown below.

2 Defining a Resource Set

A Resource Set is defined in terms of the IRIs of resources that are its members. Determining whether a candidate resource is, or is not, a member of the set, can therefore be done by comparing its IRI with the data in the set definition. Importantly, defining the Resource Set in terms of IRIs allows us to verify whether the candidate resource is in the set without having to fetch and parse it, or perform a DNS lookup, thus maximizing processing efficiency in many environments.

We define a range of methods to support set definition by IRI, and provide support for methods defined in other Recommendations.

2.1 Constraints on IRI components

The syntax of an IRI, as defined in RFC 3987 [IRIS], provides a generic framework for identification schemes that goes beyond what is demanded by the POWDER use cases [USECASES]. We therefore limit our work to IRIs with the syntax: scheme://iuser@ihost:port/ipath?iquery#ifragment, as shown below:

For the scheme, ihost, port, ipath, and iquery IRI components we define corresponding IRI constraints,
the value of most of which is a white space-separated list of strings, any one of which must match
the relevant portion of the IRI of the candidate resource.
The exceptions are the iquery and ihost components, which we discuss further in sections
2.1.2 and 2.1.3.

The iuserinfo and ifragment components are not used in POWDER IRI set definitions directly as it is felt that these may add a layer of unnecessary complexity with few practical applications. That said, it is important not to discard these components when processing the candidate resource's IRI. Furthermore, IRI sets may be defined using additional vocabularies as set out in Section 3. That extension method, or the use of the includeregex and excluderegex properties (see Section 2.3 below), means that user info and fragments can be used in IRI set definitions if required.

Formally, an IRI set definition D is expressed by one or more IRI constraints of the form C = IRI_component_matches(?x, {string1 | string2 | … | stringn}), where ?x is a variable denoting the IRI component under consideration, and {string1 | string2 | … | stringn} denotes a set consisting either of string string1 OR string2 OR … OR stringn.

Any number of IRI constraints C1, C2, …, Cn can be declared, and, as stated in Section 1.2, the overall IRI set is the intersection of the sets that can be interpreted from IRI set definitions corresponding to Cn. With some exceptions, each particular IRI constraint can only appear 0 or 1 times.

Strings are matched according to one of four rules:

startsWith, meaning that the IRI component starts with any of the strings listed in the value of the relevant IRI constraint;

endsWith, meaning that the IRI component ends with any of the strings listed in the value of the relevant IRI constraint;

exact, meaning that there is an exact match between the candidate IRI component and at least one of the strings listed in the value of the relevant IRI constraint;

contains, meaning that at least one of the strings listed in the value of the relevant IRI constraint appears somewhere in the IRI component.

Recognizing the great diversity of potential uses and set definition requirements, multiple IRI constraints are defined relating to the path component. Furthermore, for each constraint there is a ‘negative’ constraint, that is, a constraint whose value is a list of strings that must not be present in the relevant IRI component.

Table 3: Basic IRI constraints used to define IRI sets. These and other elements introduced in
subsequent sections are summarized in the Appendix.

†includepathcontains may appear any number of times within an IRI set definition, so that it is easy to create one in which multiple strings must be present in paths. This is in contrast to all other terms in Table 3 which can only occur 0 or 1 times, since the IRI of a candidate resource can only have one scheme, one host etc.

As a quick example, the set of all resources on example.org, whether fetched using specifically http or https, where the path component of their IRIs starts with foo, and where the path does not end with .png or .jpg is defined thus:

As outlined in Section 1.4, the POWDER GRDDL transform maps the IRI constraints in Table 3 to regular expressions against which the candidate IRI can be matched. These are shown in Table 4 below where var means the value of the XML element following processing as set out in the formal semantics document [FORMAL]. In brief this turns white space separated lists of strings into alternative values within the regular expression such that:

<includehosts>example.org
example.com
</includehosts>

becomes

(example\.org|example\.com).

Table 4. Template regular expressions for IRI constraints that take a white space separated list of values. See Section 2.3
for details of the meta character escaping used in these regular expressions.

These template regular expressions may be useful in processing POWDER documents directly but other methods of determining whether a candidate IRI does or does not match a particular constraint are equally valid.

Although ports are clearly integers, POWDER treats them as a string in the same way as the other constraints in Table 3. Port ranges (such as 80-100) are not supported but note that the value of includeports and excludeports is a white space separated list so that multiple ports may be enumerated.

Query strings typically contain a series of name-value pairs separated by ampersands thus:

?name1=value1&name2=value2

These are usually acted on by the server to generate content in real time and the order of the name-value pairs is unimportant. For practical purposes ?name1=value1&name2=value2 is equivalent to ?name2=value2&name1=value1. As a result, a significant amount of processing must be done to determine whether or not a candidate IRI is or is not an element of an IRI set that includes either the includequerycontains or excludequerycontains IRI Constraints.

To keep such processing manageable, the includequerycontains and excludequerycontains
IRI Constraints take a single value not a white space separated list of values. Including two or more
name/value pairs in an includequerycontains or excludequerycontains means that all those
pairs must be present in the query string for the candidate IRI to be a member of the set.
Section 2.6 includes a further discussion on creating unions of multiple IRI sets
where it is necessary to create a set defined in terms of alternative name/value pairs.

By default, the POWDER GRDDL transform assumes that the delimiting character in a query string is the ampersand (&). However, an alternative delimiter can be specified as the value for the delimiter attribute on includequerycontains and excludequerycontains constraints. Example 2-3 below shows this.

The GRDDL transform splits the value provided for the includequerycontains or excludequerycontains IRI Constraints into its constituent pairs at the delimiting character and the presence of each name-value pair within the candidate IRI is then tested for independently. The template regular expression for such a test is:

Where d is the delimiting character and q is the name-value pair. The Formal Semantics document [FORMAL] sets this out in more detail.

An important consequence of this processing model is that within the query string, only complete name-value pairs or value-less parameters are matched. More precisely, only complete query conjuncts in the query string are matched. As complete query conjuncts we consider any minimal substring of the query string that has ? or d before the first character and d or $ after the last character, where, as in the template regular expression, d is the query delimiter and $ is the end-of-string.

If the value of includequerycontains in Example 2-3 were changed to simply abcdef (rather than id=abcdef,group=12345) then:

http://socialnetwork.example.com/?group=12345,id=abcdefwould not match, even though the query does contain the string abcdef

http://socialnetwork.example.com/?abcdefwould match.

http://socialnetwork.example.com/?abcdef=ijklmwould not match

Again, a POWDER processor may use alternative methods to determine whether a given name-value pair is present in a candidate IRI but the template regular expression is used in the GRDDL transform to generate the POWDER-S shown in Example 2-4. Notice that the pre-processing described here allows POWDER-S to use the same restriction on the wdrs:matchesregex data property as the other elements in Table 3.

Equivalence of internationalized domain names is defined in terms of ASCII case insensitive
comparison of the output of the ToASCII operation (section 2 of [RFC 3490]).

If a regular expression is constructed to match the ihost component of an IRI, then it MUST
be constructed to match the ASCII representation of that component.

Specifically, to generate a regular expression from an includehosts or excludehosts constraint,
the value of each space separated token under the ToASCII operation, with the
UseSTD3ASCIIRules flag unset and the AllowUnassigned flag set, MUST
be used. [RFC 3490] Behavior if the ToASCII operation fails is undefined.

When internationalized (non-ASCII) domain names appear in the values of the includehosts
and excludehosts constraints, they SHOULD be represented either as the output of a successful
ToASCII operation, or as a string of Unicode characters that is mapped to itself when the
ToASCII and ToUnicode operations are successfully applied, in this order.
(Note, in particular, that behavior for these strings is well-defined, since the ToASCII
operation must succeed by definition.)

2.1.4 IRI/URI Canonicalization

Before determining whether an IRI is or is not a member of a given set, it is important to make sure that the
IRI is represented in the same way as the data against which it is being matched. We set out below a series of steps that
are designed to help processors make a best effort in this regard, which are consistent with
RFC 3986 [URIS], RFC 3987 [IRIS],
URISpace [URISpace] and XForms [XFORMS].

2.1.4.1 Character encoding

If not already so encoded, the IRI character string is converted into a sequence of Unicode [UNICODE] characters.

Percent encoded triples are converted into the characters they represent (e.g. %c3%a7 becomes ç etc.). Note the hexadecimal digits are case-insensitive. However, except for the query string, reserved characters as per Section 6.2.2.2 of RFC 3986 [URIS] must not be converted to literals, as that may invalidate the URI/IRI - the reason a URI would contain (for example) %2F instead of / would be to distinguish between a literal /, such as in 'his/hers', and the / which is used as a path separator.

+ characters in the query string are converted to spaces

Table 5: Examples of percent-encoding conversion

Input IRI/URI

Canonical form

http://example.com/staff/Fran%c3%a7ois

http://www.example.com/staff/François

http://example.com/my%20doc.doc

http://www.example.com/my doc.doc

In this next example the %2F is a literal slash, not a path separator, and so is left as %2F

http://www.example.com/foo/his%2Fhers

http://www.example.com/foo/his%2Fhers

Normalized to Form C, as defined in Character Model for the World Wide Web 1.0: Normalization [CHARMOD-NORM].

2.1.4.2 Default Values and Case Folding

Where the authority is present, but the scheme is absent, the scheme should default to http.

If the Path is absent, a path of / is appended.

Change the scheme string to ASCII lowercase.

If the host string does not completely consist of ASCII characters, apply the
ToASCII operation to the host string, with the UseSTD3ASCIIRules flag unset and
the AllowUnassigned flag set [RFC 3490].
Change the host string to ASCII lowercase. Note that behavior if the ToASCII operation
fails is undefined.

2.1.5 Data encoding

To complement the IRI canonicalization steps described in the previous section, related processing steps must also be carried out on the strings supplied as set defining data.

The following steps should therefore be applied to each item in the list separately.

If not already so encoded, the strings are converted into a sequence of Unicode characters.

With the exception of the reserved characters defined in Section 6.2.2.2 of RFC 3986 [URIS], percent encoded triples are converted into the characters they represent.

If the data refers to the query string, any sequence of + characters is converted to a space and all %-encoded characters are converted to the characters they represent, including those reserved in RFC 3986 [URIS].

If the data relates to the host, and does not completely consist of ASCII characters, the ToASCII operation is applied as described in Section 2.1.3

If the data relates to the scheme, it is normalized to ASCII lower case.

If the data relates to the host, trailing . characters are removed.

Any values given for the IRI constraints includepathstartswith, excludepathstartswith, includeexactpaths or excludeexactpaths must begin with the / character which is pre-pended if absent.

If the IRI set definition includes values related to the port then matching of the data against the candidate resource's IRI must be carried out as follows:

If the set definition includes the IRI constraint includeports then, when matching, if the default port for the candidate resource's IRI is present in the list of supplied values, but the candidate resource's IRI does not specify the port, the candidate resource IS an element of the set IF all other conditions are met.

If the set definition includes the IRI constraint excludeports then, when matching, if the default port for the candidate resource's IRI is present in the list of supplied values, but the candidate resource's IRI does not specify the port, the candidate resource is NOT an element of the IRI Set.

2.2 Grouping using Wildcards: The includeiripattern and excludeiripattern IRI constraints

It is anticipated that resource groups will typically be defined in terms of the
domains and sub domains from which they are available. In order to provide as much flexibility as possible
in this regard, the includeiripattern and
excludeiripattern properties allow domains and sub-domains
to be substituted by a wildcard character (*) according to the following EBNF (originally developed by
the Web Application Formats Working Group [WAF]):

scheme and port are used as defined in
RFC 3986 [URIS]. domain is an internationalized
domain name as defined in RFC 3490 [RFC3490].

It follows that:

<includehosts>example.com</includehosts>

and

<includeiripattern>example.com</includeiripattern>

are equivalent. However, *.example.com, meaning resources on sub-domains of example.com but not on example.com itself, is not a valid value for includehosts.

In contrast to the IRI constraints shown in Table 3, includeiripattern and excludeiripattern take a single pattern, not a white space separated list of values. Note that paths and query strings MUST NOT be included in the pattern. If these are required in an IRI set definition, the relevant IRI constraints from Table 3 can be used.

Any processing method that accurately tests a candidate IRI against the value of an includeiripattern or
excludeiripattern element is valid but the POWDER GRDDL transform does it in the same way as the other
IRI constraints, namely by creating a restriction on the
wdrs:matchesregex and wdrs:notmatchesregex properties as shown
in the example below. Full details of the transformation are provided in the Formal Semantics
document [FORMAL] however it is worth noting here once again that
if a regular expression is constructed to match the ihost component of an IRI, then
it MUST be constructed to match the ASCII representation of that component.

Example 2-5: An IRI Set defined using the includeiripattern and
excludeiripattern constraints

It is believed that the IRI constraints discussed above will be easy to use and cover the overwhelming
majority of POWDER use cases. However, the use of strings with fixed matching rules clearly presents a restriction
on flexibility. To support fully flexible set definition by IRI, the
includeregex and excluderegex
properties take a Regular Expression and should be applied to the candidate resource's complete IRI (after following
the canonicalization steps above and ensuring that regular expressions are
constructed using the ASCII representation of IRI components). In common with
includepathcontains and excludepathcontains, and unlike the other IRI constraints,
includeregex and excluderegex may occur any number of times in an IRI set. This
follows from the GRDDL transformation associated with POWDER introduced in
Section 1.4. The same section notes that the regular expression
syntax used is that defined by XML schema as modified by XQuery 1.0 and XPath 2.0 Functions and
Operators [XQXP].

For POWDER-S, the regular expressions are copied verbatim as values
for the wdrs:matchesregex and wdrs:notmatchesregex properties.

N.B. The value of the includeregex and excluderegex properties MUST be a single Regular Expression, not a white space-separated list.

As an example, the set of all the resources hosted either by example.org or example.net, where the path component of their IRIs starts either with foo or bar, can be defined thus:

Example 2-6: IRI set definition by regular expression (not including character escaping)

2.3.1 Safe Use of includeregex

Example 2-7 uses a modified version of the regular expression given Section 2.1. This is the safest method but is not, perhaps, the most natural way to proceed. If a less rigorous approach is taken it is easy to make mistakes when specifying regular expressions, and incorrect regular expressions in set definitions will have one of two possible (and obvious) consequences

The intention of the regular expression given in Example 2-8 is probably to say "all resources on
example.org with a URI beginning with https." However, as the regular expression is not
anchored at either end, what this actually means is "all resources on example.org where the URI
includes https". Thus this IRI set includes both of:

https://www.example.org/page.html

http://www.example.org/why_we_use_https.html

Adding in anchors at the beginning and end of the regular expression can have equally undesirable consequences.

Example 2-9: A second example of a bad set definition by regular expression

In Example 2-9, the intention is, again probably, to define the set of "all resources on example.org fetched using https only". However, adding both the ^ and $ anchors at the beginning and end of the regular expression means that the whole IRI must be https from start to finish — which can never be true so this IRI set is equivalent to the empty set.

Example 2-10 shows one possible way to encode the intended set definition.

Example 2-10: An example of a correct set definition by regular expression

Whilst Example 2-10 'works', the potential dangers of using regular expressions mean that it is generally better to use component strings where possible. Example 2-10 is therefore better written as shown in Example 2-11 below.

Example 2-11: A re-write of Example 2-10 without using a regular expression

2.4 Grouping by IP Address

It is noteworthy that POWDER does not define any special procedures where the host component of an IRI is expressed as an IP address. These are treated as strings, not as a sequence of digits. If the intention is to define an IRI set that encompasses a particular group of resources however they are accessed then it may be appropriate to include both the domain name and associated IP address as two space separated values in an includehosts element for example. However, this assumes that there is a one to one relationship between the domain name and the IP address which, of course, is often not the case.

As noted in Section 1.2, POWDER defines sets of IRIs, not of the resources that they identify. IRI sets must therefore be defined with care. For operational reasons, a user agent MAY perform a DNS or reverse DNS lookup to match domains names and IP addresses but this is very much application-specific.

2.5 Enumerating Elements of an IRI Set: the includeresources and excluderesources Constraints

It is useful to be able to include or exclude IRIs from sets by simple listing. The includeresources and excluderesources constraints support this, both of which take white space separated lists of IRIs. To give a simple example, the set of all resources on example.orgexcept its stylesheet and JavaScript library can be encoded as shown in Example 2-12 below.

Example 2-12: IRI Set definition using the excluderesources constraint

As emphasized throughout this document, each constraint and its value creates a set definition of its own and the full IRI set is the intersection of those sets. Thus an alternative way of looking at Example 2-12 is to say that a candidate IRI is a member of the IRI set IF it is on example.org AND does not have the IRI http://www.example.org/stylesheet.css AND does not have the IRI http://www.example.org/jslib.js.

2.6 Complex Sets: Negation, Conjunction and Disjunction

POWDER allows a DR to express any grouping of resources whatsoever, no matter how complex.

Atomic negation is achieved by complementing each IRI constraint that includes certain IRI
components by one that excludes them, and vice versa; furthermore, all includeX and excludeX constraints
are mutually exclusive. The analogous properties matchesregex and notmatchesregex properties are used in POWDER-S. Negation of complex constraints is not supported.

Conjunction of atomic propositions (both positive and negative) is inherent in the basic model - an IRI must match all the constraints if it is to be an element of the set. The GRDDL transform uses owl:intersectionOf to render in POWDER-S iriset elements with multiple constraints.

The disjunction of conjunctions of atomic propositions (both positive and negative) is also possible, as a DR may contain multiple iriset elements, and if any of them holds, then the DR holds. The GRDDL transform encodes multiple iriset elements as multiple clauses in POWDER-S.

It follows from the above, that POWDER allows the expression of Disjunctive Normal Form propositions. Since arbitrarily complex propositions can be brought into DNF (DNF Theorem), it follows that POWDER allows the expression of any proposition.

Example 2-14 shows a Description Resource defining the set of IRIs on example.com with a path beginning with /foo and those on example.org where the path starts with /bar.

Example 2-14: A Description Resource with its scope defined by the union of two IRI sets [XML]

3 Extension Mechanism

In this document we have specified various methods for defining sets of resource identifiers. The elements are clearly designed to be used with information resources available on the Web, identified by IRIs containing host names, directory paths, port numbers, and so on. The POWDER grouping vocabulary can be easily extended by new elements, defined via GRDDL transformation, which build upon the elements defined by POWDER. As examples, in Sections 3.1 and 3.2 we show how other methods of defining IRI sets that may suit particular situations can be transformed into POWDER-BASE.

Furthermore, there is no fundamental reason to constrain the domain of POWDER descriptions to HTTP IRIs, so there should not be unnecessary constraints on how the protocol works. In other words, the domain of grouping extensions does not need to be HTTP IRIs, but may be any kind of IRIs. As an example, in Section 3.3 we show such an extension for ISAN numbers.

It should be noted that the treatment of non-HTTP IRIs is one of the basic motivations behind the two-step GRDDL transform from POWDER to POWDER-BASE to POWDER-S, outlined in
Section 1.4 and fully specified in the Formal Semantics document [FORMAL]. If POWDER were rendered into POWDER-S in a single direct transform, the only XML language from which to derive extensions would be POWDER, which would oblige POWDER extensions to include HTTP-specific IRI restrictions such as includehosts, even if they are meaningless for the domain of the extension.

In the intermediate POWDER-BASE language, on the other hand, all HTTP-specific elements have been rendered as regular expressions, using the includeregex and excluderegex IRI restrictions, as POWDER-BASE only requires that these two restrictions are supported. Developers of non-HTTP extensions and tools are advised to use POWDER-BASE to derive their extension from, instead of POWDER, as this relieves them of the obligation to also implement the HTTP-specific IRI restrictions in their tools.

XML elements suitable for defining sets of URIs or IRIs from schemes other than HTTP may be created
and a GRDDL transform defined that renders such IRI sets in POWDER-BASE. This is an generic extension mechanism since a conformant POWDER Processor, as defined in the Description Resources document [DR], MUST be able to process POWDER-BASE. For clarity: POWDER-BASE is not a separate encoding of POWDER — it is all done in the wdr namespace — merely a restricted form of POWDER that just has the two possible child elements of iriset.

Developers of POWDER tools MAY directly implement extensions they know about, and MAY include support for
transformation technologies such as XSLT so that unknown extensions can be processed.

3.1 Extension Example: Custom IRI Patterns

As an example of a service-specific extension, consider a service which uses unix shell wildcards instead of regular expressions, so that www.example.org/* means "all the resources on www.example.org fetched using HTTP." Such a system is easily used within an IRI set, only requiring the definition of a near copy of the POWDER schema [WDR] with a single IRI constraint shell:includepattern as child element of its IRI set element (good practice when defining shell:includepattern would be to also define shell:excludepattern).

A publisher of a document using shell:includepattern SHOULD define a GRDDL transform
that will generate a POWDER-BASE document as shown the example below.

Example 3-1 An IRI set definition using a custom IRI pattern and the corresponding POWDER-BASE definition.

Note that the custom IRI pattern SHOULD NOT be used in a document with its root element in the POWDER namespace since the only valid child elements of the iriset element within a POWDER document are those defined in this document.

3.2 Extension Example: Custom Site Structure

Many content providers serve dynamic content stored in a database, so that IRIs express queries to that database. This kind of IRI will have certain structure but this is typically neither obvious nor easily human-interpreted.

As an example, consider sport.example.com, a sports news site, where IRIs look like the one shown in Example 3-2. The adopted scheme is systematic so that sport=2&countryID=16 provides a front page with news about Greek basketball and links to various Greek basketball leagues, sport=3&countryID=16 a front page about Greek volleyball, etc.

Example 3-2 Sample IRI from site serving dynamic content. sport=1 stands for football and countryID=16 stands for Greece.

http://sport.example.com/matches.asp?sport=1&countryID=16&champID=2

A POWDER document providing metadata about this Web site would have to use regular expression matching with explicit reference to the numerical values in the country and sport fields of the query. This process is error-prone, and requires extensive changes if the underlying database schema is modified or extended.

As an alternative, the site developer may provide a POWDER-like scheme that abstracts away from the specific database fields to allow reference to sports and countries, as shown in Example 3-3. Description Resource authors can then use the properties in this extension to generate POWDER-BASE documents that are valid even if the site schema is modified, as long as the site developer updates the relevant transformations.

Example 3-3 An IRI set definition using site-specific extensions and the equivalent definition using standard POWDER-BASE vocabulary.

3.3 Extension Example: ISAN

The International Standard Audiovisual Number [ISAN1] is a globally-unique, centrally managed and permanent numbering system for the identification of audiovisual works and versions. Following ISO 15706 [ISAN3], [ISAN3-2], the ISAN numbers are written as 24 bit hexadecimal digits in the following format [ISAN2].

-----root-----

episode

-version-

ISAN

1881-66C7-3420

-

0000

-7-

9F3A-0245

-U

The root segment of an ISAN number is assigned to a core work. When the core work is a serial, episodes are identified with a non null episode segment. Versions are assigned in the version segment and refer to changes in the audiovisual content, being a different language or soundtrack, subtitles, editions, promotional trailers, and so on.

Since ISAN numbers are URNs [URN], and hence IRIs of the urn: scheme [URIS], a vocabulary can readily be defined to allow IRI Sets to be defined based on ISAN numbers. The terms might be along the lines of:

includeRoots — the value of which would be a white space separated of hexadecimal digits and hyphens that would be matched against the first three blocks in the ISAN number.

includeEpisodes — a white space separated list of hexadecimal digits and hyphens that would be matched against the 4th block of 4 digits in the ISAN number.

includeVersions — a white space separated list of hexadecimal digits and hyphens that would be matched against the 5th and 6th blocks of 4 digits in the ISAN number.

The set of all audio visual resources that relate to two particular works might then be defined as shown in Example 3-4.

Example 3-4: An IRI set definition using an ISAN number pattern and the corresponding definition using standard POWDER vocabulary

4 Conformance Criteria

An IRI set definition is a Conformant IRI set definition if it adheres to the specification described in this document.

More precisely:

An IRI set definition is an XML Conformant IRI Set Definition if it contains at least one of the child elements set out in Section 2 and adheres to the cardinality rules stated. These are summarized in Appendix A.

Space separated lists of strings in an IRI set definition adhere to the rules described in Section 1.

IRI strings in an IRI set definition adhere to the rules described in Section 2.1.3.

Extensions to IRI set definitions adhere to the rules described in Section 3.

The Unicode Consortium. The Unicode Standard, Version 4. ISBN 0-321-18578-1, as updated from time to time by the publication of new versions. The latest version of Unicode and additional information on versions of the standard and of the Unicode Character Database is available at http://www.unicode.org/unicode/standard/versions/.

6 Acknowledgments

The editors duly acknowledge the earlier work in this area carried out by Jo Rabin.
Jeremy Carroll and David Booth developed the operational and formal semantics model
which was further developed by Stasinos Konstantopoulos. Eric Prud'hommeaux, Addison Phillips and Thomas Roessler
made significant contributions to the development of the canonicalization section. The editors gratefully
acknowledge the further contributions made by Régis Flad of ISANIA and members of the POWDER Working Group.

Resolution of open question on choice of Regular Expression syntax. Now use XML Schema REs as modified by XPath/XQuery for consistency with other W3C work - the syntax more than meets POWDER's requirements. Data type to be defined in POWDER's own XML Schema

Added hyperlinks to the first mention of each Class and property, pointing to its entry in the vocabulary document

Removed includeUserInfo and includeFragments properties since these are not strictly part of HTTP, the former can cause security issues, especially when written as username:password, and grouping by fragments is very vague since there is no sure way to define the end of a fragment.

The document has been updated throughout the reflect the introduction of the tw-stage GRDDL transform from POWDER to POWDER-BASE to POWDER-S.

Ports are now handled as strings (not as numbers) so that port ranges are no longer supported by POWDER

includeexactqueries and excludeexactqueries deleted

Canonicalization and data encoding sections updated following comments from Thomas Roeseller, Eric Prud'hommeaux. Text now advocates a 'Best Effort' approach tending to false negatives, rather than attempting to define a comprehensive approach to IRI canonicalization.

Grouping by IP address and CIDR block - this has been dropped, largely to mainatin the simplicity of POWDER-BASE and POWDER-S. Deriving a regular expression or some other processing rules from a CIDR block is very cumbersome. Section replaced with short text explaining that IP addresses will be tteated as strings.

Section on redirection deleted. DR doc will mention issue of redirection in the context of the POWDER Processor. E-mail exchange with Alan Ruttenberg and others provided important insight

Section 3 (extension mechanism) tidied up.

Slight change to the regular expression given in section 2.1 (the \/\/ moved to before the first ?)

Words 'of resources' removed from introduction of OWL Class in POWDER-S taking the place of an IRI set

Wording added to make it explicit that the processing works only for vaild IRIs following comments made on the member-only list

Section 1.2 amended to ensure that any user info or fragment in a candidate resource's IRI is not lost during processing.
Regular expressions in Table 4 and quoted elsewhere in the document updated accordingly. This follows
an e-mail exchange concerning a semantic application..

Line added to Section 2.1.3.3 to advise IRI set authors to specifically exclude any IRIs that they know may lead to false positives.

Cardinality of in/excludequerycontainsbrought into line with other IRI constraints so they can only occur 0 or 1 time.

Added a “Conformance Criteria” section, before the references, referring to the conformance criteria defined in the Description Resources document. The “References”, “Acknowledgements”, and “Change Log” sections, as well as the related subsections, have been re-numbered accordingly.