Erik Wilde on Services and APIs

139 posts categorized "Information Architecture"

Wednesday, November 02, 2016

Recently, W3C published the first working draft of a protocol called PubSub. It is on the W3C TR track and thus intended to become a W3C recommendation. While the name is new, it is the protocol that previously has been know as PubSubHubbub (PuSH). Before Adopting it, W3C rightly assumed that this name may not be such a great naming choice. PubSub also may not be such a great naming choice, but that's a different issue that was discussed in a previous post.

PubSub is a protocol for a publish/subscribe mechanism, but instead of covering all aspects of this picture, it only specifies the interactions with the PubSub hub. The high level picture looks like this:

Some service publishes a topic, which is some sort of feed or stream of entries about that topic. Typically, on the web such a topic would be consumed in a polling fashion, with clients polling for updates, and thus running into the typical problems of polling (balancing polling speed and update speed for the best combination of resource consumption and delay).

A consumer may be interested in receiving pushed topic updates instead. Such a service may be advertised by the topic publisher (linking to the hub that provides a push service), or the consumer may learn about the hub through other means.

The consumer subscribes to the topic at the hub, with the subscription being a combination of the subscribed topic, and a 4. callback URI* where the hub is supposed to push updates. The consumer has to verify the subscription by responding to a challenge sent by the hub.

When the hub learns about a topic update (either by the topic publisher notifying the hub, or by other means such as through polling), it pushes the update to all subscribers of the topic. This happens in the form of HTTP requests, which are initiated by the hub, and use the registered callback URIs.

Subscribers may or may not be online when the hub attempts to push updates. It is up to the hub to implement a strategy that probably will involve some retries, but also some condition under which failed delivery attempts of updates are considered as permanent failures.

Generally speaking, this picture depends a lot on the reachability of the consumers. In order to make this more manageable over time, the assumption is that subscriptions are only valid for a certain time, and then have to be renewed. This allows hubs to manage subscriptions in a more efficient manner.

One aspect that is not covered by the PubSub specification is the actual media types of topics and pushed update messages. Historically, PuSH started with specifically supporting Atom, so the assumption was that both topics and push messages would be Atom feeds.

(The nice side-effect of this was that by using "Atom Tombstones", PuSH was a nicely simple sync protocol, covering the creation, modification, and deletion of topic entries.)

PuSH then moved away from supporting a specific media type, and the same is true for PubSub. It simply says that the media type of the published topic and the media type of the push messages must be the same.

In its first draft, PubSub is a bit vague about how exactly topic updates and push messages are related, as it allows for "diffing" without saying what this is supposed to mean. It also is not entirely clear how "fat push" and "thin push" are covered, since in theory it would be possible to have hubs that translate between these two kinds either way (possibly depending on the preferences of subscribers).

Being a first public working draft, PubSub still needs a bit of work to be as well-defined and stable as it should be for a Web-scale protocol. If you are interested in Web-scale and decentralized PubSub mechanisms, now is a good time to look at "W3C PubSub" and maybe even engage in the ongoing community discussion.

Tuesday, October 18, 2016

Being faster at designing and delivering new experiences and products has become one of the main rallying cries of "digital transformation". As a modern organization, you have to stop struggling with the IT side of your goals (or create artificial dichotomies such as "bimodal IT"), and embrace a culture that allows you to move quicker as a business, supported by business-aligned IT.

It is good to be more nimble when it comes to developing new products, which is what Microservice Architecture (MSA) allows you to do. But it also is important to understand that in many cases, customers will interact with more than one product, and thus some form of cross-product quality control is necessary. The term that is increasingly being used for this is Customer Experience (CX), and it is distinctly different from the product-focused perspective of User Experience (UX).

This piece is about the difference of UX and CX, because these are established concepts when it comes to user/customer-facing products. It is worth noting, however, that in the API space, Developer Experience (DX) for APIs and API landscapes follow the same general pattern, and may encounter the same problems. DX in the API space is the equivalent of UX in the UI space, but there is not (yet) a term for "API CX". It seems that the more fine-granular API landscapes are becoming, the more it might necessary to think about "DX across API products". Let us look at a UX/CX scenario as an illustration.

One example is the current crop of native apps. They often are functionally limited, which is understandable because replicating the complete web-based product portfolio of an organization often is not possible or desirable in a single mobile app.

Mobile app product teams often "solve" this problems by sprinkling the native app with links to web-based products, providing these links to the web-based products in an effort to make the missing functionality at least accessible. This is a good idea in principle, but often creates the following CX:

Using the native app (which might have a tested and coherent UI/UX), a customer wants to accomplish something that is not supported in the native app. The app launches a web link, either in an embedded browser, or in the mobile device's regular browser.

The user is taken to a different product, the web app. This often has a different UI/UX approach, and increasingly has a mobile-specific UI/UX. This mobile web app often also is limited in functionality. Assuming the user wants to do something outside of the mobile web app's capabilities, they are now left to search for the "desktop version", which sometimes is almost impossible to find or launch on mobile devices.

Assuming the user found a way to get to the desktop version, they are now in a third UI/UX, often with impossibly small characters and heavily loaded with scripted frameworks that are not working very well on mobile devices.

This process of trying to interact with an organization's products is not that unusual. What's special about it is that is spans three products, and many organizations do not have the right setup to capture and evaluate this CX. Instead, all three product teams (native mobile app, mobile web app, desktop web app) do UX testing specifically for their product, and consequently report great scores for their UX tests.

This problem of a mismatch between a product scope and the scope of what a user wants to do happens for UI and well as for API scenarios. And it is not a problem in itself: It is smarter to avoid integrating everything, because then we are back to the monolith again. However, still having a perspective that's big enough to go across products, and using the right approach to improve CX for the entire product portfolio, that is something that many organizations are still struggling with. It is also something that will increasingly be a good measurement for maturity along the path of digital transformation.

It is a challenge for something as product-focused as MSA to come up with a good story for this cross-product optimization. This may be one of the areas where we will see new practices and patterns emerging, with organizations trying to find the best balance of developing new and engaging products, while still having an eye on the bigger CX picture outlined above.

Personally, I am wondering when the idea of "cross-API DX" will at least be seen as important enough to get its own label. It took a while for "CX" to become a well-known concepts, so maybe it will still take a few years before API landscapes and the ease of which they can be utilized have reached the point where these issues are looked at systematically, and get their own names and practices.

The idea of profiles originates in the idea for safely evolving formats and services, as explained in more detail in "Patterns for Robust Extensibility". The assumption is to start from a meaningful core model, which is an extensible format or service. A profile then identifies one specific way of extending it, in the sense that the profile-identified extension still adheres to the core semantics, but adds constraints and/or extensions that go beyond it. One important aspect of profiles is that they must always be permissible to ignore the profile and just use the core semantics, and still something useful can happen.

One example used in RFC 6906 are podcasts, i.e. feeds for multimedia content. While you can look at these in a regular (i.e., non-podcast-aware) feed reader and get useful information out of them, a podcast-aware client can be more helpful and for example display available media metadata such as media running time.

An argument can be made (and has been made in a variety of ways) that because a podcast is a feed, no profile identification is necessary. Instead, if a client supports podcasts, then it can simply look at a feed and figure out whether it satisfies all the additional podcast constraints, and if it does, then it can be treated as a podcast, without being labeled as one. This is what in programming is called duck typing: If it walks like a duck and quacks like a duck, then in all likelihood it is a duck.

The duck typing argument can be applied to profiles: If a feed can be extended to carry all podcast-specific data, and still be treated as a regular feed, then why go through the additional effort of calling it a "podcast" and making it identifiable?

There is nothing inherently wrong with that argument, because in a well-designed ecosystem of a meaningful and extensible core formats, and existing extensions to it, the evolution of producers and consumers is loosely coupled (that is the purpose of the whole enterprise after all), and everything can be treated at runtime as it is encountered.

So what are the arguments for going beyond that and to use profiles and related mechanisms? There are two main reasons:

If applications encounter extensions, then how do they find out what those mean, if they do not have built-in knowledge of them? One approach to do this is by using a registry, where extensions can be registered along with some metadata. At the minimum (and often as the only thing), that metadata links to available documentation, so that any extension that is encountered can be traced back to the place where it is defined and explained. This usually is not done at runtime, but is a robust mechanism to be able to track the ongoing evolution of the extension ecosystem, and update applications as required.

Going one step further: If you define a coherent and meaningful set of extensions, then you might want to give this set a name and thus make it identifiable. The reason for this is quite simple: If you want to have meaningful conversations about what to produce or consume, then you must be able to identify it. By creating a "property bag" and calling it "podcast" (and let's say also make it self-describing via a profile URI), you can talk about feeds being podcasts, for example in conversations where you can then say "I can deal with a variety of media feed formats, but prefer to use the podcast variety".

Essentially, only when you are able to call a duck a "duck" can you ask somebody to please hand you a duck. This might or might not be something that matters in your service scenario. But if it does, then going beyond the duck typing approach and creating a self-describing label for a "duck" may be useful. And in that case, profiles as defined in RFC 6906 might be a useful way of doing this instead of having to reinvent that particular wheel.

Monday, April 25, 2016

Extensibility is key to creating APIs that not only work initially, but that also can evolve in terms of being appropriated for different contexts, and/or evolving over time. Extensibility is not the only pattern that robust and evolvable APIs should follow, but a necessary and important one.

In the interest of focusing on patterns and best practices, this post describes general requirements for an API to be robustly extensible, and will not make assumptions about a particular metamodel that's being used. In the old days, XML would have been a popular choice, and schema languages such as XSD or RELAX NG might have been used to (partially) define the extensibility model. Nowadays, JSON is a more likely choice, and since (so far) no schema language has gained widespread popularity, API designers are more on their own. In RDF land, while there are "schemas", they are not good at capturing the things that matter for extensibility, but the new SHACL language (currently under development) may change that.

Why not absolutely necessary for robust extensibility, it is assumed that the API uses hypermedia (one of the core constraints of the REST style), which means that extensibility not only applies to extending individual representations, but navigational affordances as well, so that for example additional links can be added without disrupting the application flow of those clients that do not know those "extensions of the navigable hypermedia space".

While not strictly necessary, it can be helpful to make extensions discoverable. This is what RFC 6906 calls "profiles", and this post can be seen as an illustration of how to use profiles for API and representation design. It is not necessary to make extensions discoverable through the profile mechanism, but the constraints explained in the following subsection are the exact constraints that RFC 6906 profiles must follow, and making them discoverable can be useful.

1. Meaningful Core Semantics

It should be possible to build meaningful applications using just the core semantics. More importantly, the core semantics must always apply and remain unaffected by any extensions. Only this way it is possible to have independent evolution of publishers and consumers (Mark Nottingham's "HTTP API Complexity" provides a good overview of possible scenarios).

1.1 Meaningful Core Clients

It should be possible to build meaningful clients just using the core semantics. The reason for that is simply that the core semantics are the minimal fallback semantics (see following subsection), and if these are so minimal that it's hard to imagine how anything useful could be done with them, then it means that the overlap between the envisioned extensions may be too small, and that a more focused set of use cases should be tackled.

1.2 Reasonable Fallback Behavior

It always must be safe for either side of a conversation to fallback to default behavior, meaning that it always must be possible to safely ignore extensions. Clients not knowing extensions simply ignore them, which is why this strategy sometimes is referred to as a "must ignore" policy.

The opposite, a "must understand" policy, means that extensions either by default are or can be labeled as mandatory, meaning that essentially they create a rift in the ecosystem of implementations supporting them, and those that don't. For organically growing ecosystems of implementations, this kind of rift has a lot of negative side-effects, most importantly it is hurting interoperability, and crippling the network effect.

2. Well-Defined Extension Points

Extensibility can be achieved in different ways, and to a large degree, the underlying metamodel of a representation already makes some ways natural and easy-to-achieve, and others rather unintuitive and not a natural choice. If an API uses a schema language (which also is more natural for some metamodels than for others), then the facilities of that schema language very likely decide on how extension points for a model are designed.

Regardless of the specifics (which metamodel is being used and if/how schema languages are used), the most important aspect is that it is clearly documented in the API which extension points it has. A good way to do that is to clearly spell out where and how exactly extensions can appear, and to rule out any other ways of adding extensions. As described in the following section, it also is necessary to make it clear what to do when "extensions" do not follow these rules.

It is very helpful for any API to have a set of test cases that test the extension points, both in terms of adding test cases that leverage all possible extension points, as well as adding some that do not follow the extension rules.

The reason why extension points should be well-defined is that this makes it possible for implementations to know what to expect, so that they discover extensions when present, and may even report them to higher application layers. For example, when writing a library for an API, then that library ideally should not just handle the core API, but should also support all extension points and at the very least report when extensions are present.

Well-defined and well-documented extension points also have the benefit of giving extension designers clear guidelines how to extend the core format, creating a more consistent landscape than if no such guidance was available.

3. Well-Defined Processing Model

While extension points are necessary, it is equally necessary to define what implementations should do about them. This is what a processing model is all about: Telling implementers how their implementations have to behave when encountering extensions.

One part of that are the "must ignore" and "must understand" policies mentioned already. Those are two different models of how implementations are supposed to behave when encountering extensions. The "must ignore" policy is the better way to go in a loosely coupled ecosystem, because otherwise every extension tightly couples those that support it.

If "must ignore" is picked, then a logical conclusion is that it must be disallowed that extensions change any of the core semantics (as mentioned already). Apart from just requiring to ignore extensions, in some cases it may be possible to have processing models for them, such as the HTML rule that unknown elements must be ignored, but their contents must be rendered. However, such a rule may be harder to define for the machine-processing in APIs.

As mentioned above, the processing model is what tells implementers not just where to expect extensions, but also what to do about them. In generic implementations (such as libraries), this may just mean to make those discoverable and accessible to applications, which can then figure out if they know the extension, and if so, how to process it.

In summary, having meaningful core semantics, well-defined extension points, and a well-defined processing model makes it more likely that an API or a representation can serve as a robust foundation for an evolving ecosystem of extensions. In a follow-up post, I will discuss the different models how to manage those extensions, describing different models such as complete decentralization, registry-based models for easier discovery, and completely centralized models.

Tuesday, August 04, 2015

It is generally considered to be good practice to make URIs (the identifiers of resources on the web) as stable as possible. However, there are many things on the Web that will not last forever; many of those by design. Some of those even disappear according to a schedule, such as an archive that stores resources only for a limited period of time, or transient services such as URI shorteners that may only provide redirection services for a fixed period of time.

A little while ago in a thread on apps-discuss it was discussed that a mechanism for exposing this information could be made available through an HTTP header field.

It may be interesting for clients to know that some resources will become unavailable at a certain point in time. Clients can either inform their users, or maybe adjust their application logic. The newly proposed HTTP Sunset header field takes a very simple form to convey this information in an HTTP response:

Sunset: Sat, 31 Dec 2016 23:59:59 GMT

A client seeing such a header field in a response may conclude that it is likely that the resource will become unavailable at that point in time. It may start serving 404, 410, or some 3xx code, depending on how the service is designed (it is conceivable that they report 451, if they have been taken down for legal reasons).

And how does all of this relate to caching? Not at all, actually. The important thing to understand about Sunset is that it is all about resource expiration, and not resource state expiration. HTTP caching tells you how to hold on to resource state so that you can reliably reuse representations. Sunset tells you when a URI might go out of business, but tells you nothing about if and when it might change state before doing so. Which means that even if a resource advertises a Sunset, regular caching information and methods may still apply and have the same HTTP semantics as always.

Location has quickly moved into the mainstream of the (mobile) Web and it continues to be a strong driver of research activities, addressing a wide range of Web-related topics, including search, retrieval, mining, extraction, analytics, mobility, services, and systems. After the initial boost and consolidation of approaches based on the simple use of geospatial coordinates, we now see an increasing demand for more sophisticated systems, stronger retrieval, mining, and analytics solutions, and more powerful semantics. Location is playing a key role as a context factor for users, but also as the implicit or explicit place of resources and people. It also is an important factor in mobile and geo-social applications and is driving geospatially-aware Web data mining.

Following the successful LocWeb workshops in 2008, 2009, 2010, and 2014, LocWeb 2015 will continue this workshop series at the intersection of location-based services and Web architecture. Its focus lies in Web-scale systems and services facilitating location-aware information access. The location topic is understood as a cross-cutting issue equally concerning Web information retrieval, semantics and standards, and Web-scale systems and services.

The workshop is expected to establish a common, integrated venue where the location aspect can be discussed in depth within an interested community. We aim for a highly interactive, collaborative full day workshop with ample room for discussion and demos that will explore and advance the geospatial topic in its various relevant areas. New application areas for Web architecture, such as the Internet of Things (IoT) and the Web of Things (WoT) mean that there will be increasingly rich and large sets of resources for which location is highly relevant. We expect the workshop to further the integration of the geospatial dimension into the Web and promote challenging research questions.

LocWeb solicits submission under the main theme of Web-scale Location-Aware Information Access. Subtopics include geospatial semantics, systems, and standards; large-scale geospatial and geo-social ecosystems; mobility; location in the Web of Things; and mining and searching geospatial data on the Web. The workshop encourages interdisciplinary perspectives and work describing Web-mediated or Web-scale approaches that build on reliable foundations, and that thoroughly understand and embrace the geospatial dimension. We are also interested in work tying into ongoing W3C activities such as Web of Things, Social Web, Data Activity, and Geolocation WG.

Topics of Interest

Location-Aware Information Access

Location-Aware Web-Scale Systems and Services

Location in the Web of Things

Large-scale Geospatial Ecosystems

Standards for Location and Mobility Data

Location Semantics

Modeling Location and Location Interaction

Geo-Social Media and Systems

Location-Based Social Networks

Geospatial Web Search and Mining

Visual Analytics of Geospatial Data on the Web

Location-Based Recommendation

Geo-Crowdsourcing

Mobile Search and Recommendation

Submission Instructions

We solicit full papers of up to 6 pages and short papers of up to 3 pages describing work-in-progress or early results. Authors are invited to submit original, unpublished research that is not being considered for publication in any other forum.

Workshop submissions will be evaluated based on the quality of the work, originality, match to the workshop themes, technical merit, and their potential to inspire interesting discussions. The review process is single blind.

Accepted workshop papers will be published in the WWW companion proceedings and available from the ACM Digital Library. This may be regarded as prior publications by other conferences or journals.

For inclusion in the proceedings, at least one author of the accepted paper has to register for the conference. At the time of submission of the final camera-ready copy, authors will have to indicate the already registered person for that publication.

Presenters are encouraged to bring demos to the workshop to facilitate discussion.

Monday, September 01, 2014

With the iWatch hype reaching all-time highs, it seems clear that the landscape of wearables will change quite dramatically over the next year or so. But as usual, it's much harder to predict how things will change, instead of just saying that they will change.

In order to find out at least for myself, I recently started playing around with more services and devices than already usual. I always have been a e-gadget addict, but now I am reaching new heights: On yesterday's long trail run in the Sierra, I realized that I was carrying around 4 GPS devices: a Garmin 910XT GPS watch, my iPhone, my GPS-enabled camera, and my SPOT satellite tracker. That's probably a bit more than needed, but the only ones I actually use are the Garmin (displaying basic data such as moving time and distance), and the camera (and I never enable its GPS, because it takes way too long to get a good GPS fix). The SPOT is for emergencies only, and the iPhone is just for running the Moves app.

How did it get that bad? About a month ago I started using a Basis watch, mostly to start collecting data for our research activities around wearables and personal fitness goals. I stopped wearing it pretty soon because the band causes irritated skin, and the watch turned out not to be waterproof. I also disliked the incredibly dim display. But even during the short time I used the watch, I discovered that what I found most intriguing was something that I never would have coinsidered.

The Basis is pretty smart at figuring out what a person is doing. Not in many different ways, but good enough to distinguish walking, running, and cycling. I really liked how when I was getting around town, I could simply look at the watch and it would tell me that I had been walking or cycling for however-many minutes.

Since the Basis hardware had various issues, I recently switched to the Moves app. It does everything that I liked in the Basis. It does not give me heart rate or the other two nonsense measurements (perspiration and skin temperature) that the Basis records. But more importantly, it has GPS and thus is much better at being a meaningful diary (they also team up with Foursquare, so the app will suggest meaningful place names for many places).

A day in town may look like the picture shown here. Have some coffee in the morning, then stay at the office most of the day, pick up the car from the mechanic in the afternoon, and get some pizza in the evening. Moves has a great UI to make it as easy as possible to fix some of things that the app may get wrong, so ending up with such a geo-diary takes almost no effort. But it still means that I have to go to the Moves app and clean things up. Which at this point, I am willing to do because it's for the greater good of research, but this is where wearables (remember? this is where i got started...) enter the picture.

What a good wearable could add here are two main things that right now are missing from the Moves app:

I am not a big fan of carrying my phone everywhere. If an iWatch/iPhone app combo is done well (more on that later in a more technical post on wearables, WoT, and REST), then all I need to carry is the wearable, and it'll sync as soon as it gets in touch with the mothership again (such as requesting place names from Foursquare, and so on). What's most important here is a wearable app that's very specifically designed to work well when not paired with a phone.

Instant feedback on the wearable a bit like the Basis: Tell me that I have been walking/cycling/driving for some amount of time, and let me fix it if the wearable app guessed wrong (Basis does not allow fixing at all, Moves takes a bit of work to get to the UI place where I can do it).

In the end, what users end up with is a very personal, very contextual way of looking at their history. Of course this then could be used for Big-Data-style sensemaking, but that's a different issue (more about this in a later post). Mostly, it's a great way of building up a diary of places and tracks. And with a convenient feedback unit on your wrist, it's easy to see how it's building up, to intervene with direct feedback/corrections, and to not have to carry around the phone everywhere.

One of the interesting questions is where this data is being kept, and who gets access to it. I'd prefer this to be something that I own and control, which is not the case with Moves. I would be perfectly happy to pay money for this, but I guess as usual, that will not even be an option, and you end up paying with giving up privacy as the only accepted currency.

Being a major location geek, I am excited about the prospect of a better and more flexible wearable, and the iWatch may turn out to be for wearable space what the iPhone was for the smartphone space (even though the latter was probably even more trailblazing, by redefining so many aspects of the category). I have been using my GPS for about 5 years now, but only for sport activities. The Strava heatmap shown here is the result of many runs trying to cover all Berkeley Hills roads (blue means less coverage, read means more runs along the same road). Building up a similar personal geography for everyday activity may not just be fun, it also can be useful in terms of better understanding one's life, and figuring out things such as how to better get around town to get things done.

As said initially, I guess it's safe to say that the iWatch will change the landscape of wearables. Personally, my hope is that it focuses on being a good platform, and not so much on getting one single application right. That may be the best and maybe even the only way to avoid the current fate of most wearables, which is that they are novelties that early adopters get and play around bit, and then they get discarded.

Current wearables get discared quickly because they are too narrowly focused on doing just one thing, and there is no platform thinking backing them up. If there is one single thing that Apple is great at, it's platform thinking, so the iWatch very well could be another game changer, 7 years after the iPhone changed the smartphone category.

Thursday, April 17, 2014

Data Science is a subject that is mentioned a lot, and many people are saying that they're doing it. but the term often seems to take on different meanings, and depending on who's referring to it, it may mean surprisingly different things. here is a modest attempt to identify two intertwined but still moderately clearly separated sides of Data Science.

there of course is substance behind the term, even though it may have become a bit obscured by the recent inflationary use of the term. with pretty much every term that has become hype, it can make sense to first start with some basic definition before using it, so that at least in the context of a conversation, it is relatively clear what the term refers to.

when thinking about a good definition for Data Science, and how it can be used in a somewhat well-defined way, it seems to make sense to separate two sides or views of what Data Science encompasses. of course these two sides are related and connected, but they are fairly distinct, and it may make sense to clearly call out when only one of them is referred to (assuming that when Data Science is used without further qualification, it refers to both of them).

here is an attempt to identify these two sides, and maybe the labels could be a bit snappier, but i am mostly interested in getting the discussion started, and getting a better understanding of what others think Data Science is and isn't:

Vertical Data Science (VDS): this side refers to methods around making sense of raw data, combining data, and making sense of the combined data (this iterative process can be repeated any number of times). this side has a big overlap with Big Data, and in a corporate context, this is often referred to as Business Intelligence (BI). the main aspect here is that ownership and management of the data is in one hand, and the main task is to use these large amounts of data for sense-making.

taking the proverbial needle in the haystack, VDS is all about methods to find needles quickly, in very large haystacks, and to be able to search for things that may not even be needles, but fit some interesting patterns (finding something needly).

Horizontal Data Science (HDS): given that VDS allows us to expose BI services from one data owner/manager/needle-searcher, the question then is how to design a Service-Oriented Architecture (SOA) around these services. REST is often used as one way to implement this vision. in HDS, the implementation of a single VDS provider is opaque; what matters is that VDS providers expose data-driven services that can be used to implement sense-making across an ecosystem of a variety of those providers.

taking the proverbial needle in the haystack, HDS is all about methods how to search across haystacks, and how to provide an ecosystem of haystacks where people get access to haystacks, but where haystack owners can also control and limit access to them, so that for example certain privacy aspects can be protected.

while this separation of VDS and HDS may be rough and incomplete, it seems to be useful to explain some of the large differences in opinion when it comes to explaining that Data Science is and isn't. VDS is all about sense-making and processing, and is necessary to be able to deal with the deluge of data in many data-driven sciences. HDS is all about combining results and setting results in new contexts, based on the assumption that all participants in the larger science picture have the freedom to implement their VDS in which ever way they like.

i am certainly biased here, but it seems to me that more people are talking about VDS than about HDS. and that may be a reflection of the same picture that you find in data-driven corporate settings: while there are clearly identifiable stakeholders (such as product groups) that have to get the VDS side done, the incentives and the stakeholders to oversee HDS are less clearly defined, and this side of the picture thus sometimes gets lost.

i am wondering how many people would (a) agree that VDS and HDS indeed are two valid sides of the Data Science coin, and (b) if they see that one is more prominently mentioned and/or pushed that the other. personally, i think that both sides are essential: science needs good methods to work in data-driven scenarios, but it also needs a solid foundation how to robustly combine those scenarios, so that new studies and experiments can be easily based on previously disconnected datasets and results.

Bundling Functionality

one way of describing profiles is to look at them as capability bundles (i am just making this term up here), which are supposed to identify a specific way in which an extensible and open data format is used to accomplish a more specialized goal. specifically, the idea of a profile is not to just identify a single extension element or attribute.

instead, consider podcasts used as one example in the RFC. or, you could also think of OPDS, another feed-based content distribution architecture. in both cases, the specifications are using feeds as their foundation, i.e. you can read podcasts and OPDS feeds with a generic feed reader. this satisfies one of the requirements for scenarios that are a good fit for profiles: it should be possible to have meaningful applications using the services without knowing/using the profile. specifically, for this reason it would not be a proper use of profiles to say that podcasts or OPDS are profiles of RDF or XML, because you cannot meaningfully consume podcasts or OPDS just based on on parsing RDF or XML. you need knowledge of the concept of feeds to have something that minimally works (but of course a generic feed reader will not support any of the features that are specific to podcasts or OPDS).

the second requirement for using profiles in a useful way is that profiles should identify meaningful and self-contained capability bundles. both is true for podcasts and OPDS, which start from plain feeds and then add features and capabilities to them in order to address a specific set of use cases. don't give in to the temptation to make things too fine-granular and turn every single property/feature into a profile. this would lead to an inflation of profiles, and make working with them awkward and unnecessarily complicated.

The OO View

apart from the capability bundles concept, another useful way of thinking of them is in terms of the OO concept of a subclass. instances of subclasses can be handled in exactly the same way as instances of their superclass, but they have additional constraints, and/or have additional behavior. the same can be said of resources using profiles: they can be used according to their profile (i.e., as an instance of the subclass), or they can be used according to their media type (i.e., as an instance of their superclass), ignoring the fact that they follow a profile.

taking this view even further, you could say that in that OO view each superclass can have multiple subclasses (i.e., it is possible for a resource to specify more than one profile), but subclasses only have one superclass (because there onle is one media type in which a profile is used).

however, since the notion of a profile is fairly open, there is nothing that keeps profile designers from saying that their profile can be used with different media types. one obvious example could be RDF, which is an abstract model, and has a number of possible serializations (all identified by different media types). a profile could be based on RDF's abstract model, i.e. not be specific for any specific serialization, and then could be identified in different RDF media types through the same profile URI, making sure that the profile can be recognized in a way that is indepedent from the specific serialization.

Profile URIs

as defined by the RFC, profile URIs are purely identifiers: clients are expected to recognize them when they are used with profile-typed links or as profile media type parameters, and then deduce from that that a resource follows a certain profile. this means that profile URIs never have to be dereferenced, and thus could also use non-referencable URI schemes such as URNs.

however, as with other identifier-style URIs (XML namespace URIs come to mind), it is a good idea to use referencable HTTP URIs, and provide some documentation (or other helpful resources) at that URI. also, when choosing the profile URI, it might be a good idea to stick to the naming advice from the XML Namespaces section on Comparing URI References: Because of the risk of confusion between URIs that would be equivalent if dereferenced, the use of %-escaped characters in [...] names is strongly discouraged.

providing documentation at a profile URI is a good idea, but there is no standard what to provide there. it probably makes sense to follow the pattern of XML namespace URI pages such as the XSLT namespace, which links to a variety of helpful resources, so that developers accessing the URI (hopefully) find what they are looking for.

the idea of profiles is simple and fits into the architecture and limitations of today's standards and tooling. the hope is that by using profiles, resources on the web will become a little more self-describing, and it will be easier to properly design scenarios where services are extending others, but still want to maintain a working relationship with the service ecosystem they are building on.

personally, adding profile support to atom is the next step in the grand plan, but other than that, hopefully other representations and media types will start supporting profiles as well. it's very simple: mint a URI, ideally provide helpful information at that URI, and then start using this profile URI in your representations and maybe in media types as well. if you have any questions about how to best do this for a specific representation and/or media type, please feel free to comment or send me an email.

Saturday, February 02, 2013

is location a web concept (yet)? not really so far, even though of course there are many location-based services (LBS) out there. but many are trapped in specific services such as google maps or foursquare, often for business reasons, sometimes also for technical reasons. there still needs work to be done to turn location and LBS into first-class citizens on the web. as an illustration, this week brought two interesting developments:

garmin is one of the biggest makers of GPS devices. in the consumer market, they have been plagued by what i call the sony syndrome: really great hardware, but both the software and the service offerings backing it are mediocre at best, or just pathetic (other companies afflicted by this syndrome are suunto and motorola). case in point are the new online capable edge GPS devices. i have an edge 800 for my bike and always thought how cool it would be to pair it with strava and get alerts of upcoming segments. the new edge devices are now online capable by pairing them via bluetooth with your smartphone, but get this: the only site on the web they can talk to is garmin's own connect service. yes, that's apparently what garmin thinks online-capable means.

seeing something like this always is interesting to analyze: is it for business reasons, or because of a lack of vision? in this case it must be the latter, because garmin is not making money off of connect (which also happens to be one of the not-so-stellar services for sharing GPS-based activity data out there).

exhibit 2: the fabulous raceshape.com service recently launched a strava-based heatmap service, where users can create heatmaps of all of their activities. it's fascinating, because you can see hundreds of rides and runs on one simple map. the map image shown here is a screenshot of a full interactive version where raceshape has generated thousands of tiles which summarize three years of GPS track data.

now imagine a GPS device you can take with you, that at each and every turn can show you where you went before, and where you might see new things (or simply switch to the global heatmap to see where people in general go and don't go). by pulling the heatmap into a GPS device, you could do this and whatever else people come up with as ideas. you would have an open ecosystem for ideas around location, many people could build specialized LBS sites, and since garmin is not making any money with connect itself, they wouldn't lose money if people ignore connect's mediocre and limited services, and instead would subscribe to their favorite services. recent bird sightings. geocaching. places of historical interest. botanical guides. you name it, just let somebody build it, plug it into the ecosystem, and there's one more incentive to buy a garmin for a previously indifferent group of people.

what would it take to make this happen, apart from more forward-looking software and service strategies one the part of hardware specialists? an open architecture to build LBS that allow devices to dynamically subscribe to and pull data in from LBS in a standardized way. in our tiled feeds work, we have developed such an architecture, which interestingly is based on the model that raceshape is using now: tiles for various zoom levels to represent LBS features. check out our demo app with a couple of (hardcoded) sample LBS providers, and all of this is completely decentralized. the image shown here is using my iPhone locations (pulled from the phone before apple fixed this particular privacy snafu).

imagine being able to simple configure a couple of URIs as LBS providers, powering up the GPS, pairing it with the smartphone, and then the information you are interested in is shown right in front of you. you could even cache this when you know where you're going to be, because it's all REST-based and thus LBS tiles can be requested and cached on the device, if you're not going to be online all of the time.

my newest hope for a company getting it is leikr, the OSM-based GPS watch just funded through kickstarter. maybe if they have enough strategic vision to identify a competitive edge, they will recognize this opportunity that is still being left wide open by the incumbents. currently, they seem to plan to be pretty tightly coupled to endomondo, which to some extent is understandable (for the activity management part). however, enabling LBS to be freely pulled into the watch could be the great differentiator, and it remains to be seen which company in this space will first understand that services matter, ecosystems matter, and openness matters.