Microsub-spec

The Microsub spec provides a standardized way for clients to consume and interact with feeds collected by a server.

The Microsub server is responsible for managing the accounts you follow, retrieving updates from them, and the Microsub endpoint provides the feed entries in a normalized format for easy consumption by clients.

Per CC0, to the extent possible under law, the editor(s) and contributors have waived all copyright and related or neighboring rights to this work. In addition, as of 2018-02-18, the editor(s) and contributors (2017-04-09 onward) have made this specification available under the Open Web Foundation Agreement Version 1.0.

Design Goals

The goal of Microsub is to simplify the process of building a reader, since there are many moving parts when consuming external content.

In general, when subscribing to a feed, a reader should use WebSub if the feed is enabled with it, but may need to fall back to polling if not. Depending on the format of the feed, there can be many variations in the actual data available at the feed. For example, there are several different ways an h-entry can represent the author of the entry, described at authorship. There are also multiple ways a list of h-entrys can appear in an h-feed.

The role of the Microsub server is to normalize the data in the wild and turn it into a simpler format for displaying in clients. Clients should never have to second guess or doubt any data they receive from the Microsub server. The assumption is that the server has done all the verification and normalizing of the data, and it is ready to display to the user. The fewer checks and conditionals that clients have to write the better.

Endpoints

The Microsub endpoint is where the client will make all API requests. All API requests require authentication with a Bearer access token that the client needs to obtain. If the client does not have a preexisting relationship with the server, then the following method of discovery and authorization should be used to obtain an access token and discover the Microsub endpoint.

It is possible for a client to be pre-configured with a Microsub endpoint, or to use other methods of obtaining an access token if there is a preexisting relationship between the client and the server providing the Microsub endpoint.

Discovery

The client first performs discovery on the user's profile URL to find the Microsub endpoint and authorization endpoint. Given a user's profile URL, perform an HTTP GET request and look for either a <link rel="microsub"> or HTTP Link header with a rel value of microsub. Additionally, look for links with rel values authorization_endpoint and token_endpoint.

The Microsub endpoint URL MUST NOT contain a fragment, and MAY contain query string components. If the URL contains a query string, then any GET requests MUST properly append the additional parameters to the query string, and POST requests MUST NOT send the query string properties in the post body. e.g. making a GET request with the additional query string component "action=config" to the endpoint "/endpoints?type=microsub" would result in a URL of "/endpoints?type=microsub&action=config"

(Note: The client will likely want to also find the Micropub endpoint for the user so that the client can post replies and other interactions to the user's website.)

Authentication and Authorization

Channels

Channels are described by the following properties:

uid - a string representation of a user-specific unique ID for the channel. This uid will be unique for each user, but may be duplicated across different users. Some implementations will use constant strings such as "example", while others may use numeric database IDs such as "15029932", a random string like "1NYnmUVYR5qBVXYBzt", or a URL such as "http://user.example.com/channel/foo". The valid characters for a uid are any URL-safe character.

name - the display name for the channel. This may include any valid UTF-8 sequence. The client should use this name when displaying the name of the channel in the interface.

{
"uid": "indieweb",
"name": "IndieWeb"
}

Servers must always have a channel with the uid notifications, and must always have at least one other channel for a user.

Some actions may want to apply to every channel, so the uid of global is reserved for this purpose. Actions such as mute that want to mute a user across every channel use the channel uid of global.

Users

All users are identified by profile URLs, with some constraints. User profile URLs MUST use either the http or https scheme, and MAY contain path and query string components, and MUST NOT contain fragments.

Actions

All operations in Microsub are considered "actions", and are specified with a query string or form body parameter of action.

channels

search

preview

follow / unfollow

timeline

mute / unmute

block / unblock

Actions that operate within the context of a channel can accept a query string or form body parameter of channel specifying the uid of the channel to use.

Timelines

action=timeline

GET

Retrieve the entries in a given channel.

Parameters:

action=timeline

channel={uid}

after={cursor}

before={cursor}

The response will include a property items with an array of post objects. See below for documentation on the format of items.

Search

action=search

The "search" action exists to provide a UI for the server to respond with the full URL of possible things to subscribe to. For example, a user should not be expected to type the exact URL of a feed to subscribe to, but instead should be able to enter partial matches, e.g. entering aaronparecki.com should return the full URL of https://aaronparecki.com/.

Using the "search" action, a client can provide a single text field where the user can enter either partial URLs or even arbitrary search terms, and the server can reply with a list of URLs that can actually be subscribed to. This also provides the ability to have a confirmation step where users can see a preview of what they will be subscribing to before they actually do so.

If the search term is a URL or partial URL, the Microsub server SHOULD fetch the URL if not already known, and discover any feeds at that URL that can be subscribed to. The server may also return feeds that are already known that match the search term, for example if another user on the server has previously subscribed to a matching URL.

Searching for Content

TBD: implementing a search API for searching past posts that the Microsub server has indexed.

action=search

channel={uid}

query={term}

The presence of the "channel" parameter indicates to the server that the client wants to search for posts rather than search for feeds. The channel parameter can be set to an individual channel uid, or "global" to search across all channels.

Preview

action=preview

The "preview" action exists so that the client can display a preview of a URL to the user before the user wants to create a subscription for it. The preview should show as much about the URL as the server can determine, such as basic profile information about the user, and a few recent entries by the user. There should be no permanent side effects created by previewing a URL, and as much as possible, the URL being previewed should not be provided with identifying information of the user who is previewing the URL.

The response includes the list of items in the feed if available, in the same format as returned by the #Timelines API call.

When a request to the follow endpoint is made, the Microsub server registers the follow action, and begins delivering content at that URL into the channel. The Microsub server can subscribe to the target URL via any mechanism available, but most often will attempt a WebSub subscription for its HTML+Microformats, or Atom/RSS feed, and fall back to polling if that fails.

New entries at the followed URL will appear in the channel when fetched from the channel's timeline. The Microsub server may fetch existing entries in the feed and add them to the channel at its discretion.

The response is a JSON representation of the channel, the same as is returned when listing all feeds followed in a channel.

The Microsub server may remove all of this feed's items from the channel, or may leave them in place, at its discretion. If you are used to treating these channels as an IRC or Slack timeline, it would be more appropriate to leave the old items in the channel, just stop delivering new ones. However if you are more used to treating these channels as a Twitter or Facebook feed, then you may want the server to remove them from the channel.

TODO: Should there be another parameter for the client to specify whether to remove previous entries or leave them?

Muting

action=mute

Clients should provide a "mute" option in the interface. This allows the user to mute someone's profile, hiding all posts with the muted user's profile URL as the author from being displayed.

Muting users will cause all posts by the muted user to be hidden from display. The server MAY still store the posts internally, so that un-muting the user will cause past entries to appear again.

Any side effect at the server SHOULD NOT cause the muted user to know they have been muted. Muting users SHOULD NOT have any externally visible side effects.

For example, in the context of the Salmention spec, the server should still behave as if the muted user was not muted.

Unmute

To unmute a user, use action=unmute and provide the URL of the account to unmute. Unmuting an account that was previously not muted has no effect and should not be considered an error.

Blocking

action=block

Blocking users will cause all previous posts by the blocked user to be hidden or deleted, and future posts by that user should not be stored. Additionally, the server SHOULD NOT produce any content or side effect that would notify the blocked user about a post. It is acceptable for the blocked user to know they have been blocked.

For example, in the context of the Webmention spec, the server should not send webmentions even if the user mentions the blocked user in a post. In the context of the Salmention spec, the server should stop sending follow-up webmentions to the blocked user.

GET

action=block

channel={uid}

Retrieve the list of users that are blocked in the given channel.

TODO: document the response format

POST

action=block

channel={uid}

url={url}

Block a user in a channel, or with the uid global blocks the user across every channel.

Channels

action=channels

GET

action=channels

Retrieve the list of channels for the user.

The response will contain a channels property with the list of channel uids and names. The uid=notifications channel must always be the first in the list, as clients are expected to treat it separately and not show it in the channel list.

Servers should support tracking the read state of items in channels, and return the number of unread items when the channel list is queried. If the server does not support read state tracking, then the server must not return the unread property in the response.

POST

To create, update, or delete channels, the client sends a POST request with the channels action.

To create a channel, the client includes the name of the channel to create. The uid of the channel will be assigned by the server.

action=channels

name={channel name}

To update a channel, the client includes the uid of the channel to update in the channel parameter, and includes the new name of the channel. Changing the name of the channel MAY change the uid of the channel, and if it does, the server MUST return the new uid in the response.

action=channels

channel={uid}

name={channel name}

To delete a channel, the client includes an additional parameter method=delete. Note that the default and notifications channels can not be deleted.

action=channels

method=delete

channel={uid}

Both creating and updating a channel MUST return the uid and name properties for the channel.

Set Channel Order

Channels are ordered according to the user's preference. When returning the list of channels, the notifications channel must always be the first in the list, as clients are expected to treat it separately such as showing it as a separate icon, not in the main channel list.

To modify the order of channels, the client sends a POST request with a list of channel IDs in the new ordering.

Only the order of the channel IDs specified will be changed. While this command works equally well specifying two, three or more channels, this provides two primary methods for adjusting channel orders.

To move a channel up or down, the client can include the two IDs of adjacent channels in the new desired order.

The client can specify a new ordering for all the channels by providing the full list of channel IDs in the new order.

Order Algorithm

From the server's perspective, the below is a description of how to process the order command.

Given a list of channels with IDs:

[a b c d e f g h]

and a command to set the order of the following in the list: [d a c g]

Assign the items in the initial list a numeric index:

[a b c d e f g h]
[1 2 3 4 5 6 7 8]

Build a new map with the input items and their existing numeric order:

[
1 => a
3 => c
4 => d
7 => g
]

For each item in the input list in the given order, set the value in the map:

[
1 => d
3 => a
4 => c
7 => g
]

For each item in the new map, change the value at the corresponding numeric index of the list:

[d b a c e f g h]
[1 2 3 4 5 6 7 8]

Types of Feeds

The specific types and formats of feeds that can be followed is out of scope of Microsub. Instead, it's up to the Microsub server to support whichever feed formats it wishes. Typically, Microsub servers will prefer a Microformats 2 feed such as an h-feed or list of h-entrys, and will then fall back to finding an Atom or RSS feed. Other types of feeds may be supported, but clients should not make any assumptions about which formats are supported, and should make use of the "preview" action so that users have an indication of whether a subscription will succeed.

Objects

Posts

Posts are the basic object used in the API. Posts can be short status updates, photos, videos, podcast episodes, checkins, and many other content types. Post objects returned in the "items" array MUST be valid jf2 post objects.

If there are any items returned in the response, the server MUST return a "before" value that will retrieve items before all the returned items in the list.

If there are no items returned in the response, the server MUST NOT return a "before" value.

If there are additional items available that were not returned in the response, the server MUST return an "after" value that will retrieve the next page of items.

If there are no more items available, the server MUST NOT return an "after" value.

As far as the client is concerned, the "before" and "after" values are arbitrary strings. This allows the server to internally use whatever specific implementation is most appropriate for its backend technology. (Often this will be either a timestamp or a unique ID identifying the first and last items in the returned list.)

To make a timeline request for the next page of results, the client adds the "after=xxxxx" parameter to the query string. This allows easy navigation through the whole list of items in the channel.

While the user is reading the timeline, the client will likely also want to poll the timeline to see if new posts have been added since it was originally requested. Since not all Microsub servers will support streaming, the client needs an efficient way to poll for new items. The client adds "before=xxxxx" to the query string to request items that come before the first item returned in the previous request. This way the client can poll that until new items appear, and only the new items will be returned.

Example Paging Workflow

The user loads the client and makes a request for the timeline for the default channel:

/microsub?action=timeline

The server replies with the newest 20 items, and includes before=5a1713e55a171588 and after=5a1713e55a17136c

The user scrolls to the bottom and clicks "load more". The client makes a request for the next set of results:

/microsub?action=timeline&after=5a1713e55a17136c

The server replies with 3 more items, and does not include an "after" paging cursor in the response, indicating that there are no more items in the timeline.

Meanwhile, in the background, the client polls the timeline to find newer items by using the first "before" cursor that was returned in the initial request:

/microsub?action=timeline&before=5a1713e55a171588

The server replies with an empty items list and an empty paging object indicating there are no new items

After some interval, the client polls for new items again

/microsub?action=timeline&before=5a1713e55a171588

In the time between the two polls, there have been 25 new items added to the timeline, more than one page of results. The server replies with the newest 20 items list, and includes new before and after values, before=5a1724ad5a171599 and after=5a1722ea5a171280

Since the client sees there is an "after" cursor, it immediately fetches the next page of results using the original "before" and the new "after" value:

Limiting Results

Microsub servers SHOULD set a default limit on the number of items returned in lists. A reasonable default limit is 20 items. Microsub servers SHOULD support an additional query parameter limit which clients can use to indicate the requested limit of number items returned. Microsub servers MAY set an upper or lower bound on the values they accept for the limit, and MAY return a different number of items in the list than the client requets, for any reason. Clients should not expect the number of results returned to exactly match the number of results requested.

Authentication and Authorization Details

The client builds an IndieAuth authorization request URL at the authorization endpoint, and directs the user's browser there. In a native client, the client should use a system-native browser, rather than using a web view embedded in the application. See OAuth 2.0 for Native Apps for more details.

Build a URL with the following query parameters:

me={the user's profile URL} - the URL that the user entered at which the Microsub endpoint was found

response_type=code

client_id={the client's URL, e.g. its home page}

state={random state} - the client should generate a unique state value, and verify that it matches when the user is redirected

redirect_uri={the client's redirect URI} - for native apps, this may include a custom URL scheme

scope={requested scope} - a space-separated list of scopes that the client is requesting

Scopes

Microsub defines the following scopes:

read - this is the minimum scope clients should request. this allows clients to have read access to channels.

follow - allows the client to manage the following list

mute - allows the client to mute and unmute users

block - allows the client to block and unblock users

channels - allows the client to manage channels

Additionally, the client may request Micropub scopes, in order for the user to be able to reply or like posts from within the client.

create

update

delete

The recommended set of scopes to request is read follow mute block create, which enables a rich set of interaction on the client, while also protecting the security of the user by default.

The user will visit the authorization endpoint, and if they approve the request, their browser will be redirected back to the client's redirect URI with a code and state in the URL.

The me value returned MAY be different from the original me value input, but MUST have a matching host name. This enables support for multi-user websites, and allows the user's server to normalize profile URLs, e.g. it will always return https://aaronpk.example/ even if the user initially enters http://aaronpk.example.

The scopes returned MAY be different from what the client requested, based on whether the user choose to deny certain scopes, or grant additional scopes during the authorization request.

Design Considerations

Why a single endpoint instead of individual endpoints for each operation

Many similar APIs such as the Twitter API or Wordpress API use unique URLs for each type of operation: following, muting, fetching posts, etc. Microsub instead takes an RPC style approach, where all requests are made against a single endpoint, with the operation is specified with a query or form parameter.

This allows more flexibility in the design of the server, since the spec is not imposing a URL design on the server, each can choose a URL for the Microsub endpoint that makes sense for itself.

This also makes clients easier to write, since all requests are made against one base URL rather than needing to either keep track of a fixed URL pattern, or have a configurable URL pattern.