Frequently Asked Questions

General

How does Superfeedr deal with Encodings?

One of the other benefits of using Superfeedr is that the data pushed to you will always be UTF-8 and you won't have to deal with encoding issues. We will make sure we encode everything correctly into UTF-8 with the following rules : first we'll trust the HTTP headers of the content we poll, then, we'll look at the encoding of the XML documents (if applicable), finally we will try to guess the best encoding if both of the previous methods failed. If the document is not UTF8, then, we will encode to UTF-8.

How expensive is Superfeedr for subscribers?

If you use Superfeedr when subscribing to any of the hubs we host (like Tumblr, Posterous, Foxnews, Getglue... etc), it's free. You can get all that content by subscribing to their hubs, as you would do for any other hub. Technically, your code should not even be aware of the fact that this hub is using superfeedr :)

If you use Superfeedr as a subscriber for all your feeds, if you want to benefit from our normalization, from JSON, from our digest notification feature, or if you want to retrieve the past content of a feed, then, we will charge you for each notification we send. You can find details on this page. In any case, remember that user Superfeedr is always cheaper than building your own polling infrastructure!

Duplicate entries/notifications : why do they happen?

We do all we can to avoid these, but you should make sure your app deals with them. As you know we use a very distributed architecture, which means that even our datastores are replicated. There is always a risk that replication to "too slow" sometimes, and that 2 parsers might run almost at the same time and see the same entry twice.

However, if you see recurring duplicate entries for a given feed, you may want to check that you haven't subscribed twice to this feed, with different endpoints. We send notifications for each {feed url | endpoint} pair.

The "next fetch date" for some of my feeds seems inconsistent?

When you add a feed that we're not monitoring yet, we have to assigned it a next fetch date. To make sure this feed gets fetched as soon as possible, we assign it a date in the past (2012-01-15T22:25:05+00:00 to be precise). Hopefully, if you wait a few seconds, you should soon see a more accurate next fetch date.

How do you (and I) handle redirections in the feeds to which I subscribed?

Redirections should be transparent to you. When you give us a feed url, whether it is a permanent redirection, or a temporary redirection, we will follow it to get the final content. The only limitation is that we won't follow more than 5 redirections. You should make sure, however that you give us atomic (final) urls when you can.

Why do you call 15 minutes Real-time?

Because real-time in computer science is not synonymous of "instantaneous". Wikipedia says thatreal-time computing (RTC) is the study of hardware and software systems that are subject to a "real-time constraint"—i.e., operational deadlines from event to system response.
So, technically real-time means that we can guarantee delivery of new entries within 15 minutes of their publication.
Over time, our goal is to actually decrease this limit to 10, 5, 3 or even 1 minute, and we're already working on that.

PubSubHubbub

The notification is empty with PubSubHubbub. Why?

The notification include the new entries in the BODY. It is a POST request. However many HTML parsers in most of the high level language usually prepare this data and parse it into hashes (assuming it's coming from a form.) In the case of the PubSubHubbub notification, the notification cannot be parsed as a 'form' POST, so you need to access the raw data in the body.

What's the difference between sync and async in PubSubHubbub API?

The verification can be performed within the subscrption request cycle (that's sync) or outside of it (the hub terminates the subscription request and then, verifies the intent). Some hosting provider (like Heroku) will limit the number of concurrent connection, so you have to chose async with them.

What's the difference between the default hub and other hubs that Superfeedr hosts?

Superfeedr hosts several hubs for several publishers, including Tumblr, Posterous and many others. These hubs work exactly like you expect and as defined in the spec. Then, we have the default which works with any feed, as well as with the track feeds. It uses the exact same protocol, API. The only different is that you have to use HTTP Basic Auth to susbcribe to these feeds. This means that you can't use the feeds that get their updates with the default hub on applications like Google Reader (as Google Reader does not have a Superfeedr account!)

XMPP

Superfeedr uses XMPP, which is also known has Jabber. However, XMPP is much richer than just the chat application and defines a lot of extensions and chat is just one example of what XMPP does. Unfortunately, there is no XMPP client which supports PubSub (XEP 060) at the moment. However, with the Psi client, you'll be able to see all the incoming traffic and even be able to send some stanzas (if you're not afraid of writing some XML).