Syndication is for suckers

It should be a simple problem. Assemble a list of thirty or so student weblogs, allowing them to choose their own platforms, and create a reasonably readable aggregated metablog of all their entries.

Every “feed blender” type application I have tried simply collapses under that number of feeds, and collectively represents a huge time suck over the past few weeks years.

I like the Grazr widgets, but the widget does not track unread entries or provide any sense of which entries are new. Ditto for the promising new Ginger release of Netvibes: promising, but a portal view just doesn’t cut it.

Don’t even start with me on Technorati. It was never reliable, and now I’d rather depend on carrier pigeons or pneumatic tubes. I tried to be a good ELI conference blogger last week, did my Technorati tagging to no effect whatsoever.

Until now, the best simple hack I had come up with was using the results of a tagsearch from Google Blogsearch sorted chronologically, and running the RSS through the ever-reliable Feed2JS. It wasn’t perfect, but it worked OK.

But there are perils with depending on the kindness and stability of third party software. And Google is evidently not immune to that. It seems that Google blogsearch has adjusted its algorithm somehow. In any event, the number of returned entries has decreased in recent weeks, even as the student entries have piled up. Indeed, today there are two fewer entries than yesterday.

Why don’t you run your own copy of the Venus software? It can take OPML or a CSV file (such as that produced by Google Spreadsheets) as an input and will generate you a website plus a single RSS and Atom feeds containing all the posts which you could then plug into something else if you wanted.

The prototype that Bill worked out is really, really close to what you need. I’m working on hammering in support for groups and subgroups, and integrated rating systems so people can vote posts up and down ala digg. It should scale to handle hundreds of feeds (or more), and do cool stuff like generate tag clouds of all aggregated posts, etc…

Once that all gets working properly, this feels extremely close to the Eduglu myth. The problem is, it’s been _really_ close for a year now 🙂

Hey Brian, I’d love to hear more of what you’d be interested in doing with feeds. While most people think of Grazr as just our feed widget, we’ve been steadily working on a lot of core feed transformation technology “under the hood”. If all goes well, we’re planning on releasing some of these new features in a paid form soon. It sounds like a lot of what we’re doing actually fits pretty well with what you’re looking for.

As an earlier prototype/example, we did a time sorted, time stamped feed stream for Loic Lemeur of the recent LeWeb conference in Paris. It represented something like 250 feeds of attendees, filtered on keywords like “leweb”.

While I think Bill and D’Arcy Drupal craziness is a good call here, given the prototype and what little testing I could do with UMW Blogs, I found BDP RSS with WordPress was a quick, relatively low-overhead approach.

It is NOT eduglu, and the Feeds API for Drupal is far more sophisticated, but BDP RSS has allowed us to feed blogger, TypePad, wp.com, and a few other blog types into one aggregated feed without a problem.

Besides it being a relatively manual process (which the Drupal solution is too, and I think there has to be a easy way to allow people to sign their blog up), the issue that something like BDPRSS cannot filter is another issue.

Like Jon, I am watching these comments with baited breath (redbaited breath).

Hi Brian, I think someone has done the sort of thing you’re after at the OU – see http://www.greenhughes.com/aggregator
Is this the type of thing? If so let me know and I can ask Liam how he did it.
Martin
PS – loved the Technorati comment, how can something so broken be the best we’ve got?

Your post sparked my curiosity — so, just for kicks, I grabbed a copy of the OPML file from edu_rss and used it to populate a Drupal aggregation site. From start to finish, it took about three and a half hours to build out a rudimentary site aggregating nearly 500 different feeds. The server running this is nothing special — just a subdomain on a testing box.

Your post sparked my curiosity — so, just for kicks, I grabbed a copy of the OPML file from edu_rss and used it to populate a Drupal aggregation site. From start to finish, it took about three and a half hours to build out a rudimentary site aggregating nearly 500 different feeds. The server running this is nothing special — just a subdomain on a testing box.

I’m very interested in what Bill and D’Arcy are doing using Drupal, too.

One variant I’d like to see is where Drupal aggregates all users’ own blogs on a site, and those blogs can either be outside blogs (stored in a user’s profile info) or (by default) blogs hosted in the Drupal system itself.

One other I noticed from Bill’s work so far is that it doesn’t seem that Drupal is generating a a single overall RSS feed for all the blogs it is aggregating (or else an OPML file), but probably that is not hard to correct.

The whole Drupal aggregation thing with FeedAPI really is pretty easy to set up and has worked 100% of the time since it went live. If you are at least semi-familiar with Drupal, you can get a simple aggregation site set up in 5-10 hours.

@Doug Holton — re: “One variant I’d like to see is where Drupal aggregates all users’ own blogs on a site, and those blogs can either be outside blogs (stored in a user’s profile info) or (by default) blogs hosted in the Drupal system itself.” — this is very possible — if a user has an account on a Drupal site, they can be given rights to blog. Then, they could be allowed to enter the feed for their external blog, and both the internal and external posts would be connected to the user’s profile within the Drupal site.

RE: “One other I noticed from Bill’s work so far is that it doesn’t seem that Drupal is generating a a single overall RSS feed for all the blogs it is aggregating (or else an OPML file), but probably that is not hard to correct.”

The OPML export is coupled with the administrative features of the FeedAPI, so it’d be unwise for me to expose export to anonymous users. I’m also finding a surprisingly large number of links to dead feeds, as the authors have moved on to a different platform, or are using a different method to aggregate their posts. As I clean it up, and if there’s interest, I might kick out the OPML file for all the aggregated posts.

@Phil Wilson: RE “Wow Kyle, that is a LOT of time!” — I don’t know if you are being serious or not, but 5-10 hours is a miniscule amount of time relative to the power of the tool you create. You can easily spend much more than that attempting to cobble together something that doesn’t work nearly as well from third party services where you have no control over your data.

@Phil Wilson — My 5-10 hour guess is assuming you are unfamiliar with Drupal and FeedAPI (as most people reading this blog are). I spent around 4-5 hours setting up the class site and that was my first time using FeedAPI (a fairly complex piece of software to configure). If I were to do it again, it’d take me much less time ~1-2 hours. As another datapoint, Bill put together his sample aggregator site in 3.5 hours.

I’ve got about 23 student blogs aggregated at http://intronewmediastudiess08.umwblogs.org. 22 of the feeds are from umwblogs; one is from the student’s own domain. I’m using the WPAutoblogs plugin to aggregate (actually, it republishes) on the front page (a blogpost page). I’ve had problems getting some of the feeds to work the first time out, I think because the various themes inside WPMU feed RSS out in slightly different ways (here I reach the energy barrier at the end of my knowledge). My workaround has been to put one feed in the WPAutoblog window at a time, save settings, run script, and keep messing with the feed URL (add “feed” to the end, use a trailing slash, take off the trailing slash, copy the exact RSS URL from the actual blog, etc.) until I get no error messages. Complete trial-and-error approach, and I don’t really know what I’m doing, but I got error-free aggregation, so *something* worked.

For sidebar comment aggregation, I use the BDP RSS sidebar widget. BDP RSS accepts feeds readily, with none of the “magpie” errors generated by the WPAutoblog script. The sidebar widget works well for comment aggregation, then, with the exception of the annoying way in which any new student post generates a spurious “echo comment” that shows up in the sidebar aggregation. Not ideal, then, but still eminently workable.

I’ve just started experimenting with BDP RSS to aggregate the blogs posts themselves on a page, and the initial results (due no doubt to my ignorance) were not happy. It looked to me as if using BDP RSS as a primary blog post aggregator broke the page template–though I got the feeds to work, the formatting was all wrong. WPAutoblogs, on the other hand, simply republishes the initial posts and the formatting is retained.

One caveat from this year’s metablog experience: the Anarchy Media Player plugin has the default YouTube aspect ratio as 425×350. The real YouTube default is 425×355. Until I found this problem, I was finding that students who embedded YouTube videos in their blog posts would break the formatting on the metablog. That problem seems to be fixed now.

One more thing: I’ve been able to aggregate blogs by grabbing feeds from different categories within WPMU, thus enabling students to use one blog for multiple classes. I haven’t tried aggregating by tag, yet, though feeds from tags seems to be a feature supported in WPMU 1.3.

So my metablog eduglu is more like scotch tape, mucilage, and post-its, but it works for what I’m trying to do now. And the metablogs look trim and cool, like a WP blog should look.

The metablog is handy as a dashboard, and I suspect students are using it as a reader (I know I do), but I kinda wish we wouldn’t, because the metablog means they’re not reading the blogs in their initial environments. On the other hand, Bloglines or Google Reader (or any reader) would be essentially the same in daily practice, so oh well. I have turned comments off on the metablog so folks will leave comments on the original blogger’s site. The NMS class seems to navigate this arrangement OK, and I think the Rock/Soul/Prog class is getting the idea too.

I have been trying to push del.icio.us harder this year and find the RSS feed I grab from class tags (umw_nms_s08 and umw_rsp_s08) to be essential parts of the metablog. I’m sorry it took me so long to get to this point, but glad to be here now.

Eventually, I hope students (with help as needed/desirable) get to the point of mastering their own eduglu designs, that is, to create the info architecture that best serves their needs.

If you have hundreds of feeds to work with, I guess you need what you describe, some kind of meta-blog. But for only 30 or so, how about a page showing the most recent post (or 2) from each blog separately? A PageCast from PageFlakes would do that without any kind of difficult configuration. It would also have the advantage that the most prolific bloggers wouldn’t drown out the less prolific, and you’d get a snapshot of every student’s work.

@Rick — setting this up within Drupal, you could build this out with a views/panels combination, or you could create a list of imported feeds, and link to the actual posts off the list —

In short, Pageflakes will work perfectly well for a small number of feeds where you don’t really care about having control over your date, or if all you want to do is list feeds/link to posts.

The thing that I like about the Drupal-based solution is that you are actually republishing content, and that’s where the fun can start. This opens up options for OER’s, classes, portfolios, and what have you.