Podcasting

We weren’t originally going to add podcasting support to NetNewsWire 2.0—but after intense lobbying by a thousand and one podcast enthusiasts, we changed our mind.

Here’s the gist of how it works:

1. Whenever there’s a news item with an enclosure, NetNewsWire tells you, and gives you buttons you can click to download it to disk or to iTunes.

2. You can tell NetNewsWire to download enclosures automatically or not.

3. You can tell NetNewsWire to send audio enclosures to iTunes. You can tell it to use the name of the feed as the playlist name, or you can specify a playlist name. You can also specify a genre and you can tell NetNewsWire to delete downloaded files after sending them to iTunes.

4. You can create a smart list that shows all your enclosures.

(See some of the screen shots further up to see what all this looks like.)

This is all still beta, of course—there may be bugs, and the feature may still evolve based on feedback.

More features

There are a bunch of other cool things I want to point out—but I’ll save them for follow-up posts...

Right now, NetNewsWire marks it as unread. My policy has always been to err on the side of marking things unread—on the assumption that, otherwise, you might miss something important.

(Actually, NetNewsWire marks it as updated, and you have a pref that says whether or not to mark updated items as unread. Which most people leave turned on.)

The RSS 2.0 spec doesn’t address the issue of when to consider an item significantly changed—and I’m not sure the spec should say anything about that.

Of course, I don’t want new prefs or special cases, I want the best policy.

Perhaps changes to <link>s should not cause an item to be marked unread. What do you think?

On a related topic...

People have often asked me why we can’t use the <link> element as a unique identifier and permalink for news items.

It can’t be a unique identifier because it may change. The New York Times feeds, for instance, change their <link>s frequently: they include a query string that allows you to get past the registration system, and that query string changes. (It’s the New York Times feeds that prompted the discussion above about ignoring <link> changes.)

<link>s can’t be permalinks because there is nothing that says that a <link> has to be a permalink—it could point to another site entirely.

Which is why I always ask people to use <guid>s in their feeds, so we can identify items, so we can know when an item is an updated version of a previous item.

(About Atom there is one thing I adore: that guids—called <id>s there—are mandatory. I wish they could be mandatory in RSS too.)

Below are some—but not nearly all—of the things planned for the future.

Fixing that time bug!

The most commonly reported bug is actually a WordPress bug, unfortunately. In some configurations, the time on the server for a post sent from MarsEdit is incorrect: it doesn’t take into account the time zone difference properly.

My hope is that this will be fixed in WordPress 1.3.

(By the way—there are many MarsEdit users who love WordPress. My point isn’t to knock it: WordPress is quite cool.)

Blogger and titles

MarsEdit doesn’t yet support titles for Blogger.

It will—but it requires implementing the Atom weblog editing API first. This is MarsEdit’s top priority.

WYSIWYG editing

Dave Hyatt has said publicly that WebCore will support HTML editing. We plan to use it to add WYSIWYG editing to MarsEdit.

It will be a choice, of course—those of you who prefer plain text won’t have to use the WYSIWYG editing feature.

Cocoal.icio.us support

On one hand, you have people who say, “I want each app to do one thing and do it well.”

On the other hand, you have people who say, “I want an app that puts it all together, the whole package.”

Is del.icio.us support part of that “one thing” that weblog editors should do? Is it a necessary part of “the whole package?”

I don’t know!

But, when in doubt, if there’s another app to work with, why not support that other app instead of redoing the work? After all, MarsEdit already works with a variety of newsreaders, browsers, and text editors—why not work with Cocoal.icio.us too.

An opposite case could be made, surely, and it wouldn’t be wrong—but my personal preference is to work with other apps as much as possible.

More...

The above list is not comprehensive—there’s plenty more. MarsEdit is just getting started.

But the above covers the most common bugs and feature requests we’ve been hearing.

Mitosis

The genesis of MarsEdit was the idea of mitosis, that we could remove NetNewsWire’s weblog editor and create a new, separate weblog editor—and thereby create a better newsreader and a better weblog editor.

We had long planned to support external weblog editors in NetNewsWire—but it wasn’t until autumn 2003 that we considered supporting only external weblog editors. That’s when we first sketched out MarsEdit’s user interface.

To my delight, the final version of MarsEdit looks very, very much like our original vision, done originally as a non-functioning prototype in Interface Builder.

Splitting up NetNewsWire like this was a big risk, though, and we didn’t know how it would be received. (It turned out that the feedback far surpassed our hopes.)

We not only split up the product but created an open interface so that various combinations of newsreader and weblog editor could work together. This is something we’re very proud of—even though it increased the risk.

Programming stuff

Early on, before testers even saw it, I had a few challenges...

1. Morphing user interface

I had to develop an adaptable user interface that morphs based on the capabilities of different weblog systems—and not have the morphing be obnoxious.

This was a response to one of the major problems with NetNewsWire’s weblog editor: fields that a given system couldn’t use were just disabled rather than disappeared. This led to lots of email and bug reports. Though I’m not generally a fan of UIs that morph, it had to be done here—and I was utterly pleased with the result.

At first, actually, I was entranced—I used to just keep changing the weblog system to watch the animation of fields appearing and disappearing. But I got over it and went to work on the next thing.

2. Asynchronous XML-RPC

I had to do a better job dealing with asynchronous XML-RPC calls than I did in NetNewsWire.

Here’s the deal—you click a button like Post or Refresh, it kicks off a call to the server to do something. MarsEdit waits for the response—but it can’t lock up while waiting, it has to be responsive to your commands, and it has to do things like run a progress indicator. And calls have to be chainable: make a second call after the first completes, but not before.

Well, I had this working in NetNewsWire’s old weblog editor, but I was never pleased with it, and there were times when the UI would miss the response, and a progress indicator would run forever.

This was just an architecture job. I ended up with a system both cleaner and better than what was in NetNewsWire. (The second time is usually better than the first, after all.)

What I ended up with was solid plumbing. With leaky plumbing, you spend all your time patching it and not enough time working on the UI—and the UI is where you need to spend your time.

I have barely touched this code (except to add a minor feature or two) since MarsEdit first went into private testing, which says alot for the code.

From a high level, every net operation looks like an Objective-C method call that, instead of returning a value, returns immediately and calls back to a method when it’s done. Since the code is object-oriented, maintaining state is almost not even an issue. (Simple stuff, nothing revolutionary, just plain old-fashioned goodness.)

3. Documents

Cocoa has wonderful built-in support for document-based applications.

The only trouble is, that support assumes that you’re saving documents to disk.

The whole point of MarsEdit was to be document-centered, like email—but, also like email, you don’t save files to disk, you send your document to the internet somewhere.

I had to learn about Cocoa’s document-based app features—and, at the same time, I had to learn how and where to over-ride it so that it didn’t think it was loading and saving disk-based documents. This turned out to be difficult—lots of trial-and-error. The docs don’t talk about this much.

And the early, private testers would tell you that I didn’t have all the kinks worked out in the first versions they saw.

Design philosophy: maximum elegance

Throughout the process of working on MarsEdit, the phrase “maximum elegance” repeated in my head. The idea was to keep it as simple and focused as possible.

As I’ve written before, weblog editing is far more complex than email: you have things like categories and text filters and trackbacks and all this stuff you don’t have with email.

The phrase “maximum elegance” was just a personal reminder to myself to simplify as much as possible. With something as complicated as weblog editing, you have to be relentless about simplification, or it will get away from you.

Note, for instance, how short and small MarsEdit’s menu is. How many other productivity apps do you use have such a small menu? Note how the design of document windows is influenced by Apple Mail rather than, say, Microsoft Word.

The “lightness” of an application is a matter of feel rather than number of lines of code or number of resources. It’s a design issue. I wanted MarsEdit to feel weightless in order to balance the heavy complexity of weblog editing.

Note to developers—regular folks, please skip this—there is, oddly, a danger to making an app feel light. People sometimes get the impression that it’s not light but slight—that it can’t possibly have very many features and couldn’t have taken much time to develop, so it can’t possibly be worth paying money for. That’s not true, of course. (It’s like that old line about not having enough time to write a short letter.) However, even though there’s this danger, it’s better to go for lightness, because the vast majority of Mac users appreciate quality.

The early, private testers were a huge help with this. I simplified—but they simplified even more. There was lots of feedback about stuff that could be removed from these early versions, and I think I used all of it. (I’m a strong proponent of development-by-subtraction—after all, MarsEdit exists because we removed the weblog editor from NetNewsWire.)

Of course, there’s no design nirvana. MarsEdit is very close to my original vision, and that’s wonderful—as long as the original vision is good. But is there room for improvement? Could the user interface be better still? Yes, of course. (And I already have plenty of ideas for how to make it better.)

Last-minute features

Once MarsEdit was in public beta, it was fairly close to what 1.0 would become, but of course there were bugs to fix.

And it turned out that there were a couple features I was putting off until after 1.0 that really needed to be in 1.0:

1. Preview with text filters—Markdown and similar.

2. Customizable list of URLs to ping.

That’s not to say there weren’t plenty of other feature requests, but so many people asked for these two that it became apparent that they had to be in 1.0. If I could have waited on these, I would have.

Before deciding to implement them, I had to answer a few questions for each feature:

1. How much time would it take to develop?

2. How much time would it take to test?

3. Is this feature a likely site of bugs, or will it be straightforward?

4. Can it fit in the existing user interface without major disruption?

For text-filter-preview and customizable pings list, I guessed (and all you can do is make an educated guess) that they would be quick to develop and test, that they’re straightforward, and that the user interface wouldn’t require many changes.

Another popular feature request was supporting titles for Blogger. This we didn’t do in 1.0, because it would take too long to develop and test and it would be a likely site of bugs. It sounds crazy—we’re just talking about titles, no big thing, right?—but it required adopting the Atom editing API, which is a big job. (Now that 1.0 has shipped, this is MarsEdit’s top priority, by the way.)

Wrong turns

There weren’t many features pursued that I had to drop or change drastically. Just a few things:

1. At one point during the private beta I wanted to add what I thought was a cool Rendezvous-based feature—but it didn’t interest the testers, and finally it didn’t interest me personally that much, and I dropped it before spending programming time on it.

2. The first versions of MarsEdit did previews quite a bit differently: the preview appeared in a drawer attached to the document window. This was cool for one major reason: it tied the editing window and the preview together. In a way, this is much better than having a separate, single preview window. But it had a serious drawback: you couldn’t resize the preview independently of the size of the editing window.

At one point I considered doing the preview as a splitview in the document window, which would have let you semi-independently resize it—but I ended up going for a separate preview window. (There were testers on all sides of this issue, by the way: there was no consensus. Sometimes a solution is obvious to everyone but the developer, but not this time.)

3. For a long time during the beta process the app icon looked very much like the Firefox icon. (This was just coincidence—the MarsEdit icon was originally created before the Firefox icon was created. The final icon Bryan Bell created is fantastic. I love it.)

The real story behind the name MarsEdit

In an alternate universe, MarsEdit is an outliner instead of a weblog editor—and its name is MarsLiner. It has the exact same icon MarsEdit has.

When I first asked Bryan to make a Mars-with-spaceship icon—way back in early 2003 (I think—could have been 2002) it was for an application named MarsLiner. The idea was to do an outliner that could fill in for MORE.

It’s been a sore spot in my computing life that no outliner for OS X feels as good to me as MORE did. This is purely subjective, of course—there are several really great outliners for OS X. But I want MORE, and I was willing to write it myself. (Just the outliner, that is—I didn’t care about the presentation stuff in MORE.)

The idea was to have a text-oriented outliner—I didn’t care about embedding movies and sound clips and whatnot—that was designed for keyboard users, felt very light, and was super-fast.

The rough draft of this idea was the Notepad in NetNewsWire 1.x. But MarsLiner was to be a huge improvement, it was supposed to be the outliner of my dreams.

But then I discovered something important: most people don’t care about outliners. And the people who do care about outliners, many of them would want the embedded media features that I didn’t care about. So I realized that the market would be small, just a subset of the outliner market, which is small enough already—and there are already some great outliners already.

When we decided to bag MarsLiner and do a separate weblog editor instead, I wanted to use the name “Mars” somehow and use Bryan’s cool icon. Hence the name MarsEdit. We rationalized the name by saying it represents editing at a distance, since you’re not editing local documents, you’re editing documents that live on the web somewhere.

But really it was because I like Mars and spaceships and we already had a great Mars icon.

How much programming time had I spent on MarsLiner? Very little, thankfully—I built NetNewsWire’s Notepad as a stand-alone app. I didn’t even get as far as supporting multiple documents or doing a save command.

But still today I wish for the outliner of my dreams.

From time to time I’m tempted to do it as a Terminal-based thing, all done with ncurses. This way it would have to be purely keyboard-based; it wouldn’t be able to display pictures or movies; it would be fast.

Coincidence, not planning—a trailer for a remake of War of the Worlds was posted today. I can’t tell if the invaders are from Mars or not going by the trailer—but I sure hope so. (Thanks to Robert Daeley for emailing me about the trailer.)

Adriaan Tijsselling: State of the API address: “In the end, though, it still depends on the blog system developers and how far they want to go in supporting and (properly) implementing the Atom API (without resorting to poor hacks).”

I agree with pretty much everything Adriaan says in this post—and especially the part about not “resorting to poor hacks.” Until you’ve tried to write a weblog editing client that works with a bunch of different systems, you have no idea what a pack of spiders are the various implementations.

Adriaan talks about having a weblog editing API “wrapped in a consistent, tightly-specified, well-documented IETF-controlled XML format and internet standard.” Right on to that.

An additional hope of mine is that, at the same, the API doesn’t take a computer scientist to implement. If it’s difficult and complex, both Adriaan and I will handle it, yes. But the thing to remember is that there’s a ton of creativity and interesting ideas in the scripters and hobbyists out there, and they’re not going to tackle stuff that just takes too long to see any results. That’s one of the reasons that XML-RPC and RSS have had success—they’re so easy to get started with. And that’s worth remembering.

Clark Venable: I Wish There Were An Easier Way To Give Software: “What I’m envisioning is a service, by Kagi for example (as they’re the ones that seem to accept payment for most such software), that will print the name of the software, its web site, and the registration code on a gift card that can then be tucked in a holiday card and given as a gift or stocking stuffer, or even mailed as an e-gift (like an iCard).”

If I do translate lt and gt (at parse time), then &lt;blink&gt; turns into <blink> — and, consequently, you don’t see it, because now it’s an HTML tag. The test clearly says it should be displayed as <blink>, which means that the string value has to be &lt;blink&gt;.

I’ve been rewriting the Atom parser in NetNewsWire 2.0 — and I just got it to pass the internationalization tests Sam Ruby posted earlier this year.

The screen shot looks funny, but it’s actually an indication that the tests passed.

I’m looking for more tests — and, especially, for weird Atom feeds. By “weird” I don’t mean bad or invalid, I mean out-of-the-ordinary. For instance, I haven’t found any feeds yet that use base64 encoding, and I want to test that I have base64 decoding working now. So, if you know of any weird feeds, please send me email or post a comment.

(I love my new Atom parser, by the way. Sometimes you write code that is surprisingly geometric and elegant and you can just plain see how right it is. And other times you don’t. It’s like the difference between the Parthenon and the nests birds make. Both are beautiful in their way — but, when it comes to software, you want the Parthenon instead of a semi-chaotic accretion of small sticks.)

Which is an absurd claim — it wouldn’t be much of an aggregator if it were true. But — here’s the deal — to me as the developer, it is true.

I’ll explain...

But first: the below is of interest mostly to beginning programmers and power users who are curious about how things work under the hood. It’s a post about architecture, about time-tested and mother-approved techniques. (Truly mother-approved. My mom is a code reuse expert.)

Inside NetNewsWire

The part you see of NetNewsWire is an application that knows how to display data in NetNewsWire’s internal data format. That format is a collection of custom objects that use strings and arrays and so on.

For instance, an individual item in a feed is a data-item object, and that object contains attributes such as title, link, description, and so on. And a feed is a data-source object, and it has a set of attributes — including an array of data-item objects.

These objects are the same for RSS and for Atom, even though the feed formats are different. Even if more feed formats come to be, I’ll keep using these same data-item and data-source objects. These objects don’t know thing one about RSS and Atom. They’re more abstract: they’re like the ideal of a feed and feed items.

So... how do we turn raw Atom and RSS feeds into these objects? Part of the answer is to use a black box.

The black box

In high-level terms, it works like this. NetNewsWire has the raw text of a feed. It doesn’t know anything about it — it doesn’t even know if it’s RSS or Atom or whatever. NetNewsWire’s goal is to turn that raw feed into its data-source and data-item objects.

So it turns to the black box (RancheroXML) and says, “Yo, here’s the text of a feed, please do your magic and give me back some data I know how to use.”

The box does its thing — we’ll get to that — and then returns a simple, agreed-upon intermediate data format (using standard Cocoa dictionaries, arrays, strings, and so on) that NetNewsWire can quickly and easily turn into its own data-source and data-item objects.

Some things to note:

1. NetNewsWire knows almost nothing about the box. It knows how to send it raw feed text, and it knows about the data format the black box returns. That’s all it knows.

2. The black box knows zero about NetNewsWire. “NetNewsWire? Never heard of it.”It knows how to parse feeds, and it knows how to create an intermediate data format that NetNewsWire also knows about.

I’ll explain why this is done this way, and then we’ll open the black box, and see that boxes can contain other boxes.

(Quick question: Is this black box real or virtual? In the case of NetNewsWire, it’s real. Do a Show Package Contents then look at Contents/Frameworks/. But black boxes can be virtual.)

Reasons for using a black box

Maybe it seems like a lot of trouble to go to. But here are a few of the good reasons for using the black-box approach.

1. I can use that same black box in other applications.

MarsEdit uses this same black box, for example. The box knows as much about MarsEdit as it does about NetNewsWire — nothing, that is. Had I the time and interest, I could write five or ten or a hundred more apps that use this box.

2. Concentration.

When I’m working on the box, I can put blinders on — I don’t have to think about my applications, I can concentrate entirely on some smaller task inside the box.

Every big programming task is really a collection of small tasks. Using boxes helps break tasks up into their smaller components.

For instance, when I’m working on NetNewsWire’s HTML display, I’m not thinking about feed parsing, and I shouldn’t be. In fact, the RancheroXML project isn’t even usually open, since I rarely need to work on it. That’s a bunch of code put away neatly until I need to work on it.

3. Freedom.

A black box is like Las Vegas: what happens in the box stays in the box.

As long as it does what it’s supposed to do — take raw feed text and return that data format — then anything at all could happen inside the RancheroXML box. (In fact, I just re-wrote my Atom parser, which lives in the box. NetNewsWire has no idea — and doesn’t care.)

4. Multiple programmers.

Imagine multiple programmers. If each person (or pair or group) had their own box to work on, that might work out rather well, no?

Inside the black box

What happens inside the black box could change. It wouldn’t matter to NetNewsWire — as long as the black box takes input and returns the right kind of output.

1. Raw feed text comes in to a parser-finder class. This class doesn’t know about Atom or RSS or XML — but it does have a list of parser classes that live in the box. (A parser class is a class that knows how to parse a specific feed format. RancheroXML has one for RSS and one for Atom.)

2. For each parser class, the parser-finder asks, “Hey, do you want to parse this text?” And it lets it examine the text.

3. If a parser class says yes, then it gets to do the parsing, then returns to the parser-finder that agreed-upon data format, which is then passed back out of the box to NetNewsWire.

So the thing to know is that the parser-finder itself knows nothing about Atom or RSS. There could be more feed formats, and it would know nothing about those too. It treats the parser classes as a row of black boxes, all the same.

But the parsers are not the same — except in that very small part the parser-finder sees. In fact, my RSS and Atom parsers are wildly different. (Mainly because every time I write one of these I learn how to do it better next time.)

Final thoughts

This whole thing is not new with me or unique to NetNewsWire or even particularly special — at all — it’s just how programming is done. It doesn’t get you out of having to deal with things like memory management and performance. Using black boxes doesn’t magically make software good.

And then sometimes there are things that seem to violate the black box principle. For instance, NetNewsWire’s data-item class didn’t know about summaries, since summaries don’t exist in RSS. (The data-item classes were created with RSS in mind, because Atom didn’t exist when I created them.) While the addition of Atom as a new format is theoretically only a matter of concern for the RancheroXML box, it affected NetNewsWire too: it had to know that feed items could contain summaries.

But that’s okay, because what really happened is that the superset of what a feed item may contain did change, so NetNewsWire’s data-item object had to change.

NetNewsWire still doesn’t know Atom from oranges. Or RSS from bananas.