I fully agree with Soeren - and even to some extent with Wil Shipley - that custom UI and a custom look aren't bad. For example, I liked the fact iChat and iTunes were the only brushed-metal apps on my Mac way back when. It allowed me to more easily pick them out of the pile of windows to quickly answer an instant message or pick a different song.

I even think it might be handy to have something like "labels" to let you tint each application in a different colour, if you want to. I also think that brand identity - as long as it doesn't get in the way of actually using the program - is a good thing, for exactly the same reason: A particular icon style and colour scheme make it easier to pick out partially-obscured windows.

However, I do not think that custom controls should be used very often. Apart from issues like accessibility or wrong mouse tracking behaviour (like the built-in "undo" that you get when you click a button and then release the mouse outside instead of over it), the problem is that users still get confused:

I still have people asking me whether they should single-click or double-click, simply because there's no clear rule anymore now that a single click launches an item in the dock and the dock - as opposed to, say, the launcher - uses icons instead of buttons. And if you've ever seen someone double-click a "delete" icon in a window toolbar, making them inadvertently delete two list items and not noticing it, you'll know that this isn't necessarily harmless.

So, as soon as developers start introducing odd-shaped buttons, people will wonder: Is this like a segmented control or some bevel buttons which can be used like checkboxes or radio buttons? Is this a button that will immediately trigger an action? While often the use of nouns, verbs or adjectives as labels gives away what a button does (if it has a text label and not just an icon), there are other cases where it's ambiguous. Ideally, developers would either pick decent wording or make the control look like all others that work the same way on this system. But when developers use system controls, they have those as a safety net for that case where the wording is ambiguous (e.g. "debug title" could mean "title for debugging" or "debug a title", and it's easy to overlook such ambiguities because the developer's mind is already on the right track).

The guys who did Apple's HIG had oodles of reasons for recommending you don't roll your own controls if you can avoid it. But they don't outright forbid it. So, play it safe as much as you can, and only do custom stuff if you have a very good reason to do it, and if you've had all options available (including standard controls) thoroughly tested, not only by you, but also by lots of test users, disabled people, and complete newbies to the Mac.

Sören Kuklau writes:"Soeren must have hit a gold mine of great ideas for postings somewhere."
Thanks! :-)
"However, I do not think that custom controls should be used very often."
Neither do I. I am usually more of a "Usability Nazi", freaking out over every UI element that isn't "the way it's supposed to be". I assumed that most people reading my blog know this and thus look at the post from the perspective of someone who usually would *never* dare even suggest that consistency has its limits.
"And if you've ever seen someone double-click a "delete" icon in a window toolbar, making them inadvertently delete two list items and not noticing it, you'll know that this isn't necessarily harmless."
Well, that particular example is based on a misunderstanding, however: double-clicking an item always meant "perform the default (most plausible) action on this". It never meant "treat this like a button". The system tray in Windows, in theory, is a good example of this (in practice, it's a horrible one, since almost no system tray icon behaves like another): single-clicking a tray icon shows up a menu with one of the menu items in bold. Double-clicking a tray icon performs the action the bold menu item would have performed. (So much for the theory, anyway.)
The whole confusion over single- versus double-clicking goes all the way back to one Mark Andreessen who hated double-clicks and believed there should never be a need for them. The application whose interface he was working on, thus, did not use double-clicks and instead performed its most important action with a single-click. That application was Netscape, and because of it, all modern browsers click a link with a *single* click, in total contrast of behaviour elsewhere in the UI. (The only browser I can think of that supposedly -- I never had the pleasure of being able to use it myself -- did it properly was "the original", World Wide Web, for NeXT. A single click would *select* a link, allowing you to do something with it.)
"I still have people asking me whether they should single-click or double-click, simply because there's no clear rule anymore now that a single click launches an item in the dock and the dock - as opposed to, say, the launcher - uses icons instead of buttons."
Absolutely. Unfortunately, we will likely not get out of this mess any more. This is why it's so crucial to avoid UI design mistakes to begin with: once they spread, you will never get back, because some people have already gotten used to going about things the wrong way.