elb's Blog

Google Talk Denial of Service

As some of you may have noticed, our XMPP Multi-user Chat (MUC) has
been under a pretty steady denial of service attack from Google Talk
for some time now. An apparent bug in the Google Talk XMPP implementation
not only prevents Google Talk users from effectively joining the MUC, but
causes every user who tries to generate a pair of join/part messages every
five minutes on a server-induced heartbeat. This has completely swamped
attempted conversation in the MUC, as there are quite a few users with
Google Talk XMPP accounts attempting to join the conversation.

We are likely to ban gmail.com connections to the MUC in the near future,
pending resolution of this problem. We had hoped that Google would take
notice and fix things in a timely manner, but the damage to productivity
has become great enough that we're going to have to take steps on our own.
Apologies to any gmail users who might wish to join the conference —
but you may rest assured that your current connections aren't working,
anyway, so we're not really taking anything from you.

Giving Back

While you probably can't call the Pidgin developers wild-eyed
radicals (heck, a fair number of us don't use any software
projects started after about 1998 with any regularity), we do have
the tendency, from time to time, to shake up the Pidgin UI in a
quest for improvement. Unfortunately, these genuine quests for
improvement are almost always accompanied by a rich cacophony of
"I hate the new <whatever&gt!". There are also often
responses which are more useful and coherent, but this article is
not about those responses.

Our standard response to such exclamations is to ask the user to
specify what, in particular, they liked about the old behavior and
don't like about the new behavior, and why. This seems pretty
reasonable to us, but often it elicits responses like "I just like
what I like", or "why should I have to justify my tastes?".
Unfortunately, such responses are far more common than one might
expect. The problem with this sort of attitude is that it nearly
completely precludes any possibility of actual progress,
and largely just so that the user in question does not have to give
back to the community which provided Pidgin for their use in the
first place.

Not everyone is a programmer, and we know that. Another standard
response (and, again, a reasonable one) is indeed "patches
welcome", but we realize and respect that not every user is
prepared to take us up on that offer. For those who aren't,
describing how changes to Pidgin have affected work flow and
usability in some level of detail is a great way to give
back to an open source community. Such feedback allows us to
learn from our mistakes (as well as our successes) and improve our
"product" -- and in a way which is just about guaranteed to be
positive for the user taking the time to help out! Of course, we
cannot promise to accommodate every work flow or implement every
user's favorite feature, but we certainly can take those work
flows and features we are aware of into account as we move
forward.

Often, while we as Pidgin developers and contributors are engaged
in trying to fix behaviors which have proven to be unpopular,
users will request (a term which I use as a euphemism, as "demand"
is normally more accurate) that an option be added to Pidgin which
allows both the new and old behaviors as alternates. Leaving
aside the fact that often the new behavior really is
broken, and needs to be fixed regardless, this is a disingenuous
suggestion. In a recent (on-going, at the time of this writing)
debate regarding changes in the behavior of input area sizing on
IM windows, I
wrote an email which said the following about this:

Options are expensive.

Each individual option may be simple or trivial, but they add
up. Options to do with UI behavior are particularly
expensive, due to their frequent interactions with the
vagaries of UI toolkit behaviors, etc.

This expense takes the form, mostly, of subtle bugs and extra
programmer effort. We have had numerous examples of options
which appear to be simple on the surface causing numerous bugs
and undesirable behaviors. A recent and memorable example of
this was the option for persistent conversations -- I think we
all agree that the ability to have conversations "open"
without an open window is one which is appreciated by many
users, but this seemingly simple feature has caused quite an
avalanche of small and not-so-small bugs. In fact, there is
speculation that the one pixel high input area which some
people are seeing stems from this very option.

Anyone who tells you that options are not expensive is either
not a programmer, or has never worked on a large program. It
is true that orthogonal options in a clean code base can be
implemented with little cost in many circumstances,
but even the most forward-thinking and competent programmers
will be blind-sided by subtle interactions on occasion,
especially when throwing third-party libraries and code into
the mix.

So, if options are an avenue of last resort, we're left with three
other immediate choices in the face of an unpopular (honestly,
usually this means "unpopular to a minority", but try convincing a
FOSS user holding a minority opinion that he or she does not
represent the Gospel truth, some time) development decision:
revert the change to previous behavior, keep the new behavior
unchanged, or figure out what is wrong with the change and forge
an even better solution which satisfies all parties. Again, from
the email previously quoted, I said the following on this subject:

Progress is good.

The question is not whether Pidgin can remain exactly the same
forever (it obviously can), but whether we want it
to. This should be particularly understandable in the world of
open source software, where people consider a project which
has not released in as little as a few months "abandoned" and
ripe for discard. We want to make Pidgin better, we want to
make it easier to use, and we want to make it more pleasant to
use. This means that statements like "I want the old behavior
back", with no justification, are not useful to us. On the
other hand, explaining WHY you want the old behavior back, and
what precisely it is about the old behavior which makes Pidgin
better and more useful to you, allows us to improve our
software. It may be (as many users have found in the past)
that the old behavior isn't really what you want, it
simply afforded some particular functionality which the new
behavior has lost. If that functionality can be identified,
we might even be able to provide it in a superior way which
you find more pleasant and useful to use.

The feature currently under debate is a great example of this.
We have had a couple of users who have explained why
they need large input areas, and the circumstances under which
this functionality is useful to them. We have more or less
agreed (I believe) that the new behavior is not
sufficient for all use cases, and we do intend to
improve it. Using the "why" we have heard from some users, we
will hopefully be able to do so in a way which cleanly adds
the new functionality while preserving their requirements.

Reverting to old behavior should be seen as a last-ditch
effort which is giving up on an attempt to make Pidgin better.
It might be the right choice, but the change was made for a
reason -- in this case, many people like and
want input areas which automatically resize.
Numerous iChat and Adium users, for example, have specifically
asked about this behavior in the past.

So there it is; we want to avoid options except where options are
truly the right choice, and we want to be able to make progress in
improving Pidgin, even when those improvements cause behavior to
depart from previously established norms in significant ways. In
order to accomplish this, we need help -- as Pidgin developers, we
use Pidgin in only a limited number of work flows and situations.
Input from users with differing habits and expectations gives us a
broader perspective, and allows us to improve Pidgin for everyone.
A lot of our best ideas and UI changes over the years have been
inspired by comments by users, comparisons to other IM clients or
software packages, and even third-party contributions in the form
of patches or code.

All pleas to the betterment of Pidgin aside, defending assertions
of goodness and badness is simply the right thing to do.
Beyond even that, it is a reasonable bar for developers to apply
to users when judging whether their opinions and commentary are
worthy of time and
effort. While it is true, as discussed above, that not all
software users are software programmers, all software users
should, with some application of effort, be able to express their
opinions about features and changes in a clear and logical
fashion. While some Free Software users seem to think that their
right to use and modify software extends to a right to have
software modified for them (we have taken to calling this Free
Software User Entitlement Syndrome), the fact of the matter is
that Free Software is a privilege, not a right. (Lest anyone
become upset with this, note that I do not mean that the rights
granted by, e.g., the GNU Copyleft are not rights; bear with me.)
This is because that software must be created by someone, and that
someone has every right to create the software they wish to
create. Once that software is created and released, every Free
Software user has the right to use it in all of the glory allowed
and guaranteed by its licensing, but the choice to create rests
with those who have the ability and inclination. Not
surprisingly, those with the ability and inclination to create
Free Software may choose to spend their valuable time assisting
users who give back. This is not only more rewarding
than helping users who issue nothing but demands, but it seems (at
least to me, and to some other Free Software developers) to
produce a better product. Changes made with an eye to existing
and articulated use cases are guaranteed to help at least the
person who articulated the use case, for example.

Consider the issue to be similar to democracy. If you didn't vote
in the election, you can't complain about the results. If you are
unwilling to provide useful input to the developers of
the software you use, then you forfeit any right to ask for
changes. The difference between useful input and an impudent
demand is nothing but justification, clear explanations, and
logical argument. This is not to say that every FOSS developer
you approach is beholden to implement any suggestion which you
take the time to defend, but said developer certainly can
be expected to at least consider your suggestion fairly if you put
in some effort. Remember, at the end of the day, it is
the developers' sweat and effort that go into the software, so the
buck stops there.

Help us help you

In the various forums through which the Pidgin developers provide
support for building and using Pidgin, we see some very clueful
users who are enjoyable to work with, and a lot of users who make
support truly painful. In my opinion, there is one major thing
that users can do to help us make their Pidgin support experience
more enjoyable for all involved parties, and it's almost completely
effortless!

Now, before I go any farther, let me clarify what I mean by
"clueful" in the above paragraph. We do not ask that our users
know anything in particular about computers, operating systems,
package management, or any given topic; it is necessary that a
user have a certain fundamental knowledge from which we can work,
but that knowledge, in the right hands, is relatively minimal.
Instead, I am talking about clue related to receiving and making
use of help given. This is more a life skill than a computer
skill, and it seems that it is sadly lacking in an unfortunate
proportion of circumstances. I hope that this is more an
ignorance than a primary trait, and that it can be corrected via
some simple education.

Definitions out of the way ... what is that major,
important skillset in being helped? This isn't going to come as a
surprise to anyone, but it seems that a lot of people don't think
about it:

The most important skillset for effectively receiving help from
a support forum is listening and following directions.

I can't claim to know what the root cause is which causes this
process to fall down, but there do seem to be a few false pretexts
under which supplicants often labor. They are:

"I just need a little bit of help, it'll save time if I ignore
suggestions that I know are unrelated."

This comes up unbelievably often. The first thing to
remember to fight this tendency is that you are coming for
help because you have a problem you cannot solve, and the people
who are helping you are "experts" in the field. You may be
entirely convinced that your problem is unrelated to a particular
topic, but the developer who is helping you quite possibly knows
something that you do not. Often, more often than I care to think,
we ask a user to check some particular setting, feature, or aspect
of the environment, and they refuse, on grounds that they "know"
it to be correct/unrelated/etc. Sometimes this position is
correct, but often it is not, and when the user is
finally convinced to look into it, the problem can be
immediately rectified and everyone involved can move on. This
particularly comes up when a user believes that the reason the
particular question is being asked is to spite their personal
favorite operating system, window manager, distribution, or
whatever is perceived as being related; while the characteristic
in question may in fact be the reason a developer is asking, the
question is probably being posed because there is actual reason to
believe the problem exists. Stepping around the tendency to become
defensive and simply answering the question will allow everyone to
move on, either having solved the problem, or looking for the next
most likely cause.

"I've already looked at all possible causes, I just need
an oracle to tell me what's wrong. I can't be bothered to check
for those causes again, it'll just waste time."

This is really related to the previous point — if you really
had checked everything, and your conclusions were truly correct,
you wouldn't need to be consulting the experts, would you? That
aside, it is staggering how often a setting that a user "has
already checked," and "knows" is correct, is mis-set and causing
all of the problems they want to ask about. Again, if a developer
or support person is asking about a particular setting, option, or
compile flag, it is probably because it is known to be related. A
quick double-check won't hurt anything, and it might save a lot of
time in the end. Note that a common behavior that
support personnel see is a response of "yeah, it's set correctly,"
and "yes, I just checked after you asked," when the user has
not checked, believing they already know the answer.
This is particularly frustrating, and quite embarrassing (for the
user) when the user is finally convinced to look at the option and
admit to having lied, when it is not set as believed.

"I don't understand the questions I'm being asked, but
asking for clarification will just make me look dumb. I'll make
something up, instead."

This characterization is almost certainly an oversimplification,
but it sure appears to happen often, to those who
provide support. If someone who is trying to help you asks a
question, and you don't understand the question or don't know how
to answer, please ask for clarification. The more quickly the
correct questions can be answered, the more quickly the root
problem can be identified.

"I'll supply some helpful guesses as to what I think the
problem is, and start answering questions once I'm done. The
developers may not have thought of my insightful
predictions."

This is a hard one, because sometimes the user does
know something that the support person does not, and it really is
salient — but in general, the less informed a speculation
is, the less it is worth. The support person you are dealing with
very likely has more informed speculation, so hear them out before
you start second guessing with your own intuitions. A related
point is...

"I've just made some major changes to my
software/hardware/network/something, but I'm sure that's not
relevant. The fact that I need to seek support for something which
worked until about the same time the change was made is probably
coincidental. I don't need to mention the changes."

Wow, does this one happen all the time. There is
tension between this and the previous mindset, but in many cases
one can distinguish the difference. Rather than coming in and
saying something like "My Pidgin just stopped working on Google
Talk, can anyone help?," when you know you just performed a major
system update, a more helpful question might be "My Pidgin stopped
connecting to Google Talk after my last reboot; I had just
upgraded about a dozen system packages before the reboot, but I
can't find the problem." If the upgrade is likely to be relevant,
the support person now knows about it; if not, no harm was done,
and help can still proceed appropriately.

"This support person suggested that there is a problem
with a piece of software/hardware that I really like. I can't
stand for this!"

I don't know where this attitude comes from, but it certainly
prevents a lot of users from being effectively helped. All I can
say about it is, remember that "Your <X> is broken" is not a
reflection upon you, your mother, your upbringing, or anything of
the like; it simply means that your <X> is broken. This may
be a transient state, a correctable situation, or a simple
incompatability with no fault to be laid; it might also be a true
fault in <X>. In any of these cases, protesting that
<X> is truly the best and not to be slighted upon is
probably not going to fix the problem. Remember also that it, even
if the person helping you hates <X> and tells you so (and,
they probably have good reason, whether the circumstances from
which that reason arise are relevant to you or not), that really
doesn't affect you — if you're convinced of the
incorrectness of that statement, the best think you can do is get
your current problem fixed and then figure out what
can be done about the dislike of <X>, if you still care.
Refusing to be helped simply because you are using some piece of
software, hardware, or distribution that the person helping you
has identified as having a problem is beyond ridiculous. Becoming
belligerent or fractious about it is even more so.

These aren't the only traps to watch out for, but they help
illustrate mindsets which are unhelpful to receiving effective
support. Generally speaking, be open minded, try not to make
assumptions or pre-judge the situation, and be prepared to receive
advice from the domain expert at hand. Even when said expert is
wrong about a particular tactic, answering their questions and
proving that is the best way to identify the situation and help
them come up with a new hypothesis. Support personnel cannot be
expected to land upon the correct solution the first time, every
time, but we do do our best — and I think, in the context of
Pidgin, our best is actually pretty good.

Finally, remember that the person who is helping you is completely
uncompensated for their time or effort. Being short, rude, or
evasive with them is not only frustrating, but likely to cause
them to give up on you. For most of us, open source development
and support is a labor of love, not compensated labor; when it
stops being fun, we stop being driven. Fortunately, most of us
find identifying the cause of a tricky problem interesting, so if
you are polite and helpful, your problem is likely to be solved
relatively rapidly.

A related topic is how not to give bad advice, or, how to
helpfully help the helpers, directed at users who are not central
to the project but who wish to assist in support. That is a story
for another time, but the advice in this piece bears on the
situation.

Terminal Illnesses

The addition of finch to the Pidgin family has caused a lot of
terminal woes to show up on IRC and in our XMPP MUC. Here are a
few musings on the state of terminals in the Unix world today.

First of all, a plea to terminal authors and packagers: Please do
not lie about your terminal's type or capabilities! Several
popular terminals claim to be xterm but have incomplete xterm
emulation (gnome-terminal, I'm looking at you), and some packagers
change terminal definitions gratuitously (Gentoo, why is your
screen advertised as screen.linux, which appears to be a broken
terminfo definition just for the sake of brokenness?)[1]. The bottom
line on this is that terminal definitions exist for a reason, and
have meaning, and the definition being used needs to match the
terminal being used. If you are using xterm, set TERM=xterm; if
you are using rxvt-unicode, set TERM=rxvt-unicode; if you are
using screen, set TERM=screen.

With that out of the way, we arrive at the next problem: locales
and locale management. In our case, finch is a bit problematic in
that it itself does not really obey locales; finch produces fixed
UTF-8 output in many cases, regardless of what your locale claims
its character type is. This really isn't good, and it's only
exacerbated by locale problems. You really need to be using a
UTF-8 locale (typically en_US.UTF-8 or en_US.utf8; replace en_US
with your appropriate country code), and your terminal and
emulated environment both need to agree on this. If your terminal
emulator thinks you're using ISO-8859-15, but your programs are
sending UTF-8, understandably bad things will happen. Generally
speaking, this means that the environment both within and
without your terminal need to agree. To further
complicate matters, there exist terminals (such as xterm) which
don't really obey the locale; to get a proper, working UTF-8
environment in an xterm, you need to provide the -u8 command line
switch. Putty and other Windows-based ssh terminals seem to share
this little quirk, probably because the Windows concept of locale
is quite different from the Unix concept.

GNU screen is just another wrench in the works, but fortunately it
can be a useful wrench if you manage to get it unstuck from the
gears. Screen, like finch and xterm, is kind of ignorant about
locale encodings and has its own mechanism to deal with them. If
you start screen as 'screen -U' or with 'defutf8 on', it will
assume that all programs within the screen send UTF-8
regardless of locale; in addition, -U tells it that your external
terminal is UTF-8. The neat trick here is that, for future
attachments to the screen, the utf-8 sent by internal applications
will be translated to the external terminal's declared character
set! This means that you can view finch's UTF-8-only renderings
in an ASCII terminal, if you play all of your cards right. Now,
you'll want to make sure (if you're a Gentoo user, particularly)
that your TERM inside screen is 'screen' and not something broken
like 'screen-linux'; if your arrow keys don't work, it's probably
something broken.

It's not clear to me why all these things aren't fixed by now.
Some of them (like terminals claiming to be something they are
not, and finch sending UTF-8 regardless of locale) are clearly
willful perpetuation of brokenness, sometimes for good reasons,
and sometimes not. Others are simply historical baggage (bizarre
non-local charset options) that might eventually go away, or at
least be obviated. In the meantime, keep your terminal and locale
ducks all in a row, and the maze can be navigated.

[1]
Update (2007-08-08 14:25):
In discussion with Derek Pomery, it became clear that the
'screen.linux' thing is not related to running screen on Gentoo,
but running screen at the console -- we had simply only seen console
users running Gentoo, I suppose. (Several of them were waiting for
their preferred DE to compile.)

Sean's Book!

Sean is right, his book is an excellent big-picture overview of
Pidgin, regardless of its name. While he correctly barbs me for
having not read the entire thing and provided comments (as I long
ago told him, in a past life, that I would do), I have read enough
of it to understand that it is in fact a fantastic resource for
Pidgin developers, or indeed Open Source application developers in
general.

That said, there was no content for big-picture documentation
within the Pidgin documentation tree, as many people have
pointed out. Consider my diagram (and any future efforts) a
contribution to that particular anemic part of the Pidgin
documentation. We would have to generate hundreds of pages of
great documentation to eclipse the book, but we don't have to
eclipse it to provide at least some useful content.

Sorry, Sean!

Getting a Grip on Pidgin

People pretty regularly ask us for some sort of documentation
overview of the Pidgin codebase. While there is a lot of
documentation for the Pidgin API, none of it is big-picture type
stuff, and this makes it hard to dive into the code and get
started solving problems, particularly for casual developers. Now,
I think often people are hoping for documentation which will make
up for experience or programming knowledge, and we obviously cannot
help with that, but it's true that some big-picture information
could not hurt.

With that in mind, I whipped out xfig this evening and sketched up
a quick architecture image. It's not UML, and it doesn't use any
sort of standard notations, but it's not supposed to be anything
like that. It gives a casual overview of the way the four major
components of Pidgin (UI, Core, Protocol Plugins, and Plugins) "fit
together":

A single image is no substitute for architecture documentation, but
hopefully it will help people see how it all fits together. Note,
in particular, that the only interfaces upwards and downwards from
the Core are via well-defined structures; this is probably the
centerpoint of Pidgin architecture. Note also that a core plugin
can use Core event hooks and code, but not UI event hooks or code,
whereas a UI plugin is free to use both. Generally speaking, we
try to keep the three vertical layers of Pidgin as well-divided as
possible from the Core's perspective, while the other layers (and
plugins, which are sort of separate from a design point of view)
are free to munge the boundaries a little bit.

If you're a developer, even a new or inexperienced developer, and
you'd like to help out Open Source Software or the Pidgin project
but aren't sure where to start, picking a corner of the Pidgin
architecture and documenting it (with the help of the Pidgin
developers!) would be a great place to get your feet wet. As a
bonus, when you're done you'll have the knowledge to start writing
that feature you've always wanted!

Going public with Pidgin

As anyone who is likely to read this page now knows, Pidgin went public
on April 6, 2007. Due to legal pressures (which I will not repeat
here, I'm sure that will be covered in plenty of places), the Gaim
project changed its name to Pidgin, and libgaim became libpurple.

In spite of the circumstances surrounding this change, I am pleased
with the result, as are the other Gaim developers. It looks like most
users are, too, judging by the feedback we're hearing. There is
certainly a vocal faction which hates the new name, and is loudly
proclaiming this to anyone who will listen, but it seems to be a small
minority. I have to assume that they just don't get the joke.

There remains a lot of work to do before Pidgin is released, but I (we)
have every hope that, now that the legal specter is at least under
control, if not gone, said work can move quite rapidly. As can be seen
from the web site at pidgin.im, as well
as the bug tracker and
development pages, a lot of branding is left to be handled -- the
current web pages are basically just the old Gaim pages, for example.

You can find out about all of that anywhere, though; I'd like to take
just a moment and plug our new revision control system, if I could.
With the move to our own hosting at pidgin.im, we are no longer tied
to SourceForge'sSubversion. Being a large and
active project, we have experienced a lot of pain due to the
limitations of SVN, specifically, and the centralized version control
system (VCS) model, generically. Due to this pain, when the
opportunity presented itself, we jumped ship to a distributed VCS
(DVCS). After some debate, we settled on monotone, which I have been using
individually for various projects for some time, now. This should have
a lot of benefits, going forward. For starters:

There is no longer any divide between "user" and "developer", as
far as the VCS is concerned. This means that third-party
developers, users, and patch writers can use all of the power of
the revision control system that was previously available only to
"official" developers. This is a huge step forward for the open
source development model, and will hopefully help simplify the
acceptance of patches and code contributions from external
developers.

Branching and merging are reasonable things to do, for basically
any reason. This should make parallel development of more complex
features simpler. In the history of Gaim, there were several
points where all development but a particular major core change
ground to a screeching halt while that major change was shaken up,
largely due to VCS limitations with first CVS, and then SVN. This
sort of snafu should be minimized, with monotone.

Developer-to-developer interaction is much simpler. Eventually,
hopefully patches can be largely superceded by monotone revision
syncs. This facilitates better communication (commit logs with
coherent commit messages are available, the history of design is
visible, etc.) and more rapid back-and-forth refinement.

Monotone still has some distance to go before it's perfect, but it's
getting closer all the time -- and it's certainly an improvement over
the old way. I'm excited for Pidgin, with all of the new development
and interaction opportunities it presents.