elb's Blog

Google Talk Denial of Service

(Original post 2009-02-01)

As some of you may have noticed, our XMPP Multi-user Chat (MUC) has been under a pretty steady denial of service attack from Google Talk for some time now. An apparent bug in the Google Talk XMPP implementation not only prevents Google Talk users from effectively joining the MUC, but causes every user who tries to generate a pair of join/part messages every five minutes on a server-induced heartbeat. This has completely swamped attempted conversation in the MUC, as there are quite a few users with Google Talk XMPP accounts attempting to join the conversation.

We are likely to ban gmail.com connections to the MUC in the near future, pending resolution of this problem. We had hoped that Google would take notice and fix things in a timely manner, but the damage to productivity has become great enough that we're going to have to take steps on our own. Apologies to any gmail users who might wish to join the conference — but you may rest assured that your current connections aren't working, anyway, so we're not really taking anything from you.

Some more information on this situation has crossed the jabber.org Operators mailing list.

Update 2009-02-03: This problem appears to have been fixed by Google; thanks to Google for getting on it, and for supporting this open protocol!

Giving Back

While you probably can't call the Pidgin developers wild-eyed radicals (heck, a fair number of us don't use any software projects started after about 1998 with any regularity), we do have the tendency, from time to time, to shake up the Pidgin UI in a quest for improvement. Unfortunately, these genuine quests for improvement are almost always accompanied by a rich cacophony of "I hate the new <whatever>!". There are also often responses which are more useful and coherent, but this article is not about those responses.

Our standard response to such exclamations is to ask the user to specify what, in particular, they liked about the old behavior and don't like about the new behavior, and why. This seems pretty reasonable to us, but often it elicits responses like "I just like what I like", or "why should I have to justify my tastes?". Unfortunately, such responses are far more common than one might expect. The problem with this sort of attitude is that it nearly completely precludes any possibility of actual progress, and largely just so that the user in question does not have to give back to the community which provided Pidgin for their use in the first place.

Not everyone is a programmer, and we know that. Another standard response (and, again, a reasonable one) is indeed "patches welcome", but we realize and respect that not every user is prepared to take us up on that offer. For those who aren't, describing how changes to Pidgin have affected work flow and usability in some level of detail is a great way to give back to an open source community. Such feedback allows us to learn from our mistakes (as well as our successes) and improve our "product" -- and in a way which is just about guaranteed to be positive for the user taking the time to help out! Of course, we cannot promise to accommodate every work flow or implement every user's favorite feature, but we certainly can take those work flows and features we are aware of into account as we move forward.

Often, while we as Pidgin developers and contributors are engaged in trying to fix behaviors which have proven to be unpopular, users will request (a term which I use as a euphemism, as "demand" is normally more accurate) that an option be added to Pidgin which allows both the new and old behaviors as alternates. Leaving aside the fact that often the new behavior really is broken, and needs to be fixed regardless, this is a disingenuous suggestion. In a recent (on-going, at the time of this writing) debate regarding changes in the behavior of input area sizing on IM windows, I wrote an email which said the following about this:

Options are expensive.

Each individual option may be simple or trivial, but they add up. Options to do with UI behavior are particularly expensive, due to their frequent interactions with the vagaries of UI toolkit behaviors, etc.

This expense takes the form, mostly, of subtle bugs and extra programmer effort. We have had numerous examples of options which appear to be simple on the surface causing numerous bugs and undesirable behaviors. A recent and memorable example of this was the option for persistent conversations -- I think we all agree that the ability to have conversations "open" without an open window is one which is appreciated by many users, but this seemingly simple feature has caused quite an avalanche of small and not-so-small bugs. In fact, there is speculation that the one pixel high input area which some people are seeing stems from this very option.

Anyone who tells you that options are not expensive is either not a programmer, or has never worked on a large program. It is true that orthogonal options in a clean code base can be implemented with little cost in many circumstances, but even the most forward-thinking and competent programmers will be blind-sided by subtle interactions on occasion, especially when throwing third-party libraries and code into the mix.

So, if options are an avenue of last resort, we're left with three other immediate choices in the face of an unpopular (honestly, usually this means "unpopular to a minority", but try convincing a FOSS user holding a minority opinion that he or she does not represent the Gospel truth, some time) development decision: revert the change to previous behavior, keep the new behavior unchanged, or figure out what is wrong with the change and forge an even better solution which satisfies all parties. Again, from the email previously quoted, I said the following on this subject:

Progress is good.

The question is not whether Pidgin can remain exactly the same forever (it obviously can), but whether we want it to. This should be particularly understandable in the world of open source software, where people consider a project which has not released in as little as a few months "abandoned" and ripe for discard. We want to make Pidgin better, we want to make it easier to use, and we want to make it more pleasant to use. This means that statements like "I want the old behavior back", with no justification, are not useful to us. On the other hand, explaining WHY you want the old behavior back, and what precisely it is about the old behavior which makes Pidgin better and more useful to you, allows us to improve our software. It may be (as many users have found in the past) that the old behavior isn't really what you want, it simply afforded some particular functionality which the new behavior has lost. If that functionality can be identified, we might even be able to provide it in a superior way which you find more pleasant and useful to use.

The feature currently under debate is a great example of this. We have had a couple of users who have explained why they need large input areas, and the circumstances under which this functionality is useful to them. We have more or less agreed (I believe) that the new behavior is not sufficient for all use cases, and we do intend to improve it. Using the "why" we have heard from some users, we will hopefully be able to do so in a way which cleanly adds the new functionality while preserving their requirements.

Reverting to old behavior should be seen as a last-ditch effort which is giving up on an attempt to make Pidgin better. It might be the right choice, but the change was made for a reason -- in this case, many people like and want input areas which automatically resize. Numerous iChat and Adium users, for example, have specifically asked about this behavior in the past.

So there it is; we want to avoid options except where options are truly the right choice, and we want to be able to make progress in improving Pidgin, even when those improvements cause behavior to depart from previously established norms in significant ways. In order to accomplish this, we need help -- as Pidgin developers, we use Pidgin in only a limited number of work flows and situations. Input from users with differing habits and expectations gives us a broader perspective, and allows us to improve Pidgin for everyone. A lot of our best ideas and UI changes over the years have been inspired by comments by users, comparisons to other IM clients or software packages, and even third-party contributions in the form of patches or code.

All pleas to the betterment of Pidgin aside, defending assertions of goodness and badness is simply the right thing to do. Beyond even that, it is a reasonable bar for developers to apply to users when judging whether their opinions and commentary are worthy of time and effort. While it is true, as discussed above, that not all software users are software programmers, all software users should, with some application of effort, be able to express their opinions about features and changes in a clear and logical fashion. While some Free Software users seem to think that their right to use and modify software extends to a right to have software modified for them (we have taken to calling this Free Software User Entitlement Syndrome), the fact of the matter is that Free Software is a privilege, not a right. (Lest anyone become upset with this, note that I do not mean that the rights granted by, e.g., the GNU Copyleft are not rights; bear with me.) This is because that software must be created by someone, and that someone has every right to create the software they wish to create. Once that software is created and released, every Free Software user has the right to use it in all of the glory allowed and guaranteed by its licensing, but the choice to create rests with those who have the ability and inclination. Not surprisingly, those with the ability and inclination to create Free Software may choose to spend their valuable time assisting users who give back. This is not only more rewarding than helping users who issue nothing but demands, but it seems (at least to me, and to some other Free Software developers) to produce a better product. Changes made with an eye to existing and articulated use cases are guaranteed to help at least the person who articulated the use case, for example.

Consider the issue to be similar to democracy. If you didn't vote in the election, you can't complain about the results. If you are unwilling to provide useful input to the developers of the software you use, then you forfeit any right to ask for changes. The difference between useful input and an impudent demand is nothing but justification, clear explanations, and logical argument. This is not to say that every FOSS developer you approach is beholden to implement any suggestion which you take the time to defend, but said developer certainly can be expected to at least consider your suggestion fairly if you put in some effort. Remember, at the end of the day, it is the developers' sweat and effort that go into the software, so the buck stops there.

Help us help you

In the various forums through which the Pidgin developers provide support for building and using Pidgin, we see some very clueful users who are enjoyable to work with, and a lot of users who make support truly painful. In my opinion, there is one major thing that users can do to help us make their Pidgin support experience more enjoyable for all involved parties, and it's almost completely effortless!

Now, before I go any farther, let me clarify what I mean by "clueful" in the above paragraph. We do not ask that our users know anything in particular about computers, operating systems, package management, or any given topic; it is necessary that a user have a certain fundamental knowledge from which we can work, but that knowledge, in the right hands, is relatively minimal. Instead, I am talking about clue related to receiving and making use of help given. This is more a life skill than a computer skill, and it seems that it is sadly lacking in an unfortunate proportion of circumstances. I hope that this is more an ignorance than a primary trait, and that it can be corrected via some simple education.

Definitions out of the way ... what is that major, important skillset in being helped? This isn't going to come as a surprise to anyone, but it seems that a lot of people don't think about it:

The most important skillset for effectively receiving help from a support forum is listening and following directions.

I can't claim to know what the root cause is which causes this process to fall down, but there do seem to be a few false pretexts under which supplicants often labor. They are:

"I just need a little bit of help, it'll save time if I ignore suggestions that I know are unrelated."

This comes up unbelievably often. The first thing to remember to fight this tendency is that you are coming for help because you have a problem you cannot solve, and the people who are helping you are "experts" in the field. You may be entirely convinced that your problem is unrelated to a particular topic, but the developer who is helping you quite possibly knows something that you do not. Often, more often than I care to think, we ask a user to check some particular setting, feature, or aspect of the environment, and they refuse, on grounds that they "know" it to be correct/unrelated/etc. Sometimes this position is correct, but often it is not, and when the user is finally convinced to look into it, the problem can be immediately rectified and everyone involved can move on. This particularly comes up when a user believes that the reason the particular question is being asked is to spite their personal favorite operating system, window manager, distribution, or whatever is perceived as being related; while the characteristic in question may in fact be the reason a developer is asking, the question is probably being posed because there is actual reason to believe the problem exists. Stepping around the tendency to become defensive and simply answering the question will allow everyone to move on, either having solved the problem, or looking for the next most likely cause.

"I've already looked at all possible causes, I just need an oracle to tell me what's wrong. I can't be bothered to check for those causes again, it'll just waste time."

This is really related to the previous point — if you really had checked everything, and your conclusions were truly correct, you wouldn't need to be consulting the experts, would you? That aside, it is staggering how often a setting that a user "has already checked," and "knows" is correct, is mis-set and causing all of the problems they want to ask about. Again, if a developer or support person is asking about a particular setting, option, or compile flag, it is probably because it is known to be related. A quick double-check won't hurt anything, and it might save a lot of time in the end. Note that a common behavior that support personnel see is a response of "yeah, it's set correctly," and "yes, I just checked after you asked," when the user has not checked, believing they already know the answer. This is particularly frustrating, and quite embarrassing (for the user) when the user is finally convinced to look at the option and admit to having lied, when it is not set as believed.

"I don't understand the questions I'm being asked, but asking for clarification will just make me look dumb. I'll make something up, instead."

This characterization is almost certainly an oversimplification, but it sure appears to happen often, to those who provide support. If someone who is trying to help you asks a question, and you don't understand the question or don't know how to answer, please ask for clarification. The more quickly the correct questions can be answered, the more quickly the root problem can be identified.

"I'll supply some helpful guesses as to what I think the problem is, and start answering questions once I'm done. The developers may not have thought of my insightful predictions."

This is a hard one, because sometimes the user does know something that the support person does not, and it really is salient — but in general, the less informed a speculation is, the less it is worth. The support person you are dealing with very likely has more informed speculation, so hear them out before you start second guessing with your own intuitions. A related point is...

"I've just made some major changes to my software/hardware/network/something, but I'm sure that's not relevant. The fact that I need to seek support for something which worked until about the same time the change was made is probably coincidental. I don't need to mention the changes."

Wow, does this one happen all the time. There is tension between this and the previous mindset, but in many cases one can distinguish the difference. Rather than coming in and saying something like "My Pidgin just stopped working on Google Talk, can anyone help?," when you know you just performed a major system update, a more helpful question might be "My Pidgin stopped connecting to Google Talk after my last reboot; I had just upgraded about a dozen system packages before the reboot, but I can't find the problem." If the upgrade is likely to be relevant, the support person now knows about it; if not, no harm was done, and help can still proceed appropriately.

"This support person suggested that there is a problem with a piece of software/hardware that I really like. I can't stand for this!"

I don't know where this attitude comes from, but it certainly prevents a lot of users from being effectively helped. All I can say about it is, remember that "Your <X> is broken" is not a reflection upon you, your mother, your upbringing, or anything of the like; it simply means that your <X> is broken. This may be a transient state, a correctable situation, or a simple incompatability with no fault to be laid; it might also be a true fault in <X>. In any of these cases, protesting that <X> is truly the best and not to be slighted upon is probably not going to fix the problem. Remember also that it, even if the person helping you hates <X> and tells you so (and, they probably have good reason, whether the circumstances from which that reason arise are relevant to you or not), that really doesn't affect you — if you're convinced of the incorrectness of that statement, the best think you can do is get your current problem fixed and then figure out what can be done about the dislike of <X>, if you still care. Refusing to be helped simply because you are using some piece of software, hardware, or distribution that the person helping you has identified as having a problem is beyond ridiculous. Becoming belligerent or fractious about it is even more so.

These aren't the only traps to watch out for, but they help illustrate mindsets which are unhelpful to receiving effective support. Generally speaking, be open minded, try not to make assumptions or pre-judge the situation, and be prepared to receive advice from the domain expert at hand. Even when said expert is wrong about a particular tactic, answering their questions and proving that is the best way to identify the situation and help them come up with a new hypothesis. Support personnel cannot be expected to land upon the correct solution the first time, every time, but we do do our best — and I think, in the context of Pidgin, our best is actually pretty good.

Finally, remember that the person who is helping you is completely uncompensated for their time or effort. Being short, rude, or evasive with them is not only frustrating, but likely to cause them to give up on you. For most of us, open source development and support is a labor of love, not compensated labor; when it stops being fun, we stop being driven. Fortunately, most of us find identifying the cause of a tricky problem interesting, so if you are polite and helpful, your problem is likely to be solved relatively rapidly.

A related topic is how not to give bad advice, or, how to helpfully help the helpers, directed at users who are not central to the project but who wish to assist in support. That is a story for another time, but the advice in this piece bears on the situation.

As further recommended reading, I suggest How to Ask Questions the Smart Way, by Eric Raymond, and Why I Hate Advocacy, by Mark-Jason Dominus.

Terminal Illnesses

The addition of finch to the Pidgin family has caused a lot of terminal woes to show up on IRC and in our XMPP MUC. Here are a few musings on the state of terminals in the Unix world today.

First of all, a plea to terminal authors and packagers: Please do not lie about your terminal's type or capabilities! Several popular terminals claim to be xterm but have incomplete xterm emulation (gnome-terminal, I'm looking at you), and some packagers change terminal definitions gratuitously (Gentoo, why is your screen advertised as screen.linux, which appears to be a broken terminfo definition just for the sake of brokenness?)[1]. The bottom line on this is that terminal definitions exist for a reason, and have meaning, and the definition being used needs to match the terminal being used. If you are using xterm, set TERM=xterm; if you are using rxvt-unicode, set TERM=rxvt-unicode; if you are using screen, set TERM=screen.

With that out of the way, we arrive at the next problem: locales and locale management. In our case, finch is a bit problematic in that it itself does not really obey locales; finch produces fixed UTF-8 output in many cases, regardless of what your locale claims its character type is. This really isn't good, and it's only exacerbated by locale problems. You really need to be using a UTF-8 locale (typically en_US.UTF-8 or en_US.utf8; replace en_US with your appropriate country code), and your terminal and emulated environment both need to agree on this. If your terminal emulator thinks you're using ISO-8859-15, but your programs are sending UTF-8, understandably bad things will happen. Generally speaking, this means that the environment both within and without your terminal need to agree. To further complicate matters, there exist terminals (such as xterm) which don't really obey the locale; to get a proper, working UTF-8 environment in an xterm, you need to provide the -u8 command line switch. Putty and other Windows-based ssh terminals seem to share this little quirk, probably because the Windows concept of locale is quite different from the Unix concept.

GNU screen is just another wrench in the works, but fortunately it can be a useful wrench if you manage to get it unstuck from the gears. Screen, like finch and xterm, is kind of ignorant about locale encodings and has its own mechanism to deal with them. If you start screen as 'screen -U' or with 'defutf8 on', it will assume that all programs within the screen send UTF-8 regardless of locale; in addition, -U tells it that your external terminal is UTF-8. The neat trick here is that, for future attachments to the screen, the utf-8 sent by internal applications will be translated to the external terminal's declared character set! This means that you can view finch's UTF-8-only renderings in an ASCII terminal, if you play all of your cards right. Now, you'll want to make sure (if you're a Gentoo user, particularly) that your TERM inside screen is 'screen' and not something broken like 'screen-linux'; if your arrow keys don't work, it's probably something broken.

It's not clear to me why all these things aren't fixed by now. Some of them (like terminals claiming to be something they are not, and finch sending UTF-8 regardless of locale) are clearly willful perpetuation of brokenness, sometimes for good reasons, and sometimes not. Others are simply historical baggage (bizarre non-local charset options) that might eventually go away, or at least be obviated. In the meantime, keep your terminal and locale ducks all in a row, and the maze can be navigated.

[1] Update (2007-08-08 14:25): In discussion with Derek Pomery, it became clear that the 'screen.linux' thing is not related to running screen on Gentoo, but running screen at the console -- we had simply only seen console users running Gentoo, I suppose. (Several of them were waiting for their preferred DE to compile.)

Sean's Book!

Sean is right, his book is an excellent big-picture overview of Pidgin, regardless of its name. While he correctly barbs me for having not read the entire thing and provided comments (as I long ago told him, in a past life, that I would do), I have read enough of it to understand that it is in fact a fantastic resource for Pidgin developers, or indeed Open Source application developers in general.

That said, there was no content for big-picture documentation within the Pidgin documentation tree, as many people have pointed out. Consider my diagram (and any future efforts) a contribution to that particular anemic part of the Pidgin documentation. We would have to generate hundreds of pages of great documentation to eclipse the book, but we don't have to eclipse it to provide at least some useful content.

Sorry, Sean!

Getting a Grip on Pidgin

People pretty regularly ask us for some sort of documentation overview of the Pidgin codebase. While there is a lot of documentation for the Pidgin API, none of it is big-picture type stuff, and this makes it hard to dive into the code and get started solving problems, particularly for casual developers. Now, I think often people are hoping for documentation which will make up for experience or programming knowledge, and we obviously cannot help with that, but it's true that some big-picture information could not hurt.

With that in mind, I whipped out xfig this evening and sketched up a quick architecture image. It's not UML, and it doesn't use any sort of standard notations, but it's not supposed to be anything like that. It gives a casual overview of the way the four major components of Pidgin (UI, Core, Protocol Plugins, and Plugins) "fit together":

A Brief Overview of the Pidgin Architecture

A single image is no substitute for architecture documentation, but hopefully it will help people see how it all fits together. Note, in particular, that the only interfaces upwards and downwards from the Core are via well-defined structures; this is probably the centerpoint of Pidgin architecture. Note also that a core plugin can use Core event hooks and code, but not UI event hooks or code, whereas a UI plugin is free to use both. Generally speaking, we try to keep the three vertical layers of Pidgin as well-divided as possible from the Core's perspective, while the other layers (and plugins, which are sort of separate from a design point of view) are free to munge the boundaries a little bit.

If you're a developer, even a new or inexperienced developer, and you'd like to help out Open Source Software or the Pidgin project but aren't sure where to start, picking a corner of the Pidgin architecture and documenting it (with the help of the Pidgin developers!) would be a great place to get your feet wet. As a bonus, when you're done you'll have the knowledge to start writing that feature you've always wanted!

Going public with Pidgin

As anyone who is likely to read this page now knows, Pidgin went public on April 6, 2007. Due to legal pressures (which I will not repeat here, I'm sure that will be covered in plenty of places), the Gaim project changed its name to Pidgin, and libgaim became libpurple.

In spite of the circumstances surrounding this change, I am pleased with the result, as are the other Gaim developers. It looks like most users are, too, judging by the feedback we're hearing. There is certainly a vocal faction which hates the new name, and is loudly proclaiming this to anyone who will listen, but it seems to be a small minority. I have to assume that they just don't get the joke.

There remains a lot of work to do before Pidgin is released, but I (we) have every hope that, now that the legal specter is at least under control, if not gone, said work can move quite rapidly. As can be seen from the web site at pidgin.im, as well as the bug tracker and development pages, a lot of branding is left to be handled -- the current web pages are basically just the old Gaim pages, for example.

You can find out about all of that anywhere, though; I'd like to take just a moment and plug our new revision control system, if I could. With the move to our own hosting at pidgin.im, we are no longer tied to SourceForge's Subversion. Being a large and active project, we have experienced a lot of pain due to the limitations of SVN, specifically, and the centralized version control system (VCS) model, generically. Due to this pain, when the opportunity presented itself, we jumped ship to a distributed VCS (DVCS). After some debate, we settled on monotone, which I have been using individually for various projects for some time, now. This should have a lot of benefits, going forward. For starters:

  • There is no longer any divide between "user" and "developer", as far as the VCS is concerned. This means that third-party developers, users, and patch writers can use all of the power of the revision control system that was previously available only to "official" developers. This is a huge step forward for the open source development model, and will hopefully help simplify the acceptance of patches and code contributions from external developers.

  • Branching and merging are reasonable things to do, for basically any reason. This should make parallel development of more complex features simpler. In the history of Gaim, there were several points where all development but a particular major core change ground to a screeching halt while that major change was shaken up, largely due to VCS limitations with first CVS, and then SVN. This sort of snafu should be minimized, with monotone.

  • Developer-to-developer interaction is much simpler. Eventually, hopefully patches can be largely superceded by monotone revision syncs. This facilitates better communication (commit logs with coherent commit messages are available, the history of design is visible, etc.) and more rapid back-and-forth refinement.

Monotone still has some distance to go before it's perfect, but it's getting closer all the time -- and it's certainly an improvement over the old way. I'm excited for Pidgin, with all of the new development and interaction opportunities it presents.