Forum OpenACS Q&A: Email alerts grouped by topic

Does anyone else follow this bboard and those at aD via email? I find
it hard to believe people really follow questions via the web -- AFAIK
there's no way to tell what questions have responses I haven't read or
for such questions which responses I haven't read other then through
lots of clicking, scrolling, and eyeballing.

So I subscribe to this forum and several of those at aD via email
alerts. This is ok, but due to the lack of In-Reply-To headers, and
the values given to From: and Subject:, reading bboard responses in
email is still a pain.

My solution is to accumulate hundreds or thousands of responses in a
mbox, then when I feel like catching up on ACS happenings, run some
regexps on the file so that my mail reader can group responses by
question and display the author in From:.

Each of these makes reading much, much better: grouping by question
allows me to easily concentrate on one question at a time, and delete
a whole group of responses if they don't interest me with one ^d
(using mutt). Showing the author in From: allows me to at a glance
read the people who I know write interesting stuff and delete
responses from people who aren't worth reading.

It would be trivial to have email alerts for bboard postings
automatically generated with the necessary headers, which would save
me from having to occasionally munge my mailbox and make it easier for
everyone else to follow bboards via email as well:

I'd just post a patch, but unfortunately I don't know exactly what
versions of [Open]ACS/bboard are running here and at aD, and it seems
that different bboards are set up to generate slightly differently
formatted alerts, so I'll describe generally what needs to be done in
hope that people with access to the OpenACS and aD forums might make
the necessary (trivial) changes:

Subject:

Currently, the subject line is formatted along the lines of

Subject: bboard_name: {Response to }question_subject

This should be changed to

Subject: {Re: }[bboard_name] question_subject

(I'm using curly braces to indicate optional fields, as [] should be
used to enclose the bboard name following a commonly-used mailing list
convention).

This change allows some mail readers to group messages by subject,
appearing as a psuedo-thread. To get real threding, you'd need
In-Repy-To:, and I'm assuming that may be harder, as I'm not sure at
what point message-ids are generated.

From:

Currently all bboard response alerts are sent with From: set to
"bboard@openacs.org" or "bboard-robot@arsdigita.com". This isn't done
to protect the email address of the poster, as this information is
included in Reply-To: (on aD, the email address and full name of the
poster are also included in the message body). It also needn't be
done to identify the email as originating from a bboard, as there are
plenty of other markers of that in the header and in a trailer
instructing the reader how to reply, etc.

I follow the bboards via email alerts as well. I agree that additional information would be useful, but part of the problem is that different people have different preferences. For example, I know a number of people who don't like to have the forum name in the subject line, while I think it is great.

The best solution that I see, given the capabilities of ACS, is to allow people to choose their own formatting. There would be a default format, but users would have the ability to customize it for their alerts. It doesn't even need to be all that complex - just some toggle buttons like "I want the forum name in the subject line - yes, no".

p.s. In the near future, I ask that the OpenACS bboards add the poster's name and email address to the mail like you get on the aD bboards. Without that, you have no idea who posted the message without visiting the thread.

I find your suggestions very welcome. I posted this same question on the aD bboards a while back, but nothing was done (though there was positive feedback).

I find myself in these same frustrations... it's just hard to read a mailbox full of bboard postings the way they are setup right now, especially after you got used to mutt's excellent threaded interface. It's hard to follow posts and it's hard to know who is answering what.

It's also hard to post a response to someone's post when that post is not the original question. I find myself having to open two browser windows: one to post the response and another to look at the response and copy/paste. Or I copy/paste from my mutt window into the browser response window.

I could make a patch to address these issues on the client side (easy). But IMHO this should be addressed on the web interface side too (harder). For example, the user should be given an option of threaded or regular interface. And you should be able to post to a certain response within a thread, instead of just the original question.

As a band-aid, you can use the procmail recipe below. It changes
the subject line to Re: question_subject [bboard_name].
Together with a threading mail reader (i.e., one that can thread by
subject lines alone, such as Gnus), reading the bboard becomes quite
comfortable. It isn't exactly rocket science, but somebody might find
it useful ...

It seems to me that (a) bboard is absolutely one of the core, core ACS modules, and (b) everybody agrees that it could use some enhancements in terms of email handling and other capabilities. The company I work for has invested some money into adding email enhancements to bboard already. Unfortunately, due to circumstances that aren't worth going into, these enhancements were added to a highly non-standard (and somewhat ugly) version of bboard in an ACS Classic 3.4 installation. At some point in the future, I anticipate we will able to invest more money into beefing up the email and other capabilities in the OpenACS 4.x bboard.

What I'm getting at here is that there seems to be a lot of interest in bboard right now and it might be worth setting up a forum devoted to it, much the way CMS has its own forum. Has anyone yet volunteered to be the lead on porting bboard?

I've customized bboards quite a bit... and I think many of my enhancements would benefit the comunity. My enhancements to OpenACS 3.2.2 are also ugly but work well. They are ugly in the sense that the code can't be easily ported to what we have now. I'd be interested in participating in the redesign forum.

There is this long thread on Arsdigita about forums from last June:Remodeling forums for arsdigita.com -- input requested

One feature that "Everyone" would want right away is the
feature that lists posts with the most recent replies as a replacement
for the current main posts.
EXAMPLE:

What's so great about Open Source Software?Ned Newbie Apr 04

How do I restore my PG Dump with the least pain
Most recent reply by Don Buccus Apr 03

The First thread has no replies and was written by Ned. The second thread has the most recent reply by Don but was started two or three weeks ago. This system is better than the current separate list
of "new answers" which often don't work right when you don't read ALL of them so some might be missed...

I'd like to see much of the ACS (but bboard especially) unified with a mailing list manager.

I have found that there are folks that only want to interact with the bboards through email, much as they do their current mailing lists. And yet, uniformly, mailing list managers out there today all have horrid and very weak web interfaces, and don't let you post new messages. I've also heard complaints/rumours/suspicion, etc. that ACS mailing list is great compared to the a mailing list manager specifically written to batch up emails, etc., and of course as we all know, the ACS does a particularly poor job at creating the headers our mail user agents (and us) like to see for threading, filtering, (and the removal of vacation messages.)

A unification would benefit both the ACS and mailing lists.

A few weeks ago I looked into integration with Mailman. Mailman is opensource and reasonably popular. I discovered then that an employee of GreatBridge is working on a mailman/postgres integration, and I am waiting for that...

The biggest disadvantage is that it would however create a python dependency on the ACS. (Which now requires Tcl and Perl.)

I am interested in this and wonder if the unification ONLY
needs better integration with a mail program. I've assumed
that qmail is the best in this regard. So you have the bboard
with messages in the database. You use a qmail interface
to allow posting new messages to a bboard/topic and replying
to messages via email that get dumped into the database.

It's certainly true that the ACS could be better integrated with qmail (and simply too), regardless of any other effort. Simply setting a few headers correctly would help identify to *us* who is writing a message, and would direct many vacation programs to not spam us all with the travel news of our various community friends.

But I believe integration of bboard with a maillist program has a wonderful benefit. It gets the ACS into communities that have members that ONLY use email and won't otherwise get on the net. That could be existing, "stodgy" communities, it could be folks with mobile email access (pager, pda, or phone/vxml), or it could be folks buying new, email-only devices to attach to their phone systems.

This means supporting people that never want to (or can) click on a link. That's supporting subscribe/unsubscribe via email. Creating a new thread via email, and replying to an old thread. It means supporting these people by letting them choose instant responses or digests, etc., again, without requiring them to come back to a website.

Everyone knows we need to be able to initiate threads and reply to them via e-mail, and no one knows why aD's one implementation of this was for ACES (the platform Michael Feldstein was referring to). ACES doesn't use bboard but rather a truly ugly hacked thing based on the general-comments module, so the work they did doesn't work with bboard. Grrrr...

I think integration with qmail is the right thing to do, given that so many ACS users are familiar with it.

Mailman doesn't take over qmail's job. It's not an SMTP mailer. Mailman is more of a combination procmail/acs-lite targeting subscription/operation/distribution/digestification(yum!) of emails.

Mailman can use qmail, and if bboard were to use mailman (or a new bboardman that is) it doesn't mean that the rest of the ACS should not be better integrated with qmail (which it should be). It's just saying that it would be nice to be able to let users manage bboard properties (including creating and replying to messages) entirely through the use of email.

This could certainly be done with qmail integration directly. To do so would mean learning and reimplementing what other mailing list manager programs have been doing for years. It would simplify ACS requirements to do it directly, as sites would not need Python. It would probably not be a simple or quick task. I think the quick task is to integrate with Mailman, especially after (this guy from)GreatBridge finishes the Mailman/PostgreSQL integration effort. When that happens, presumably, what one would need to do is just to populate certain PostgreSQL tables with new web based posts and read from various tables to determine new email based posts.

Put another way, I would trust myself to integrate with Mailman properly. I wouldn't trust myself to better integrate with qmail in a manner that would mean that users could post new messages that get distributed to other users AND I wouldn't get called at 3am with calls of how my errant bboard/qmail integration was responsible for a mail loop/DOS/imminent collapse of the net.

One concern I'd have is the degree to which we can capture bboard web input from an incoming email. For example, suppose you make use of the "topics" feature in bboard. Is it possible to create some intuitive way for users to indicate topics in their emails and have the bboard recognize it?

And what about threading? I'm not a big fan of threads (or trees, to be more accurate) in bboards, but I'm guessing that at least some folks are going to want to implement them in 4.x. Can an email provide an adequate interface to let users indicate whether they are replying to a post or its parent?

And what about identifying posters? One of the great things about bboard is that it links all your posts to your profile. We'd want incoming emails to be reliably recognized by bboard as coming from a specific registered user. In ACES, at least, making this work required some thought. Basically, (as I understand it), the ACS generated a unique "reply-to" email address for each user receiving an email alert for each post. The system could reliably identify the users because only one user could possibly be replying to a given email address. Will it be possible to make something like this work with a third-party listserver?

I believe identifying posters is trivial, threading is simple, creating new topics is hard, and getting rid of quoted nonsense and signatures is downright difficult.

Posters are identified with their email address which is typically the same as their ACS user identifier. Threading is relatively easy by encoding the thread ID into the reply-to field.

What I think is significantly more difficult is to create a simple way to create new threads. What I fear is:

One, instead of having (relatively( nice, clean threads as we have now, sites will have threads that meander, get off topic, chase two or more topics at a time, etc., as users just hit reply and begin their new topic (as I tried to emulate by changing the subject of this posting.) (And I suspect the answer is in SF since he's supposed to be in Burlingame tomorrow morning.)

Two, how does one best handle replies? Distribution lists angle brackets, right?

> One concern I'd have is the degree to which we can capture bboard web
> input from an incoming email. Can an
> email provide an adequate interface to let users indicate whether they
> are replying to a post or its parent?
>> And what about identifying posters? One of the great things about
>> bboard is that it links all your posts to your profile. We'd want
>>> incoming emails to be reliably recognized by bboard as coming from a
>>> specific registered user. In ACES, at least, making this work required
> some thought. Basically, (as I understand it), the ACS generated a
> unique "reply-to" email address for each user receiving an email alert
> for each post. The system could reliably identify the users because
> only one user could possibly be replying to a given email
> address. Will it be possible to make something like this work with a
> third-party listserver?

Well you get the idea.

Some of this can be attacked by user education much the same way that (as jim Zamboob as been recently educated), a bit of moderation, and the creation of easy ways of doing the *right* thing.

Some can be attacked by semi-intelligent software that knows how to remove signatures, and remove quoted portions.

But I bet such a system will need some tuning. If achievable, I think it would be valuable.

I would have to with Jerry. When one makes it too easy to reply, the relevance of the messages to the thread would suffer. I believe this was mentioned somewhere on Phil's website.

I think it would be most effective set the References:, In-Reply_to and Message-ID accordingly. Isn't the Message-ID set by the MTA? wouldn't a patch to the MTA be needed to put that back (or generated eternally to the MTA) in the bboard so replies can be threaded? properly?