WeLL System Status

Wed Nov 21 17:10:13 PST 2012

Amazon has alerted us that our SMTP server is running on "degraded
hardware". To get things back to normal, we will be
rebooting the server later tonight (around 10pm). If all goes well, it
should only take about 5 minutes.

Alpine users are most likely to see any
effects - while the server is rebooting, Alpine will complain if you
try to send email. Just resend the mail a few minutes later if this
happens to you. Other customers may seem some minor delays on mail being received or
sent, but in almost all cases, the delayed mail should be delivered
properly some time after the system comes back up.

Sorry for any inconvenience this reboot will cause...

____

Tue Oct 2 22:03:27 PDT 2012

All systems are up and running.

However, we have changed our DNS authoritative server, and some DNS servers are being slow to pick up this change. If you are having trouble connecting to any WELL machines, you may want to change your DNS server on your router, Mac, or PC to use 8.8.8.8 as the primary server. However, be careful to keep a record of the previous settings, since you will likely want to change these settings back at some point. Your best bet is to wait until your normal DNS server picks up the updates, which should happen within a few hours in most cases.

NOTE: It looks like there have been a few cases of mail bouncing from about 7:40pm-8:50pm, and again from about 10:05pm-10:20pm US Pacific Time. Please ask senders to resend if you are missing mail from this period.

We apologize for any inconvenience resulting from this necessary update.

____

Fri Jun 15 12:18:22 PDT 2012

Amazon EC2 services, the cloud where our servers reside, has called for us to shut down and relaunch well.com and user.well.com, the servers where WELL conferencing lives. Tomorrow, Saturday June 16, those services will be unavailable from approximately 9pm until 11:00pm Pacific Daylight Time while wolfy migrates services to new instances. (Last night there was significant disruption that was probably unrelated to this next action, but which also came from problems at the Amazon cloud. Sorry for the inconvenience.) Thank you for your patience and loyalty.

____

Fri Jun 15 09:38:22 PDT 2012

All systems are up and running.

____

Fri Jun 15 01:22:04 PDT 2012

Email is down due to problems in the Amazon Cloud. We do not have an ETA at this time, but email
may not be available until late morning or later.

____

Wed Mar 14 15:04:28 PDT 2012

We will be rebooting well.com and user.well.com Wednesday evening (Mar 14)
at about 11pm US Pacific Time to make a minor configuration change.
The reboot and maintenance will take about 20 minutes, and will effect
access to the conferences via both Picospan and the Web, as well as all
shell services (including alpine mail). We apologize for the
inconvenience.

____

Thu Mar 1 22:26:11 PST 2012

(ETA changed to midnight - we're running a bit behind schedule)

On Thursday evening between about 9pm and midnight US Pacific Time,
WELL conferencing via UNIX shell and the web will be down, with
all access to well.com and user.well.com disabled for the duration.
During this maintenance, we will be switching over to a new system
kernel - we expect that this will put an end to the kernel panics we
have seen over the last two months.

Access to your email via https://iris.well.com/, POP, and IMAP will
not be affected by this maintenance. However, please note that access
to email via the "alpine" mail program will not be available
since well.com itself will be unavailable.

We apologize for the inconvenience of this maintenance, but random
system crashes are even more inconvenient.

____

Thu Feb 23 21:15:19 PST 2012

ENGAGED IS DOWN - the database was accidentally trashed. We hope
to have it back up by 10:30pm US Pacific Time.

____

Wed Jan 11 17:00:35 PST 2012

We did not reboot Tuesday, so we will be rebooting well.com tonight, Jan 11, at about 10pm US Pacific Time. The reboot will take 5 to 10 minutes, and will effect all uses of the conferences.

____

Wed Jan 11 09:37:59 PST 2012

All systems are up and running.

____

Tue Jan 10 12:19:25 PST 2012

We will be rebooting well.com tonight at about 10pm US Pacific Time in
an effort to address the repeated system panics. The reboot will take 5 to 10 minutes, and will effect all uses of the conferences.

____

Fri Dec 16 15:43:25 PST 2011

After a reboot, people are back posting. All systems are up and running.

____

Fri Dec 16 14:55:25 PST 2011

We are having a problem with the well.com and user.well.com machines. While we resolve this problem, the conferences -- our forums -- are unreachable. Email at iris.well.com and the www.well.com web page remain up. We are sorry for the inconvenience.

____

Fri Dec 9 10:53:25 PST 2011

All systems are up and running.

____

Thu Dec 8 2011 16:57:24 PST 2011

Our service provider, Amazon, needs to reboot our systems to perform some sort of system update (presumably patches). Fortunately, we can prevent the
automatic reboot by rebooting manually. The best time for us to do this
is tonight, even though that means a short notice outage.

So, tonight at about 10pm US Pacific Time, each of the various well.com systems will be brought down then rebooted. Each reboot should take less than 5
minutes, though the reboot of the Zimbra machine may take a few minutes
longer (it takes several minutes just for Zimbra to shut down, and
another few minutes to start up).

We apologize for the short notice for this announcement.

____

Thu Oct 27 14:10:50 PDT 2011

Most remaining bugs appear to have been eradicated. If you encounter any, please report them in the cloud discussion in the welltech conference. Log in, shortcut is "g welltech" to get there.

If you use the PicoSpan interface and you're getting error messages on login, the problem may be configuration in your personal log in files. If you send a message to helpdesk, we can back up and replace your current log in files with basic files so that you can test each line to find the culprit.

____

Wed Oct 26 07:45:03 PDT 2011

The WELL is now back up and running in the
Amazon Elastic Computing Cloud (EC2). Please report any bugs by going to the cloud discussion in the welltech conference. Log in, shortcut is "g welltech" to get there. If there is a malformed conference mentioned in an error message on your your Conference List page when you first connect to open a page at "https://user.well.com/engaged.cgi?" you may want to remove the bad conference temporarily by editing your Settings. Reporting a bug could win you one of a bunch of interesting books, so don't hesitate.

____

Mon Oct 24 16:42:39 PDT 2011

Migration down-time planned

On Tuesday night, Oct 25, we finish up our migration into the
Amazon Elastic Computing Cloud (EC2). The last machines to
be moved will be the main machine (well.com), the conferencing
web server (user.well.com), and the web page server (www.well.com).

As part of this migration, conferencing (via both the command line
and web browser), as well as all access to The WELL's web pages,
will be disabled. Email will remain available, except for the
"alpine" mailer. We're anticipating that the migration should be
complete in about 6 hours: from 9pm PDT Tuesday evening to 3am PDT
Wednesday morning.

Following the migration, you should be able to use conferencing and
all web services as before, with only minor changes that most people
will not see. If by some chance you do experience problems after the
system is restored to service, don't try to change your settings please. If
we are still sorting out the move that could make things harder to
diagnose. Please just contact helpdesk with any bug reports or post in
the System Status topic in the WELL tech or News conference, so a proper
diagnosis can be made.

We apologize for any inconvenience this downtime may cause.

____

Tue Aug 30 15:48:19 PDT 2011

All systems are up and running normally.

____

Tues Aug 30 13:07:22 PDT 2011

Performance of the email system remains sluggish - we are working to restore good service.

We apologize for any inconvenience this slowness may cause.

____

Fri Aug 26 13:43:56 PDT 2011

On Monday night, Aug 29, we begin our migration into the
Amazon Elastic Computing Cloud (EC2). The first machines to
be moved will be the Zimbra mail server (iris.well.com) and a
companion server.

As part of this migration, email will be disabled while the
migration is under way. We're anticipating that the migration
should be complete in about 6 hours: from 9pm PDT Monday
evening to 3am PDT Tuesday morning.

During the migration, incoming email from other sites should
be queued at those sites, and re-sent later after we begin
accepting incoming email again. WELL customers will not be
able to look at previously received email during the outage
unless you have already moved that email onto your computer's
storage device.

Following the migration, you should be able to continue
using email using the settings that have worked for the last
few weeks - nothing should change. If you encounter problems
after the migration, please avoid changing your mail settings
and report the problems to Helpdesk or post in System Status Reports.

We apologize for any inconvenience this downtime may cause.

____

Thu Aug 18 21:37:03 PDT 2011

The WELL is now back up and running. Engaged is still presenting some
problems, but for now, you can make your own repairs using the instructions on your Conference List page.

____

Thu Aug 18 14:54:03 PDT 2011

RAID failure knocked out some services, currently being restored

Around 1:00 am on the morning of Thursday, August 18th, home directories became unreachable at The WELL, causing all services except email to fail.

At 9:30 the following morning we we were able to place the following unadorned message on the well.com site:

Most functions of The WELL are currently not available due to
a hardware failure of the RAID where most of our data is kept.
We are working on restoring full service, but do not yet have
an estimated completion time. We apologize for this inconvenience.

Please note that email via IMAP, POP, and the Web should be
available at this time.

During that time we went through some replacement hardware options, settled on one, and got it installed at our co-location facility. At about 14:45 PDT logins became possible again.

RAIDs are designed to prevent such problems when a disk fails, and we are in the process of moving well.com to the cloud, so this failure was perplexing and hopefully very soon all questions will become moot. Some services are still being repaired and adjusted at this time. Comments are welcome, please email Helpdesk at the address on the footer of this page, or swing by the welltech conference.

We are sorry for the disruption.

____

Mon Aug 1 16:42:55 PDT 2011

Mail upgrade complete: If you can't get mail you need to change settings

All systems are up and running. We now have IMAP available for all mail customers. Please be advised that we have shut
off all access to two older mail servers as of this afternoon as
part of the upgrade to email started in 2009. If you are pointing to those servers in your settings, your mail will no longer work.

If you are not getting or sending email, you probably need to update some settings. You no longer have access to smtp.well.com nor to mh.well.com (mail.well.com). All incoming and outgoing mail operations will now involve the iris.well.com server. To get the appropriate Ports, look up your appropriate settings.

1. You should have access to your new email via a direct web browser login at
iris.well.com now and henceforth. This is the easy, quick way for most people to see mail.

2. New mail help for all kinds of programs is now up at http://www.well.com/emailhelp.html
with an overview of the system, general settings and a link to a page
with more settings for specific mobile devices and computer email programs.

3. You can move your old email into the new environment by use of the
utilities page if you wish.

4. There is a way to use Alpine for UNIX access to your mail. See the help page first.

5. We attempted to migrate you with settings matching your former preferences. In some situations we were not able to do that! In particular, if you were forwarding elsewhere and leaving a copy you are probably set up with straight forwarding now, and you will want to adjust that with a web login to iris.well.com.

If you have questions, contact helpdesk or ask fellow users who have
solved the problem in the welltech conference. Thank you for your patience.

____

Mon Aug 1 09:59:46 PDT 2011

Mail upgrade is in progress.

All remaining mailboxes at The WELL are being transferred to the Zimbra email system at iris.well.com over the next several hours today. Please see the mail help page for instructions on setting up once your mailbox is converted, or use your web browser to go to iris.well.com. (Note that you cannot log in there until the upgrade script converts your mail settings.) Thank you for your patience.

____

Wed Jun 22 12:55:09 PDT 2011

All systems are up and running.

____

Monday, June 20, 2011 05:36:27 PM PDT

The Zimbra email system on iris.well.com will be unavailable Tuesday evening (Jun 21) between the hours of 10pm and midnight so that we can perform a required software update. The legacy email system will remain up during this period. Sorry for the inconvenience.

____

Fri May 13 13:50:40 PDT 2011

Members using the Legacy email server are finding their connections time out when they are trying to access mail from mail.well.com. We are working on the situation.