1. Authentication errors (“Invalid password”, “Password rejected”, or similar)
2. Timeouts or other very slow operations retrieving mail, deleting mail, or similar

Unfortunately this far the exact underlying cause of the issue has not surfaced. As this has gone on for some time and appears to be happening with increasing frequency and severity, I’m exploring alternative arrangements that would address the issue, even without a full understanding of the problem.

Please accept my apologies for the inconvenience this has caused.

Update 11:05pm ET: Mail services should be back now.

Post mortem analysis:

In our datacenter, our servers span 2 racks. In rack 14, we have a handful of important servers and a small group of UPSs to keep them online. What happened tonight was that ‘umbrella’, the NFS server backing the mail system, exceeded the capacity of its attached UPS, throwing it into overload. Normally this would not have caused any sort of cascading issue, however, the switches connecting rack 14 to our other rack were also on this UPS, thus connectivity was lost to all of rack 14 expanding the outage.

Fortunately this suboptimal layout has now been fixed, and the switches are on one UPS and each NFS server in rack 14 is on their own UPS devices. That should prevent the sort of cascading outage we had tonight, and simultaneously I’ll be talking to the datacenter crew about expanding our power allotment so we can continue to grow.

My apologies for the inconvenience this caused.