A few bumps this morning…
These were around 9:45am ET. I had to reboot both NFS servers due to recent FreeBSD kernel patches.
Continue Reading →These were around 9:45am ET. I had to reboot both NFS servers due to recent FreeBSD kernel patches.
Continue Reading →We’re experiencing a service interruption currently related to a power overload in the datacenter. This began at roughly 9:30pm ET. As of about 10:45pm ET web services have been restored but we still need to redistribute some of the power load. Updates to follow. Update 11:05pm ET: Mail services should be back now. Post mortem [...]
Continue Reading →I’ve got some brief maintenance to do tomorrow morning on the NFS server for the mail cluster. Mail will be unavailable for a short period, probably 30 minutes. Watch here for updates! Update 5/25@10:10am ET – I’m pushing this until tomorrow morning – I’ve got all the initial prep done but tomorrow gives me some [...]
Continue Reading →We had a few machines reboot just now. Still investigating… UPDATE 1 – 12:45pm ET – All services are restored. Investigation continuing. UPDATE 2 – 1:10pm ET – As best I can tell this was caused by a cascading power issue in the datacenter. I believe due to a batch task plus other normal activity [...]
Continue Reading →Just a quick note that this morning our PHP install was upgraded from 5.3.10 to 5.3.13 to patch several security issues. Please contact support if you have any issues resulting from the upgrade.
Continue Reading →Outage estimated at approximately 6 hours, from Saturday 4/7 @ 7pm ET through 4/8 @ 1am ET. Power issues affecting our primary NFS server appeared to cause some sort of boot failure. Emergency NFS migration performed. All services restored, all data should be intact. More info to come. Update – 10:15am 4/8 – MySQL InnoDB [...]
Continue Reading →A couple customers have written in this morning to report issues with SQMail. I’m looking at this now. UPDATE @10:25am ET – I found the cause – one of GeekISP’s firewalls rebooted last night, and the daemon startup sequence was such that haproxy, which is used to handle load balancing of some internal traffic, started [...]
Continue Reading →Just a quickie heads-up – I’m doing some work on the webservers this morning, so they may be a little slow until I’m done. Shouldn’t be more than 15-20 minutes, hopefully. EDIT – Took a little longer than expected (I had to remember that there is a crazy manual patch step required for libxml2, see [...]
Continue Reading →Some of you may have noticed lately that the ‘Learn FP’ mail folder has been a little less reliable lately. If you got a message that was incorrectly flagged as Spam and dropped it into ‘Learn FP’, it might end up back in your Spam folder. I’ve addressed this and it should not happen again [...]
Continue Reading →Happy Monday folks… I’ll be upgrading GeekISP’s PHP install to 5.3.10 to patch a security vulnerability announced a few days ago. Expect a little bit of a slowdown as I take various backends out of service for the upgrade. I’ll try to handle everything as quickly as possible. PS – I obviously didn’t get to [...]
Continue Reading →