I’ve got some brief maintenance to do tomorrow morning on the NFS server for the mail cluster. Mail will be unavailable for a short period, probably 30 minutes. Watch here for updates! Update 5/25@10:10am ET – I’m pushing this until tomorrow morning – I’ve got all the initial prep done but tomorrow gives me some [...]
Continue Reading →We had a few machines reboot just now. Still investigating… UPDATE 1 – 12:45pm ET – All services are restored. Investigation continuing. UPDATE 2 – 1:10pm ET – As best I can tell this was caused by a cascading power issue in the datacenter. I believe due to a batch task plus other normal activity [...]
Continue Reading →Just a quick note that this morning our PHP install was upgraded from 5.3.10 to 5.3.13 to patch several security issues. Please contact support if you have any issues resulting from the upgrade.
Continue Reading →Outage estimated at approximately 6 hours, from Saturday 4/7 @ 7pm ET through 4/8 @ 1am ET. Power issues affecting our primary NFS server appeared to cause some sort of boot failure. Emergency NFS migration performed. All services restored, all data should be intact. More info to come. Update – 10:15am 4/8 – MySQL InnoDB [...]
Continue Reading →A couple customers have written in this morning to report issues with SQMail. I’m looking at this now. UPDATE @10:25am ET – I found the cause – one of GeekISP’s firewalls rebooted last night, and the daemon startup sequence was such that haproxy, which is used to handle load balancing of some internal traffic, started [...]
Continue Reading →Just a quickie heads-up – I’m doing some work on the webservers this morning, so they may be a little slow until I’m done. Shouldn’t be more than 15-20 minutes, hopefully. EDIT – Took a little longer than expected (I had to remember that there is a crazy manual patch step required for libxml2, see [...]
Continue Reading →Some of you may have noticed lately that the ‘Learn FP’ mail folder has been a little less reliable lately. If you got a message that was incorrectly flagged as Spam and dropped it into ‘Learn FP’, it might end up back in your Spam folder. I’ve addressed this and it should not happen again [...]
Continue Reading →Happy Monday folks… I’ll be upgrading GeekISP’s PHP install to 5.3.10 to patch a security vulnerability announced a few days ago. Expect a little bit of a slowdown as I take various backends out of service for the upgrade. I’ll try to handle everything as quickly as possible. PS – I obviously didn’t get to [...]
Continue Reading →Twice in one week… fun times. It’s been back now for a bit, and I’m continuing to investigate the cause and any possible fixes.
Continue Reading →Well that was quick… we just had our first crash with the shell server since the upgrade to OpenBSD 5.0. This one was different from prior crashes under 4.7. I’m still investigating the issue in the hopes of uncovering a fix. The machine is back up and working properly again – total downtime was around [...]
Continue Reading →