Network Maintenance: Saturday, November 14, 2009 at 04:00-06:00 UTC/GMT
Posted by![[info]](http://l-stat.livejournal.com/img/userinfo.gif){kind=small}
dwell in ![[info]](http://l-stat.livejournal.com/img/community.gif){kind=small}
lj_maintenance on 2009.11.11 at 14:00
EDIT@08:16 UTC/GMT. Wow. That was ugly. I expected it to go for 30 minutes and have maybe 1 minute of broken connectivity. Instead it lasted over 4 hours and we had 10 minutes of downtime directly related to the load balancer upgrades and then another 5-10 minutes of downtime when our primary Pingback database server crashed and the secondary couldn't take over; which could have been indirectly caused by the network upgrade missing a self-VIP.
Anyways, we're up, we're working, the load balancers are barely breaking a sweat right now and I need some food and a shot of whiskey. I don't even *like* whiskey!!
Thanksmhwest and dnewhall for helping out!
---
On Saturday the 14th at 4AM UTC/GMT we will be upgrading the operating system of our network load balancers to a newer version, one that will allow us to use both CPUs! Nifty, because multiprocessing is nice.
Since we have 2 load balancers, the plan is to upgrade 1 at a time, and there really should be very little impact to our website. Hopefully you won't notice a thing and I'll get to go back to the hotel and watch some wonderful late night infomercials.
We've got a lot of exciting projects coming up for 2010 and we're hoping that we'll be able to deliver them all to you, that you will find it useful/cool/lovely and then you will use the site even more. Behind-the-scenes work like this will give us the capacity to handle the anticipated traffic, so expect a few more maintenance windows especially in the beginning of next year as we've got some neat ideas to improve performance around here! We had the recent 30-45 minute outage yesterday due to one of our logging databases filling up disk space -- not so great design coupled with my human error in handling the initial problem -- and it looks like we're going to finally have some resources to eliminate stuff like that. I can't wait!
As usual, I will be updating status.livejournal.org before and after, just in case you are not able to reach our main website during the work.
Anyways, we're up, we're working, the load balancers are barely breaking a sweat right now and I need some food and a shot of whiskey. I don't even *like* whiskey!!
Thanks
mhwest and dnewhall for helping out!---
On Saturday the 14th at 4AM UTC/GMT we will be upgrading the operating system of our network load balancers to a newer version, one that will allow us to use both CPUs! Nifty, because multiprocessing is nice.
Since we have 2 load balancers, the plan is to upgrade 1 at a time, and there really should be very little impact to our website. Hopefully you won't notice a thing and I'll get to go back to the hotel and watch some wonderful late night infomercials.
We've got a lot of exciting projects coming up for 2010 and we're hoping that we'll be able to deliver them all to you, that you will find it useful/cool/lovely and then you will use the site even more. Behind-the-scenes work like this will give us the capacity to handle the anticipated traffic, so expect a few more maintenance windows especially in the beginning of next year as we've got some neat ideas to improve performance around here! We had the recent 30-45 minute outage yesterday due to one of our logging databases filling up disk space -- not so great design coupled with my human error in handling the initial problem -- and it looks like we're going to finally have some resources to eliminate stuff like that. I can't wait!
As usual, I will be updating status.livejournal.org before and after, just in case you are not able to reach our main website during the work.
